[jira] [Commented] (HBASE-3863) Fix failing TestHBaseFsck.testHBaseFsckMeta unit test
[
https://issues.apache.org/jira/browse/HBASE-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030273#comment-13030273
]
Hudson commented on HBASE-3863:
---
Integrated in HBase-TRUNK #1910 (See
[https://builds.apache.org/hudson/job/HBase-TRUNK/1910/])
HBASE-3863 Fix failing TestHBaseFsck.testHBaseFsckMeta unit test
> Fix failing TestHBaseFsck.testHBaseFsckMeta unit test
> -
>
> Key: HBASE-3863
> URL: https://issues.apache.org/jira/browse/HBASE-3863
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Fix For: 0.92.0
>
> Attachments: 3863.txt
>
>
> Failing with:
> {code}
> Error Message
> org.apache.hadoop.hbase.HServerAddress cannot be cast to
> org.apache.hadoop.hbase.ServerName
> Stacktrace
> java.lang.ClassCastException: org.apache.hadoop.hbase.HServerAddress cannot
> be cast to org.apache.hadoop.hbase.ServerName
> at
> org.apache.hadoop.hbase.util.HBaseFsckRepair.fixDupeAssignment(HBaseFsckRepair.java:59)
> at
> org.apache.hadoop.hbase.util.TestHBaseFsck.testHBaseFsckMeta(TestHBaseFsck.java:163)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3763) Add Bloom Block Index Support
[ https://issues.apache.org/jira/browse/HBASE-3763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030242#comment-13030242 ] Joydeep Sen Sarma commented on HBASE-3763: -- Dhruba pointed me to some of these jiras. one quick comment is that _if_ the intention is to keep the filters pinned in memory - then we can convert the load at read time to: - load at startup time as quickly as possible - keep the filter pinned in memory when writing out new hfile (never have to read it in) this would also take out filter reads from client read path. > Add Bloom Block Index Support > - > > Key: HBASE-3763 > URL: https://issues.apache.org/jira/browse/HBASE-3763 > Project: HBase > Issue Type: Improvement > Components: io, regionserver >Affects Versions: 0.89.20100924, 0.90.0, 0.90.1, 0.90.2 >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Minor > Labels: hbase, performance > Fix For: 0.89.20100924 > > Original Estimate: 0h > Remaining Estimate: 0h > > Add a way to save HBase Bloom filters into an array of Meta blocks instead of > one big Meta block, and load only the blocks required to answer a query. > This will allow us faster bloom load times for large StoreFiles & pave the > path for adding Bloom Filter support to HFileOutputFormat bulk load. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2899) hfile.min.blocksize.size ignored/documentation wrong
[
https://issues.apache.org/jira/browse/HBASE-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030229#comment-13030229
]
Hudson commented on HBASE-2899:
---
Integrated in HBase-TRUNK #1909 (See
[https://builds.apache.org/hudson/job/HBase-TRUNK/1909/])
HBASE-3864 Rename of hfile.min.blocksize.size in HBASE-2899 reverted in
HBASE-1861
> hfile.min.blocksize.size ignored/documentation wrong
>
>
> Key: HBASE-2899
> URL: https://issues.apache.org/jira/browse/HBASE-2899
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Francke
>Assignee: stack
>Priority: Trivial
> Attachments: 2899.txt
>
>
> There is a property in hbase-default.xml called {{hfile.min.blocksize.size}}
> set to {{65536}}.
> The description says: Minimum store file block size. The smaller you make
> this, the bigger your index and the less you fetch on a random-access. Set
> size down if you have small cells and want faster random-access of
> individual cells.
> This property is only used in the HFileOutputFormat and nowhere else. So we
> should at least change the description to something more meaningful.
> The other option I see would be: HFile now has a DEFAULT_BLOCKSIZE field
> which could be moved to HConstants and HFile could somehow read the
> {{hfile.min.blocksize.size}} from the Configuration or use
> HConstansts.DEFAULT_BLOCKSIZE if it's not defined. I believe this is what's
> happening to the other config variables?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3821) "NOT flushing memstore for region" keep on printing for half an hour
[ https://issues.apache.org/jira/browse/HBASE-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030226#comment-13030226 ] Hudson commented on HBASE-3821: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > "NOT flushing memstore for region" keep on printing for half an hour > - > > Key: HBASE-3821 > URL: https://issues.apache.org/jira/browse/HBASE-3821 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.90.1 >Reporter: zhoushuaifeng >Assignee: zhoushuaifeng > Fix For: 0.90.3 > > Attachments: HBase-3821 v1.txt, HBase-3821 v2.txt > > > "NOT flushing memstore for region" keep on printing for half an hour in the > regionserver. Then I restart hbase. I think there may be deadlock or cycling. > I know that when splitting region, it will doclose of region, and set > writestate.writesEnabled = false and may run close preflush. This will make > flush fail and print "NOT flushing memstore for region". But It should be > finished after a while. > logs: > 2011-04-18 16:28:27,960 DEBUG > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested > for ufdr,,1303124043153.031f37c9c23fcab17797b06b90205610. because > regionserver60020.cacheFlusher; priority=-1, compaction queue size=1 > 2011-04-18 16:28:30,171 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Flush requested on ufdr,,1303124043153.031f37c9c23fcab17797b06b90205610. > 2011-04-18 16:28:30,171 WARN > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Region > ufdr,,1303124043153.031f37c9c23fcab17797b06b90205610. has too many store > files; delaying flush up to 9ms > 2011-04-18 16:28:32,119 INFO > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs > -- HDFS-200 > 2011-04-18 16:28:32,285 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: > Roll > /hbase/.logs/linux253,60020,1303123943360/linux253%3A60020.1303124206693, > entries=5226, filesize=255913736. New hlog > /hbase/.logs/linux253,60020,1303123943360/linux253%3A60020.1303124311822 > 2011-04-18 16:28:32,287 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: > Found 1 hlogs to remove out of total 2; oldest outstanding sequenceid is > 11037 from region 031f37c9c23fcab17797b06b90205610 > 2011-04-18 16:28:32,288 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: > moving old hlog file > /hbase/.logs/linux253,60020,1303123943360/linux253%3A60020.1303123945481 > whose highest sequenceid is 6052 to > /hbase/.oldlogs/linux253%3A60020.1303123945481 > 2011-04-18 16:28:42,701 INFO org.apache.hadoop.hbase.regionserver.Store: > Completed major compaction of 4 file(s), new > file=hdfs://10.18.52.108:9000/hbase/ufdr/031f37c9c23fcab17797b06b90205610/value/4398465741579485290, > size=281.4m; total size for store is 468.8m > 2011-04-18 16:28:42,712 INFO org.apache.hadoop.hbase.regionserver.HRegion: > completed compaction on region > ufdr,,1303124043153.031f37c9c23fcab17797b06b90205610. after 1mins, 40sec > 2011-04-18 16:28:42,741 INFO > org.apache.hadoop.hbase.regionserver.SplitTransaction: Starting split of > region ufdr,,1303124043153.031f37c9c23fcab17797b06b90205610. > 2011-04-18 16:28:42,770 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Closing ufdr,,1303124043153.031f37c9c23fcab17797b06b90205610.: disabling > compactions & flushes > 2011-04-18 16:28:42,770 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Running close preflush of > ufdr,,1303124043153.031f37c9c23fcab17797b06b90205610. > 2011-04-18 16:28:42,771 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > ufdr,,1303124043153.031f37c9c23fcab17797b06b90205610., current region > memstore size 105.6m > 2011-04-18 16:28:42,818 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Finished snapshotting, commencing flushing stores > 2011-04-18 16:28:42,846 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > NOT flushing memstore for region > ufdr,,1303124043153.031f37c9c23fcab17797b06b90205610., flushing=false, > writesEnabled=false > 2011-04-18 16:28:42,849 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Flush requested on ufdr,,1303124043153.031f37c9c23fcab17797b06b90205610. > 2011-04-18 16:28:42,849 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > NOT flushing memstore for region > ufdr,,1303124043153.031f37c9c23fcab17797b06b90205610., flushing=false, > writesEnabled=false .. > 2011-04-18 17:04:08,803 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > NOT flushing memstore for region > ufdr,,1303124043153.031f37c9c23fcab17797b06b90205610., flushing=false, > writesEnabled=false > 2011-04-18 17:04:08,803 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: >
[jira] [Commented] (HBASE-1502) Remove need for heartbeats in HBase
[ https://issues.apache.org/jira/browse/HBASE-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030227#comment-13030227 ] Hudson commented on HBASE-1502: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Remove need for heartbeats in HBase > --- > > Key: HBASE-1502 > URL: https://issues.apache.org/jira/browse/HBASE-1502 > Project: HBase > Issue Type: Task >Reporter: Nitay Joffe >Assignee: stack >Priority: Blocker > Fix For: 0.92.0 > > Attachments: 1502-4.txt, 1502-v2.txt, 1502-v5.txt, 1502-v6.txt, > 1502-v7.txt, 1502.txt > > > HBase currently uses heartbeats between region servers and the master, > piggybacking information on them when it can. This issue is to investigate if > we can get rid of the need for those using ZooKeeper events. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3847) Turn off DEBUG logging of RPCs in WriteableRPCEngine on TRUNK
[ https://issues.apache.org/jira/browse/HBASE-3847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030225#comment-13030225 ] Hudson commented on HBASE-3847: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Turn off DEBUG logging of RPCs in WriteableRPCEngine on TRUNK > - > > Key: HBASE-3847 > URL: https://issues.apache.org/jira/browse/HBASE-3847 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 0.92.0 > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3777) Redefine Identity Of HBase Configuration
[
https://issues.apache.org/jira/browse/HBASE-3777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030228#comment-13030228
]
Hudson commented on HBASE-3777:
---
Integrated in HBase-TRUNK #1909 (See
[https://builds.apache.org/hudson/job/HBase-TRUNK/1909/])
> Redefine Identity Of HBase Configuration
>
>
> Key: HBASE-3777
> URL: https://issues.apache.org/jira/browse/HBASE-3777
> Project: HBase
> Issue Type: Improvement
> Components: client, ipc
>Affects Versions: 0.90.2
>Reporter: Karthick Sankarachary
>Assignee: Karthick Sankarachary
>Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 3777-TOF.patch, HBASE-3777-V2.patch,
> HBASE-3777-V3.patch, HBASE-3777-V4.patch, HBASE-3777-V6.patch,
> HBASE-3777.patch
>
>
> Judging from the javadoc in {{HConnectionManager}}, sharing connections
> across multiple clients going to the same cluster is supposedly a good thing.
> However, the fact that there is a one-to-one mapping between a configuration
> and connection instance, kind of works against that goal. Specifically, when
> you create {{HTable}} instances using a given {{Configuration}} instance and
> a copy thereof, we end up with two distinct {{HConnection}} instances under
> the covers. Is this really expected behavior, especially given that the
> configuration instance gets cloned a lot?
> Here, I'd like to play devil's advocate and propose that we "deep-compare"
> {{HBaseConfiguration}} instances, so that multiple {{HBaseConfiguration}}
> instances that have the same properties map to the same {{HConnection}}
> instance. In case one is "concerned that a single {{HConnection}} is
> insufficient for sharing amongst clients", to quote the javadoc, then one
> should be able to mark a given {{HBaseConfiguration}} instance as being
> "uniquely identifiable".
> Note that "sharing connections makes clean up of {{HConnection}} instances a
> little awkward", unless of course, you apply the change described in
> HBASE-3766.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3843) splitLogWorker starts too early
[ https://issues.apache.org/jira/browse/HBASE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030219#comment-13030219 ] Hudson commented on HBASE-3843: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > splitLogWorker starts too early > --- > > Key: HBASE-3843 > URL: https://issues.apache.org/jira/browse/HBASE-3843 > Project: HBase > Issue Type: Bug >Reporter: Prakash Khemani >Assignee: Prakash Khemani > Fix For: 0.92.0 > > Attachments: > 0001-HBASE-3843-start-splitLogWorker-later-at-region-serv.patch > > > splitlogworker should be started in startServiceThreads() instead of in > initializeZookeeper(). This will ensure that the region server accepts a > split-logging tasks only after it has successfully done reportForDuty() to > the master. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3721) Speedup LoadIncrementalHFiles
[ https://issues.apache.org/jira/browse/HBASE-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030224#comment-13030224 ] Hudson commented on HBASE-3721: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Speedup LoadIncrementalHFiles > - > > Key: HBASE-3721 > URL: https://issues.apache.org/jira/browse/HBASE-3721 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.92.0 > > Attachments: 3721-v2.txt, 3721-v3.txt, 3721-v4.txt, 3721-v6.patch, > 3721.txt, LoadIncrementalHFiles.java > > > From Adam Phelps: > from the logs it looks like <1% of the hfiles we're loading have to be split. > Looking at the code for LoadIncrementHFiles (hbase v0.90.1), I'm actually > thinking our problem is that this code loads the hfiles sequentially. Our > largest table has over 2500 regions and the data being loaded is fairly well > distributed across them, so there end up being around 2500 HFiles for each > load period. At 1-2 seconds per HFile that means the loading process is very > time consuming. > Currently server.bulkLoadHFile() is a blocking call. > We can utilize ExecutorService to achieve better parallelism on multi-core > computer. > New configuration parameter "hbase.loadincremental.threads.max" is introduced > which sets the maximum number of threads for parallel bulk load. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3849) Fix master ui; hbase-1502 broke requests/second
[ https://issues.apache.org/jira/browse/HBASE-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030222#comment-13030222 ] Hudson commented on HBASE-3849: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Fix master ui; hbase-1502 broke requests/second > --- > > Key: HBASE-3849 > URL: https://issues.apache.org/jira/browse/HBASE-3849 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 0.92.0 > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1861) Multi-Family support for bulk upload tools (HFileOutputFormat / loadtable.rb)
[ https://issues.apache.org/jira/browse/HBASE-1861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030223#comment-13030223 ] Hudson commented on HBASE-1861: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) HBASE-3864 Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861 > Multi-Family support for bulk upload tools (HFileOutputFormat / loadtable.rb) > - > > Key: HBASE-1861 > URL: https://issues.apache.org/jira/browse/HBASE-1861 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 0.20.0 >Reporter: Jonathan Gray >Assignee: Nicolas Spiegelberg > Fix For: 0.92.0 > > Attachments: HBASE1861-incomplete.patch > > > Add multi-family support to bulk upload tools from HBASE-48. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3844) Book.xml (removing link to defunct wiki) and Performance.xml (adding client tip)
[ https://issues.apache.org/jira/browse/HBASE-3844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030221#comment-13030221 ] Hudson commented on HBASE-3844: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Book.xml (removing link to defunct wiki) and Performance.xml (adding client > tip) > > > Key: HBASE-3844 > URL: https://issues.apache.org/jira/browse/HBASE-3844 > Project: HBase > Issue Type: Improvement >Reporter: Doug Meil >Assignee: Doug Meil >Priority: Minor > Fix For: 0.92.0 > > Attachments: book_HBASE_3844.xml.patch, > performance_HBASE_3844.xml.patch > > > Book.xml - in the FAQ it had a link to a "Frequently Seen Errors" wiki page. > This page is labeled as defunct and doesn't even have anything useful on it > anyway. Removed the link to that page. > Performance.xml - added tip in Performance under client about attribute > selection. This is one of those "obvious but not so obvious" topics, if you > only need 3 attributes don't select the entire column family. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3863) Fix failing TestHBaseFsck.testHBaseFsckMeta unit test
[
https://issues.apache.org/jira/browse/HBASE-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030214#comment-13030214
]
Hudson commented on HBASE-3863:
---
Integrated in HBase-TRUNK #1909 (See
[https://builds.apache.org/hudson/job/HBase-TRUNK/1909/])
> Fix failing TestHBaseFsck.testHBaseFsckMeta unit test
> -
>
> Key: HBASE-3863
> URL: https://issues.apache.org/jira/browse/HBASE-3863
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Fix For: 0.92.0
>
> Attachments: 3863.txt
>
>
> Failing with:
> {code}
> Error Message
> org.apache.hadoop.hbase.HServerAddress cannot be cast to
> org.apache.hadoop.hbase.ServerName
> Stacktrace
> java.lang.ClassCastException: org.apache.hadoop.hbase.HServerAddress cannot
> be cast to org.apache.hadoop.hbase.ServerName
> at
> org.apache.hadoop.hbase.util.HBaseFsckRepair.fixDupeAssignment(HBaseFsckRepair.java:59)
> at
> org.apache.hadoop.hbase.util.TestHBaseFsck.testHBaseFsckMeta(TestHBaseFsck.java:163)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3864) Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861
[
https://issues.apache.org/jira/browse/HBASE-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030213#comment-13030213
]
Hudson commented on HBASE-3864:
---
Integrated in HBase-TRUNK #1909 (See
[https://builds.apache.org/hudson/job/HBase-TRUNK/1909/])
HBASE-3864 Rename of hfile.min.blocksize.size in HBASE-2899 reverted in
HBASE-1861
> Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861
> ---
>
> Key: HBASE-3864
> URL: https://issues.apache.org/jira/browse/HBASE-3864
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: 0.92.0
>
> Attachments: hbase-3864.0.patch
>
>
> HBASE-2899 renamed {{hfile.min.blocksize.size}} to
> {{hbase.mapreduce.hfileoutputformat.blocksize}}. However, HBASE-1861
> (committed after HBASE-2899) reverted this change.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3865) Failing TestWALReplay
[ https://issues.apache.org/jira/browse/HBASE-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030216#comment-13030216 ] Hudson commented on HBASE-3865: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) HBASE-3865 Failing TestWALReplay > Failing TestWALReplay > - > > Key: HBASE-3865 > URL: https://issues.apache.org/jira/browse/HBASE-3865 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 0.92.0 > > Attachments: 3865.txt > > > Failed last few times up on jenkins. The change to the signature of the > internalFlushCache to add passing of a MonitorVisitor meant the override here > was no longer called. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3597) ageOfLastAppliedOp should update after cluster replication failures
[ https://issues.apache.org/jira/browse/HBASE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030217#comment-13030217 ] Hudson commented on HBASE-3597: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > ageOfLastAppliedOp should update after cluster replication failures > --- > > Key: HBASE-3597 > URL: https://issues.apache.org/jira/browse/HBASE-3597 > Project: HBase > Issue Type: Bug > Components: replication >Affects Versions: 0.90.1 >Reporter: Otis Gospodnetic >Assignee: Jean-Daniel Cryans > Fix For: 0.90.3 > > Attachments: HBASE-3597.patch > > > The value of ageOfLastAppliedOp in JMX doesn't update after replication > starts failing, and it should. See: http://search-hadoop.com/m/jFPgF1HfnLc -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3846) Set RIT timeout higher
[ https://issues.apache.org/jira/browse/HBASE-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030218#comment-13030218 ] Hudson commented on HBASE-3846: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Set RIT timeout higher > -- > > Key: HBASE-3846 > URL: https://issues.apache.org/jira/browse/HBASE-3846 > Project: HBase > Issue Type: Task >Affects Versions: 0.90.2 >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Critical > Fix For: 0.90.3 > > Attachments: HBASE-3846.patch > > > As I was talking in HBASE-3669, it is really easy with the current RIT > timeout to end up in situations where regions are doubly assigned, not > assigned at all or assigned but the master doesn't know about it. As a > bandaid, we should set hbase.master.assignment.timeoutmonitor.timeout to what > the ZK session timeout is. > We had to do that to one of our clusters to be able to start it, else the > master kept racing with itself. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3825) performance.xml - adding a few common configuration changes in the 'config' sub-section
[ https://issues.apache.org/jira/browse/HBASE-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030220#comment-13030220 ] Hudson commented on HBASE-3825: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > performance.xml - adding a few common configuration changes in the 'config' > sub-section > --- > > Key: HBASE-3825 > URL: https://issues.apache.org/jira/browse/HBASE-3825 > Project: HBase > Issue Type: Improvement >Reporter: Doug Meil >Assignee: Doug Meil >Priority: Minor > Fix For: 0.92.0 > > Attachments: performance_HBASE_3825.xml.patch > > > There are a few common configurations that people make to adjust the > RegionServer memory behavior (% of block cache, upperLimit, etc.) > Adding a few entries under the 'configurations' sub-section in performance to > reference the configuration section for each item. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3431) Regionserver is not using the name given it by the master; double entry in master listing of servers
[
https://issues.apache.org/jira/browse/HBASE-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030215#comment-13030215
]
Hudson commented on HBASE-3431:
---
Integrated in HBase-TRUNK #1909 (See
[https://builds.apache.org/hudson/job/HBase-TRUNK/1909/])
> Regionserver is not using the name given it by the master; double entry in
> master listing of servers
>
>
> Key: HBASE-3431
> URL: https://issues.apache.org/jira/browse/HBASE-3431
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.0
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 3431-v2.txt, 3431-v3.txt, 3431-v3.txt, 3431-v4.txt,
> 3431.txt
>
>
> Our man Ted Dunning found the following where RS checks in with one name, the
> master tells it use another name but we seem to go ahead and continue with
> our original name.
> In RS logs I see:
> {code}
> 2011-01-07 15:45:50,757 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer [regionserver60020]:
> Master passed us address to use. Was=perfnode11:60020, Now=10.10.30.11:60020
> {code}
> On master I see
> {code}
> 2011-01-07 15:45:38,613 INFO org.apache.hadoop.hbase.master.ServerManager
> [IPC Server handler 0 on 6]: Registering
> server=10.10.30.11,60020,1294443935414, regionCount=0, userLoad=false
> {code}
>
> then later
> {code}
> 2011-01-07 15:45:44,247 INFO org.apache.hadoop.hbase.master.ServerManager
> [IPC Server handler 2 on 6]: Registering
> server=perfnode11,60020,1294443935414, regionCount=0, userLoad=true
> {code}
> This might be since we started letting servers register in other than with
> the reportStartup.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3861) MiniZooKeeperCluster.startup() should refer to hbase.zookeeper.property.maxClientCnxns
[
https://issues.apache.org/jira/browse/HBASE-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030212#comment-13030212
]
Hudson commented on HBASE-3861:
---
Integrated in HBase-TRUNK #1909 (See
[https://builds.apache.org/hudson/job/HBase-TRUNK/1909/])
HBASE-3861 MiniZooKeeperCluster.startup() should refer to
hbase.zookeeper.property.maxClientCnxns; backing it out till figure the TestHCM
OOME issue
> MiniZooKeeperCluster.startup() should refer to
> hbase.zookeeper.property.maxClientCnxns
> --
>
> Key: HBASE-3861
> URL: https://issues.apache.org/jira/browse/HBASE-3861
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Fix For: 0.90.3, 0.92.0
>
> Attachments: HBASE-3861.patch, HBASE-3861.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Currently the number of the client connections is hard-wired to 1000:
> {{{
> standaloneServerFactory = new NIOServerCnxnFactory();
> standaloneServerFactory.configure(new
> InetSocketAddress(clientPort),1000);
> } catch (BindException e) {
>
> }}}
> This should be set according to the test environment's hbase configuration.
> The property in
> question is : hbase.zookeeper.property.maxClientCnxns.
> Currently some tests such as org.apache.hadoop.hbase.client.TestHCM fail
> because the number of connections used by the HBase client exceeds 1000.
> Recently MAX_CACHED_HBASE_INSTANCES increased from 31 to 2000 on 0.90 branch:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&r1=1096818&r2=1096817&view=diff&pathrev=1096818
> and correspondingly the hbase config on the Zookeeper server-side also
> increased in hbase-default.xml:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/resources/hbase-default.xml?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&r1=1091594&r2=1091593&view=diff&pathrev=1091594
> So if MiniZKCluster looks at this setting, the test won't have this failure.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3860) HLog shouldn't create a new HBC when rolling
[
https://issues.apache.org/jira/browse/HBASE-3860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030210#comment-13030210
]
Hudson commented on HBASE-3860:
---
Integrated in HBase-TRUNK #1909 (See
[https://builds.apache.org/hudson/job/HBase-TRUNK/1909/])
> HLog shouldn't create a new HBC when rolling
>
>
> Key: HBASE-3860
> URL: https://issues.apache.org/jira/browse/HBASE-3860
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.90.2
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Critical
> Fix For: 0.90.3
>
>
> HBASE-2059 added this change in HLog.rollWriter:
> {code}
> this.writer = createWriter(fs, newPath, new HBaseConfiguration(conf));
> {code}
> Which has since become:
> {code}
> HLog.Writer nextWriter = this.createWriterInstance(fs, newPath,
> HBaseConfiguration.create(conf));
> {code}
> It's unclear to me why it needs to do that, but it bite us today because we
> swapped jars under a running hbase with:
> {quote}
> 2011-05-05 12:06:12,876 FATAL org.apache.hadoop.conf.Configuration: error
> parsing conf file: java.util.zip.ZipException: invalid stored block lengths
> 2011-05-05 12:06:12,877 ERROR org.apache.hadoop.hbase.regionserver.LogRoller:
> Log rolling failed
> java.lang.RuntimeException: java.util.zip.ZipException: invalid stored block
> lengths
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1352)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1227)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1156)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:427)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:63)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:89)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:100)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:110)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:485)
> at
> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> Caused by: java.util.zip.ZipException: invalid stored block lengths
> at
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:147)
> at
> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:105)
> at java.io.FilterInputStream.read(FilterInputStream.java:66)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(XMLEntityManager.java:2932)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:704)
> at
> com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:186)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:772)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
> at
> com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:235)
> at
> com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
> at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1266)
> ... 9 more
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3853) TestInfoServers failing after HBASE-3835
[ https://issues.apache.org/jira/browse/HBASE-3853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030209#comment-13030209 ] Hudson commented on HBASE-3853: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > TestInfoServers failing after HBASE-3835 > > > Key: HBASE-3853 > URL: https://issues.apache.org/jira/browse/HBASE-3853 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.92.0 > > Attachments: hbase-3853.txt > > > Forgot to update this test - the test was checking the redirections from > index.html to the JSP pages, but since the JSP pages no longer exist, the > test case needs to be fixed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3670) Fix error handling in get(List gets)
[ https://issues.apache.org/jira/browse/HBASE-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030207#comment-13030207 ] Hudson commented on HBASE-3670: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Fix error handling in get(List gets) > - > > Key: HBASE-3670 > URL: https://issues.apache.org/jira/browse/HBASE-3670 > Project: HBase > Issue Type: Bug > Components: client >Affects Versions: 0.92.0 >Reporter: Lars George >Assignee: Harsh J Chouraria > Fix For: 0.92.0 > > Attachments: HBASE-3670.r1.diff > > > See HBASE-3634 for details. The get(List gets) call needs to catch (or > rather use a try/finally) the exception thrown by batch() and copy the Result > instances over and return it. If that is not intended then we need to fix the > JavaDoc in HTableInterface to reflect the new behavior. > In general it seems to make sense to check the various methods (list based > put, get, delete compared to batch) and agree on the correct behavior. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3839) Expose in-progress tasks on web UIs
[ https://issues.apache.org/jira/browse/HBASE-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030203#comment-13030203 ] Hudson commented on HBASE-3839: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Expose in-progress tasks on web UIs > --- > > Key: HBASE-3839 > URL: https://issues.apache.org/jira/browse/HBASE-3839 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.92.0 > > Attachments: hbase-3839.txt, tasks.png > > > HBASE-3836 adds a TaskMonitor class which collects info about what's going on > inside processes. This ticket is to expose the task monitor info on the web > UIs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3695) Some improvements to Hbck to test the entire region chain in Meta and provide better error reporting
[ https://issues.apache.org/jira/browse/HBASE-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030208#comment-13030208 ] Hudson commented on HBASE-3695: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Some improvements to Hbck to test the entire region chain in Meta and provide > better error reporting > > > Key: HBASE-3695 > URL: https://issues.apache.org/jira/browse/HBASE-3695 > Project: HBase > Issue Type: Improvement > Components: util >Affects Versions: 0.90.1 >Reporter: Marc Limotte >Assignee: Marc Limotte > Fix For: 0.90.3 > > > The current Hbck tool will miss some inconsistencies in Meta, and in other > cases will detect an issue, but does not provide much in the way of useful > feedback. > * Incorporate the full region chain tests (similar to check_meta.rb). I.e. > look for overlaps, holes and cycles. I believe check_meta.rb will be > redundant after this change. > * More unit tests, and better tests that will test the actual error > discovered, instead of just errors true/false. > * In the case of overlaps and holes, output both ends of the broken chain. > * Previous implementation runs check() twice. This is inefficient and, more > importantly, reports redundant errors which could be confusing to the user. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3862) Race conditions in aggregate calculation
[ https://issues.apache.org/jira/browse/HBASE-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030211#comment-13030211 ] Hudson commented on HBASE-3862: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Race conditions in aggregate calculation > > > Key: HBASE-3862 > URL: https://issues.apache.org/jira/browse/HBASE-3862 > Project: HBase > Issue Type: Bug > Components: coprocessors >Affects Versions: 0.92.0 >Reporter: John Heitmann > Attachments: coprocessor_race.patch > > > The AggregationClient requests aggregations from multiple region servers in > parallel. The calculations in the reducer callbacks of the AggregationClient > are not thread safe, and therefore could return an incorrect result due to > simultaneous/interleaved execution. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3858) performance.xml / troubleshooting.xml - porting rest of PerformanceTuning wiki page
[ https://issues.apache.org/jira/browse/HBASE-3858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030204#comment-13030204 ] Hudson commented on HBASE-3858: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > performance.xml / troubleshooting.xml - porting rest of PerformanceTuning > wiki page > --- > > Key: HBASE-3858 > URL: https://issues.apache.org/jira/browse/HBASE-3858 > Project: HBase > Issue Type: Improvement >Reporter: Doug Meil >Assignee: Doug Meil >Priority: Minor > Attachments: performance_HBASE_3858.xml.patch, > troubleshooting_HBASE_3858.xml.patch > > > The rest of the HBase PerformanceTuning wiki page has been ported to the > HBase book. > The top-part of the page was ported into performance.xml > One thing I want to call out specifically is that the PerformanceTuning wiki > suggested to turn off WAL on puts for performance. I added a section in the > Client sub-section on this as "an option", however, I added several cautions > about the potential for data loss and that for high-volume loading > bulk-imports should be considered instead. > The bottom part (GC logging) was ported into troubleshooting.xml > After this has been applied, the PerformanceTuning wiki can be > de-commissioned. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3813) Change RPC callQueue size from "handlerCount * MAX_QUEUE_SIZE_PER_HANDLER;"
[
https://issues.apache.org/jira/browse/HBASE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030206#comment-13030206
]
Hudson commented on HBASE-3813:
---
Integrated in HBase-TRUNK #1909 (See
[https://builds.apache.org/hudson/job/HBase-TRUNK/1909/])
> Change RPC callQueue size from "handlerCount * MAX_QUEUE_SIZE_PER_HANDLER;"
> ---
>
> Key: HBASE-3813
> URL: https://issues.apache.org/jira/browse/HBASE-3813
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: stack
>Priority: Critical
> Attachments: 3813.txt
>
>
> Yesterday debugging w/ Jack we noticed that with few handlers on a big box,
> he was seeing stats like this:
> {code}
> 2011-04-21 11:54:49,451 DEBUG org.apache.hadoop.ipc.HBaseServer: Server
> connection from X.X.X.X:60931; # active connections: 11; # queued calls: 2500
> {code}
> We had 2500 items in the rpc queue waiting to be processed.
> Turns out he had too few handlers for number of clients (but also, it seems
> like he figured hw issues in that his RAM bus was running at 1/4 the rate
> that it should have been running at).
> Chatting w/ J-D this morning, he asked if the queues hold 'data'. The queues
> hold 'Calls'. Calls are the client request. They contain data.
> Jack had 2500 items queued. If each item to insert was 1MB, thats 25k * 1MB
> of memory that is outside of our generally accounting.
> Currently the queue size is handlers * MAX_QUEUE_SIZE_PER_HANDLER where
> MAX_QUEUE_SIZE_PER_HANDLER is hardcoded to be 100.
> If the queue is full we block (LinkedBlockingQueue).
> Going to change the queue size from 100 to 10 by default -- but also will
> make it configurable and will doc. this as possible cause of OOME. Will try
> it on production here before committing patch.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3835) Switch web pages to Jamon template engine instead of JSP
[ https://issues.apache.org/jira/browse/HBASE-3835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030205#comment-13030205 ] Hudson commented on HBASE-3835: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Switch web pages to Jamon template engine instead of JSP > > > Key: HBASE-3835 > URL: https://issues.apache.org/jira/browse/HBASE-3835 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.92.0 > > Attachments: hbase-3835.txt, hbase-3835.txt, hbase-3835.txt > > > Jamon (http://jamon.org) is a template engine that I think is preferable to > JSP. You can read an interview with some comparisons vs JSP here: > http://www.artima.com/lejava/articles/jamon.html > In particular, I think it will give us the following advantages: > - Since we'll have a servlet in front of each template, it will encourage us > to write less inline Java code and do more code in the servlets. > - Makes proper unit testing easier since you can trivially render a template > and pass in mock arguments without having to start a whole HTTP stack > - Static typing of template arguments makes it easier to know at compile-time > if you've made a mistake. > Thoughts? I converted the Master UI yesterday and only took a couple hours. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3866) Script to add regions gradually to a new regionserver.
[ https://issues.apache.org/jira/browse/HBASE-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravind Gottipati updated HBASE-3866: - Attachment: slow_balancer.rb > Script to add regions gradually to a new regionserver. > -- > > Key: HBASE-3866 > URL: https://issues.apache.org/jira/browse/HBASE-3866 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 0.90.2 >Reporter: Aravind Gottipati >Priority: Minor > Attachments: slow_balancer.rb, slow_balancer.rb > > > When a new region server is brought online, the current balancer kicks off a > whole bunch of region moves and causes a lot of regions to be un-available > right away. A slower balancer that gradually balances the cluster is > probably a good script to have. I have an initial version that mooches off > the region_mover script to do this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3837) Expose regionsInTransition on master UI
[ https://issues.apache.org/jira/browse/HBASE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030201#comment-13030201 ] Hudson commented on HBASE-3837: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Expose regionsInTransition on master UI > --- > > Key: HBASE-3837 > URL: https://issues.apache.org/jira/browse/HBASE-3837 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.92.0 > > Attachments: hbase-3837.txt > > > There have been some bugs in the past that can cause a region to get "stuck" > in transition. It's currently hard to see this without tailing the logs and > noticing periodic timeout messages, etc. > I'd like to expose the regionsInTransition map on the master UI, so ops can > quickly identify what might be causing a region to get "stuck". -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3855) Performance degradation of memstore because reseek is linear
[ https://issues.apache.org/jira/browse/HBASE-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030200#comment-13030200 ] Hudson commented on HBASE-3855: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Performance degradation of memstore because reseek is linear > > > Key: HBASE-3855 > URL: https://issues.apache.org/jira/browse/HBASE-3855 > Project: HBase > Issue Type: Improvement >Reporter: dhruba borthakur >Assignee: dhruba borthakur >Priority: Blocker > Fix For: 0.92.0 > > Attachments: memstoreReseek.txt, memstoreReseek2.txt > > > The scanner use reseek to find the next row (or next column) as part of a > scan. The reseek code iterates over a Set to position itself at the right > place. If there are many thousands of kvs that need to be skipped over, then > the time-cost is very high. In this case, a seek would be far lesser in cost > than a reseek. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3836) Add facility to track currently progressing actions/workflows
[ https://issues.apache.org/jira/browse/HBASE-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030199#comment-13030199 ] Hudson commented on HBASE-3836: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > Add facility to track currently progressing actions/workflows > - > > Key: HBASE-3836 > URL: https://issues.apache.org/jira/browse/HBASE-3836 > Project: HBase > Issue Type: New Feature > Components: master, regionserver >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.92.0 > > Attachments: hbase-3836.txt > > > A lot of troubleshooting involves answering the question "well, what is your > server doing right now?" Today, that involves some combination of > interpreting jstack output and/or trudging through logs. Problems with these > methods are: (a) users may not have direct ssh access to regionserver > machines in production environments, (b) logs are very verbose, so hard to > separate what's still going on vs stuff that might have completed, and (c) > interpreting jstack requires a pretty good knowledge of the codebase plus > diving into source code. > I'd like to add a singleton (for now) which takes care of tracking any major > actions going on in the region server and master. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3838) RegionCoprocesorHost.preWALRestore throws npe in case there is no RegionObserver registered.
[ https://issues.apache.org/jira/browse/HBASE-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030202#comment-13030202 ] Hudson commented on HBASE-3838: --- Integrated in HBase-TRUNK #1909 (See [https://builds.apache.org/hudson/job/HBase-TRUNK/1909/]) > RegionCoprocesorHost.preWALRestore throws npe in case there is no > RegionObserver registered. > > > Key: HBASE-3838 > URL: https://issues.apache.org/jira/browse/HBASE-3838 > Project: HBase > Issue Type: Bug >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha >Priority: Minor > Fix For: 0.92.0 > > Attachments: patch.txt > > > It seems the check to bypass the Observers chain is at wrong place in case of > pre/post WALRestore. It should be inside the "if statement" that checks > whether the CP is instance of RegionObserver or not. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3866) Script to add regions gradually to a new regionserver.
[ https://issues.apache.org/jira/browse/HBASE-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravind Gottipati updated HBASE-3866: - Attachment: slow_balancer.rb This script uses a lot of the code from region_mover.rb. The script should be invoked like this. HBASE_NOEXEC=true $HBASE_HOME/bin/hbase org.jruby.Main $HBASE_HOME/bin/slow_balancer.rb --debug -l 2 The -l option is the target difference between the server with the maximum regions and the server with the minimum regions. Once the delta reaches this point, the script exits. If -l is not passed, it defaults to the number of region servers in your environment. > Script to add regions gradually to a new regionserver. > -- > > Key: HBASE-3866 > URL: https://issues.apache.org/jira/browse/HBASE-3866 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 0.90.2 >Reporter: Aravind Gottipati >Priority: Minor > Attachments: slow_balancer.rb > > > When a new region server is brought online, the current balancer kicks off a > whole bunch of region moves and causes a lot of regions to be un-available > right away. A slower balancer that gradually balances the cluster is > probably a good script to have. I have an initial version that mooches off > the region_mover script to do this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-3866) Script to add regions gradually to a new regionserver.
Script to add regions gradually to a new regionserver. -- Key: HBASE-3866 URL: https://issues.apache.org/jira/browse/HBASE-3866 Project: HBase Issue Type: Improvement Components: scripts Affects Versions: 0.90.2 Reporter: Aravind Gottipati Priority: Minor When a new region server is brought online, the current balancer kicks off a whole bunch of region moves and causes a lot of regions to be un-available right away. A slower balancer that gradually balances the cluster is probably a good script to have. I have an initial version that mooches off the region_mover script to do this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2750) Add sanity check for system configs in hbase-daemon wrapper
[ https://issues.apache.org/jira/browse/HBASE-2750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-2750: - Labels: noob (was: ) > Add sanity check for system configs in hbase-daemon wrapper > --- > > Key: HBASE-2750 > URL: https://issues.apache.org/jira/browse/HBASE-2750 > Project: HBase > Issue Type: New Feature > Components: scripts >Affects Versions: 0.90.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Labels: noob > > We should add a config variable like MIN_ULIMIT_TO_START in hbase-env.sh. If > the daemon script finds ulimit < this value, it will print a warning and > refuse to start. We can make the default set to 0 so that this doesn't affect > non-production clusters, but in the tuning guide recommend that people change > it to the expected ulimit. > (I've seen it happen all the time where people configure ulimit on some > nodes, add a new node to the cluster, and forgot to re-tune it on the new > one, and then that new one borks the whole cluster when it joins) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3864) Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861
[
https://issues.apache.org/jira/browse/HBASE-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-3864:
-
Resolution: Fixed
Fix Version/s: 0.92.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to TRUNK. Thanks for the patch Aaron.
> Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861
> ---
>
> Key: HBASE-3864
> URL: https://issues.apache.org/jira/browse/HBASE-3864
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: 0.92.0
>
> Attachments: hbase-3864.0.patch
>
>
> HBASE-2899 renamed {{hfile.min.blocksize.size}} to
> {{hbase.mapreduce.hfileoutputformat.blocksize}}. However, HBASE-1861
> (committed after HBASE-2899) reverted this change.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-3864) Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861
[
https://issues.apache.org/jira/browse/HBASE-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reassigned HBASE-3864:
Assignee: Aaron T. Myers
> Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861
> ---
>
> Key: HBASE-3864
> URL: https://issues.apache.org/jira/browse/HBASE-3864
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Attachments: hbase-3864.0.patch
>
>
> HBASE-2899 renamed {{hfile.min.blocksize.size}} to
> {{hbase.mapreduce.hfileoutputformat.blocksize}}. However, HBASE-1861
> (committed after HBASE-2899) reverted this change.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3732) New configuration option for client-side compression
[
https://issues.apache.org/jira/browse/HBASE-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030167#comment-13030167
]
Karthick Sankarachary commented on HBASE-3732:
--
Oh, I see. I like the idea of keeping the value in a compressed form until the
client tries to "get" it. Perhaps we can compress the value depending on
whether it's fatter than a certain threshold? Also, given that the value
typically accounts for most of the {{KeyValue}}'s size, do we need to call
{{HFile#getCompressingStream}} if the value is already compressed up front?
> New configuration option for client-side compression
>
>
> Key: HBASE-3732
> URL: https://issues.apache.org/jira/browse/HBASE-3732
> Project: HBase
> Issue Type: New Feature
>Reporter: Jean-Daniel Cryans
> Fix For: 0.92.0
>
> Attachments: compressed_streams.jar
>
>
> We have a case here where we have to store very fat cells (arrays of
> integers) which can amount into the hundreds of KBs that we need to read
> often, concurrently, and possibly keep in cache. Compressing the values on
> the client using java.util.zip's Deflater before sending them to HBase proved
> to be in our case almost an order of magnitude faster.
> There reasons are evident: less data sent to hbase, memstore contains
> compressed data, block cache contains compressed data too, etc.
> I was thinking that it might be something useful to add to a family schema,
> so that Put/Result do the conversion for you. The actual compression algo
> should also be configurable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-1242) HBase Design Overview
[ https://issues.apache.org/jira/browse/HBASE-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-1242. -- Resolution: Later Marking as later. No work done on this issue in a while. > HBase Design Overview > - > > Key: HBASE-1242 > URL: https://issues.apache.org/jira/browse/HBASE-1242 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Evgeny Ryabitskiy >Assignee: Evgeny Ryabitskiy >Priority: Minor > Attachments: HBase_Arc6.jpeg > > > I'm going to make wiki page contain HBase Design overview. That will cover > - architecture of HBase, > - used solutions > - unresolved problems > It should be something like http://labs.google.com/papers/bigtable.html > focused on Implementation part. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-3861) MiniZooKeeperCluster.startup() should refer to hbase.zookeeper.property.maxClientCnxns
[
https://issues.apache.org/jira/browse/HBASE-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reopened HBASE-3861:
---
Caused TestHCM to OOME on trunk up on Jenkins. Backed out for now. Reopening to
investigate / fix.
> MiniZooKeeperCluster.startup() should refer to
> hbase.zookeeper.property.maxClientCnxns
> --
>
> Key: HBASE-3861
> URL: https://issues.apache.org/jira/browse/HBASE-3861
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Fix For: 0.90.3, 0.92.0
>
> Attachments: HBASE-3861.patch, HBASE-3861.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Currently the number of the client connections is hard-wired to 1000:
> {{{
> standaloneServerFactory = new NIOServerCnxnFactory();
> standaloneServerFactory.configure(new
> InetSocketAddress(clientPort),1000);
> } catch (BindException e) {
>
> }}}
> This should be set according to the test environment's hbase configuration.
> The property in
> question is : hbase.zookeeper.property.maxClientCnxns.
> Currently some tests such as org.apache.hadoop.hbase.client.TestHCM fail
> because the number of connections used by the HBase client exceeds 1000.
> Recently MAX_CACHED_HBASE_INSTANCES increased from 31 to 2000 on 0.90 branch:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&r1=1096818&r2=1096817&view=diff&pathrev=1096818
> and correspondingly the hbase config on the Zookeeper server-side also
> increased in hbase-default.xml:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/resources/hbase-default.xml?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&r1=1091594&r2=1091593&view=diff&pathrev=1091594
> So if MiniZKCluster looks at this setting, the test won't have this failure.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3861) MiniZooKeeperCluster.startup() should refer to hbase.zookeeper.property.maxClientCnxns
[
https://issues.apache.org/jira/browse/HBASE-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030154#comment-13030154
]
stack commented on HBASE-3861:
--
I just backed this out of TRUNK because its OOME'ing TestHCM up on hudson and
locally on two different platforms (I can't see why it would do it though...
must be some interaction w/ hbase-3777). It runs fine for me on branch.
> MiniZooKeeperCluster.startup() should refer to
> hbase.zookeeper.property.maxClientCnxns
> --
>
> Key: HBASE-3861
> URL: https://issues.apache.org/jira/browse/HBASE-3861
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Fix For: 0.90.3, 0.92.0
>
> Attachments: HBASE-3861.patch, HBASE-3861.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Currently the number of the client connections is hard-wired to 1000:
> {{{
> standaloneServerFactory = new NIOServerCnxnFactory();
> standaloneServerFactory.configure(new
> InetSocketAddress(clientPort),1000);
> } catch (BindException e) {
>
> }}}
> This should be set according to the test environment's hbase configuration.
> The property in
> question is : hbase.zookeeper.property.maxClientCnxns.
> Currently some tests such as org.apache.hadoop.hbase.client.TestHCM fail
> because the number of connections used by the HBase client exceeds 1000.
> Recently MAX_CACHED_HBASE_INSTANCES increased from 31 to 2000 on 0.90 branch:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&r1=1096818&r2=1096817&view=diff&pathrev=1096818
> and correspondingly the hbase config on the Zookeeper server-side also
> increased in hbase-default.xml:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/resources/hbase-default.xml?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&r1=1091594&r2=1091593&view=diff&pathrev=1091594
> So if MiniZKCluster looks at this setting, the test won't have this failure.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3865) Failing TestWALReplay
[ https://issues.apache.org/jira/browse/HBASE-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-3865. -- Resolution: Fixed Fix Version/s: 0.92.0 Assignee: stack Applied to TRUNK > Failing TestWALReplay > - > > Key: HBASE-3865 > URL: https://issues.apache.org/jira/browse/HBASE-3865 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 0.92.0 > > Attachments: 3865.txt > > > Failed last few times up on jenkins. The change to the signature of the > internalFlushCache to add passing of a MonitorVisitor meant the override here > was no longer called. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-3865) Failing TestWALReplay
Failing TestWALReplay - Key: HBASE-3865 URL: https://issues.apache.org/jira/browse/HBASE-3865 Project: HBase Issue Type: Bug Reporter: stack Attachments: 3865.txt Failed last few times up on jenkins. The change to the signature of the internalFlushCache to add passing of a MonitorVisitor meant the override here was no longer called. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3865) Failing TestWALReplay
[ https://issues.apache.org/jira/browse/HBASE-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3865: - Attachment: 3865.txt > Failing TestWALReplay > - > > Key: HBASE-3865 > URL: https://issues.apache.org/jira/browse/HBASE-3865 > Project: HBase > Issue Type: Bug >Reporter: stack > Attachments: 3865.txt > > > Failed last few times up on jenkins. The change to the signature of the > internalFlushCache to add passing of a MonitorVisitor meant the override here > was no longer called. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3744) createTable blocks until all regions are out of transition
[ https://issues.apache.org/jira/browse/HBASE-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030114#comment-13030114 ] stack commented on HBASE-3744: -- I applied the addendum w/ some massaging, no refactoring. Thanks for the patch Ted. Lets see if it cures the failing TestAdmin in 0.90. > createTable blocks until all regions are out of transition > -- > > Key: HBASE-3744 > URL: https://issues.apache.org/jira/browse/HBASE-3744 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.1 >Reporter: Todd Lipcon >Assignee: Ted Yu >Priority: Critical > Fix For: 0.90.3 > > Attachments: 3744-addendum-for-TestAdmin.patch, 3744-addendum.txt, > 3744-v2.txt, 3744-v3.txt, 3744.txt, create_big_tables.rb, > create_big_tables.rb, create_big_tables.rb > > > In HBASE-3305, the behavior of createTable was changed and introduced this > bug: createTable now blocks until all regions have been assigned, since it > uses BulkStartupAssigner. BulkStartupAssigner.waitUntilDone calls > assignmentManager.waitUntilNoRegionsInTransition, which waits across all > regions, not just the regions of the table that has just been created. > We saw an issue where one table had a region which was unable to be opened, > so it was stuck in RegionsInTransition permanently (every open was failing). > Since this was the case, waitUntilDone would always block indefinitely even > though the newly created table had been assigned. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3864) Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861
[
https://issues.apache.org/jira/browse/HBASE-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HBASE-3864:
--
Attachment: hbase-3864.0.patch
Patch which changes the config back from "{{hfile.min.blocksize.size}}" to
"{{hbase.mapreduce.hfileoutputformat.blocksize}}".
> Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861
> ---
>
> Key: HBASE-3864
> URL: https://issues.apache.org/jira/browse/HBASE-3864
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Aaron T. Myers
> Attachments: hbase-3864.0.patch
>
>
> HBASE-2899 renamed {{hfile.min.blocksize.size}} to
> {{hbase.mapreduce.hfileoutputformat.blocksize}}. However, HBASE-1861
> (committed after HBASE-2899) reverted this change.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3864) Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861
[
https://issues.apache.org/jira/browse/HBASE-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HBASE-3864:
--
Status: Patch Available (was: Open)
> Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861
> ---
>
> Key: HBASE-3864
> URL: https://issues.apache.org/jira/browse/HBASE-3864
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Aaron T. Myers
> Attachments: hbase-3864.0.patch
>
>
> HBASE-2899 renamed {{hfile.min.blocksize.size}} to
> {{hbase.mapreduce.hfileoutputformat.blocksize}}. However, HBASE-1861
> (committed after HBASE-2899) reverted this change.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-3864) Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861
Rename of hfile.min.blocksize.size in HBASE-2899 reverted in HBASE-1861
---
Key: HBASE-3864
URL: https://issues.apache.org/jira/browse/HBASE-3864
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: Aaron T. Myers
HBASE-2899 renamed {{hfile.min.blocksize.size}} to
{{hbase.mapreduce.hfileoutputformat.blocksize}}. However, HBASE-1861 (committed
after HBASE-2899) reverted this change.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3861) MiniZooKeeperCluster.startup() should refer to hbase.zookeeper.property.maxClientCnxns
[
https://issues.apache.org/jira/browse/HBASE-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-3861:
--
Resolution: Fixed
Fix Version/s: 0.92.0
Release Note: (was: added new patch with numClientCnxns set to 5000
(enough for TestHCM to pass).)
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to trunk and 0.90 branch. Tests pass locally.
> MiniZooKeeperCluster.startup() should refer to
> hbase.zookeeper.property.maxClientCnxns
> --
>
> Key: HBASE-3861
> URL: https://issues.apache.org/jira/browse/HBASE-3861
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Fix For: 0.90.3, 0.92.0
>
> Attachments: HBASE-3861.patch, HBASE-3861.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Currently the number of the client connections is hard-wired to 1000:
> {{{
> standaloneServerFactory = new NIOServerCnxnFactory();
> standaloneServerFactory.configure(new
> InetSocketAddress(clientPort),1000);
> } catch (BindException e) {
>
> }}}
> This should be set according to the test environment's hbase configuration.
> The property in
> question is : hbase.zookeeper.property.maxClientCnxns.
> Currently some tests such as org.apache.hadoop.hbase.client.TestHCM fail
> because the number of connections used by the HBase client exceeds 1000.
> Recently MAX_CACHED_HBASE_INSTANCES increased from 31 to 2000 on 0.90 branch:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&r1=1096818&r2=1096817&view=diff&pathrev=1096818
> and correspondingly the hbase config on the Zookeeper server-side also
> increased in hbase-default.xml:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/resources/hbase-default.xml?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&r1=1091594&r2=1091593&view=diff&pathrev=1091594
> So if MiniZKCluster looks at this setting, the test won't have this failure.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-2010) Ant task to create the test jar is inconsistent with hadoop
[ https://issues.apache.org/jira/browse/HBASE-2010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-2010. -- Resolution: Invalid Marking as invalid since move to maven (We are no longer like hadoop -- it uses ivy) > Ant task to create the test jar is inconsistent with hadoop > --- > > Key: HBASE-2010 > URL: https://issues.apache.org/jira/browse/HBASE-2010 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 0.90.0 >Reporter: Cosmin Lehene >Priority: Trivial > Fix For: 0.92.0 > > Attachments: HBASE-2010.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > In hadoop projects the jar-test task is used to compile the test jars. > In hbase this task is called compile-core-test. > Shouldn't we have a jar-test task that depends on the compile-core-test? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-2165) Improve fragmentation display and implementation
[ https://issues.apache.org/jira/browse/HBASE-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-2165. -- Resolution: Won't Fix We don't have this attribute any more. Resolving issue till it comes back. > Improve fragmentation display and implementation > > > Key: HBASE-2165 > URL: https://issues.apache.org/jira/browse/HBASE-2165 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.20.4, 0.90.0 >Reporter: Lars George >Priority: Minor > Fix For: 0.92.0 > > > Improve by > - moving the "blocking" FS scan into a thread so that the UI loads fast and > initially displays "n/a" but once it has completed the scan it displays the > proper numbers > - explaining what fragmentation means to the user (better hints or help text > in UI) > - Switch -ROOT- (and maybe even .META.?) to simply say "Yes" or a tick that > it is fragmented as it only has 0% or 100% available (since it has only a > single region) > - also computing the space occupied by each table and the total and - if > easily done - add a graph to display it (Google Pie Chart would be nice but > is an external link) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-2963) based on hadoop0.21.0
[ https://issues.apache.org/jira/browse/HBASE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-2963. -- Resolution: Duplicate Duplicate of hbase-2233 > based on hadoop0.21.0 > - > > Key: HBASE-2963 > URL: https://issues.apache.org/jira/browse/HBASE-2963 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.20.6 >Reporter: chenjiajun >Priority: Blocker > Fix For: 0.92.0 > > > I upgrade my hadoop from 0.20.2 to 0.21.0，but hbase is does't work! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-2765) Master should display currently unassigned regions in UI
[ https://issues.apache.org/jira/browse/HBASE-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HBASE-2765. Resolution: Duplicate oh hey, yea, I forgot I had filed this! Yes, fixed for 0.92 > Master should display currently unassigned regions in UI > > > Key: HBASE-2765 > URL: https://issues.apache.org/jira/browse/HBASE-2765 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.90.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > > The master often knows that a region is unassigned, but thinks it's getting > assigned. It would be very useful to display the current set of unassigned > regions in the UI, as well as their current state (eg where it's supposed to > be getting assigned, how long ago it last heard from it, etc). This would be > useful in debugging/diagnosing cluster issues (eg I just noticed that a > client had hung, took me a while to figure out that I had a region that was > repeatedly failing to open with an NPE due to HBASE-2729) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3861) MiniZooKeeperCluster.startup() should refer to hbase.zookeeper.property.maxClientCnxns
[
https://issues.apache.org/jira/browse/HBASE-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030040#comment-13030040
]
Eugene Koontz commented on HBASE-3861:
--
Thank you for looking at this, stack!! You are right about the
constructor-calling. Thank you Andrew for fixing it.
One thing which I can't explain is that TestHCM passes currently *without* this
patch. It seems like it should fail because the maxNumCnxns is set too low at
only 1000.
So I think my patch is right but it doesn't fix a test that's not passing
without it.
> MiniZooKeeperCluster.startup() should refer to
> hbase.zookeeper.property.maxClientCnxns
> --
>
> Key: HBASE-3861
> URL: https://issues.apache.org/jira/browse/HBASE-3861
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Fix For: 0.90.3
>
> Attachments: HBASE-3861.patch, HBASE-3861.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Currently the number of the client connections is hard-wired to 1000:
> {{{
> standaloneServerFactory = new NIOServerCnxnFactory();
> standaloneServerFactory.configure(new
> InetSocketAddress(clientPort),1000);
> } catch (BindException e) {
>
> }}}
> This should be set according to the test environment's hbase configuration.
> The property in
> question is : hbase.zookeeper.property.maxClientCnxns.
> Currently some tests such as org.apache.hadoop.hbase.client.TestHCM fail
> because the number of connections used by the HBase client exceeds 1000.
> Recently MAX_CACHED_HBASE_INSTANCES increased from 31 to 2000 on 0.90 branch:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&r1=1096818&r2=1096817&view=diff&pathrev=1096818
> and correspondingly the hbase config on the Zookeeper server-side also
> increased in hbase-default.xml:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/resources/hbase-default.xml?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&r1=1091594&r2=1091593&view=diff&pathrev=1091594
> So if MiniZKCluster looks at this setting, the test won't have this failure.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-2240) Balancer should not reassign (because of overloading) a region that was just opened
[ https://issues.apache.org/jira/browse/HBASE-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-2240. -- Resolution: Invalid Resolving as invalid (Jon argues this in body of this issue and stuff suggested in the issue -- upping balancer period -- has been implemented) > Balancer should not reassign (because of overloading) a region that was just > opened > --- > > Key: HBASE-2240 > URL: https://issues.apache.org/jira/browse/HBASE-2240 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack >Priority: Minor > > I'm running a mapreduce job. I see a region split and its daughters come on > line and then 8 seconds later, master judges the regionserver overloaded and > so closes the just-opened region. This messes up clients. They may have > just picked up the new location for their row query and now its moved again > and client has to go hunting anew. > We need to assign the vintage regions first. > I'm going to change balancer slop. Its not sloppy enough and balancing cuts > in too early. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-2297) Make it so can send email and have it show as comments in JIRA
[ https://issues.apache.org/jira/browse/HBASE-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-2297. -- Resolution: Fixed Assignee: stack Resolving. This works now (Though it shows as being from Hbase Build Acct --- can open new issue for that). > Make it so can send email and have it show as comments in JIRA > -- > > Key: HBASE-2297 > URL: https://issues.apache.org/jira/browse/HBASE-2297 > Project: HBase > Issue Type: Task >Reporter: stack >Assignee: stack > > Some discussion up in mailing list suggests it may be just a matter of > removing the 'Reply-To' which has the list in it. I filed INFRA-2533. I > also tested replying to an issue emission and changing the to to be > j...@apache.org instead and my comment went into the issue as a comment. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3861) MiniZooKeeperCluster.startup() should refer to hbase.zookeeper.property.maxClientCnxns
[
https://issues.apache.org/jira/browse/HBASE-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030036#comment-13030036
]
Hbase Build Acct commented on HBASE-3861:
-
Sounds good
> MiniZooKeeperCluster.startup() should refer to
> hbase.zookeeper.property.maxClientCnxns
> --
>
> Key: HBASE-3861
> URL: https://issues.apache.org/jira/browse/HBASE-3861
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Fix For: 0.90.3
>
> Attachments: HBASE-3861.patch, HBASE-3861.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Currently the number of the client connections is hard-wired to 1000:
> {{{
> standaloneServerFactory = new NIOServerCnxnFactory();
> standaloneServerFactory.configure(new
> InetSocketAddress(clientPort),1000);
> } catch (BindException e) {
>
> }}}
> This should be set according to the test environment's hbase configuration.
> The property in
> question is : hbase.zookeeper.property.maxClientCnxns.
> Currently some tests such as org.apache.hadoop.hbase.client.TestHCM fail
> because the number of connections used by the HBase client exceeds 1000.
> Recently MAX_CACHED_HBASE_INSTANCES increased from 31 to 2000 on 0.90 branch:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&r1=1096818&r2=1096817&view=diff&pathrev=1096818
> and correspondingly the hbase config on the Zookeeper server-side also
> increased in hbase-default.xml:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/resources/hbase-default.xml?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&r1=1091594&r2=1091593&view=diff&pathrev=1091594
> So if MiniZKCluster looks at this setting, the test won't have this failure.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3861) MiniZooKeeperCluster.startup() should refer to hbase.zookeeper.property.maxClientCnxns
[
https://issues.apache.org/jira/browse/HBASE-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030029#comment-13030029
]
Andrew Purtell commented on HBASE-3861:
---
+1, I'll commit with Stack's comment addressed also.
> MiniZooKeeperCluster.startup() should refer to
> hbase.zookeeper.property.maxClientCnxns
> --
>
> Key: HBASE-3861
> URL: https://issues.apache.org/jira/browse/HBASE-3861
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.90.3
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
> Fix For: 0.90.3
>
> Attachments: HBASE-3861.patch, HBASE-3861.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Currently the number of the client connections is hard-wired to 1000:
> {{{
> standaloneServerFactory = new NIOServerCnxnFactory();
> standaloneServerFactory.configure(new
> InetSocketAddress(clientPort),1000);
> } catch (BindException e) {
>
> }}}
> This should be set according to the test environment's hbase configuration.
> The property in
> question is : hbase.zookeeper.property.maxClientCnxns.
> Currently some tests such as org.apache.hadoop.hbase.client.TestHCM fail
> because the number of connections used by the HBase client exceeds 1000.
> Recently MAX_CACHED_HBASE_INSTANCES increased from 31 to 2000 on 0.90 branch:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fclient%2FHConnectionManager.java&r1=1096818&r2=1096817&view=diff&pathrev=1096818
> and correspondingly the hbase config on the Zookeeper server-side also
> increased in hbase-default.xml:
> http://svn.apache.org/viewvc/hbase/branches/0.90/src/main/resources/hbase-default.xml?p2=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&p1=%2Fhbase%2Fbranches%2F0.90%2Fsrc%2Fmain%2Fresources%2Fhbase-default.xml&r1=1091594&r2=1091593&view=diff&pathrev=1091594
> So if MiniZKCluster looks at this setting, the test won't have this failure.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-2465) HMaster should not contact each RS on startup
[ https://issues.apache.org/jira/browse/HBASE-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-2465. -- Resolution: Won't Fix We don't have a verifyClusterState any more and I don't think we contact each regionserver on startup any more (Reopen if I have this wrong Todd). > HMaster should not contact each RS on startup > - > > Key: HBASE-2465 > URL: https://issues.apache.org/jira/browse/HBASE-2465 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Todd Lipcon > > On startup, in verifyClusterState, the master contacts each region server > serially. If a region server is down it will retry for several minutes (if > the client retry setting is high). During this period, the master cannot be > shut down, and also isn't processing real work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-2646) Compaction requests should be prioritized to prevent blocking
[
https://issues.apache.org/jira/browse/HBASE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-2646.
--
Resolution: Fixed
Fix Version/s: 0.90.0
Assignee: Jeff Whiting
Hadoop Flags: [Reviewed]
This was applied a while back. Resolving. Thanks for the patch Jeff (Assigned
it to you).
> Compaction requests should be prioritized to prevent blocking
> -
>
> Key: HBASE-2646
> URL: https://issues.apache.org/jira/browse/HBASE-2646
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 0.20.4
> Environment: ubuntu server 10; hbase 0.20.4; 4 machine cluster (each
> machine is an 8 core xeon with 16 GB of ram and 6TB of storage); ~250 Million
> rows;
>Reporter: Jeff Whiting
>Assignee: Jeff Whiting
>Priority: Critical
> Labels: compaction, split
> Fix For: 0.90.0
>
> Attachments: 2646-fix-race-condition-r1004349.txt, 2646-v2.txt,
> 2646-v3.txt, PriorityQueue-r996664.patch, prioritycompactionqueue-0.20.4.patch
>
>
> While testing the write capacity of a 4 machine hbase cluster we were getting
> long and frequent client pauses as we attempted to load the data. Looking
> into the problem we'd get a relatively large compaction queue and when a
> region hit the "hbase.hstore.blockingStoreFiles" limit it would get block the
> client and the compaction request would get put on the back of the queue
> waiting for many other less important compactions. The client is basically
> stuck at that point until a compaction is done. Prioritizing the compaction
> requests and allowing the request that is blocking other actions go first
> would help solve the problem.
> You can see the problem by looking at our log files:
> You'll first see an event such as a too many HLog which will put a lot of
> requests on the compaction queue.
> {noformat}
> 2010-05-25 10:53:26,570 INFO org.apache.hadoop.hbase.regionserver.HLog: Too
> many hlogs: logs=33, maxlogs=32; forcing flush of 22 regions(s):
> responseCounts,RS_6eZzLtdwhGiTwHy,1274232223324,
> responses,RS_0qhkL5rUmPCbx3K-1274213057242,1274513189592,
> responses,RS_1ANYnTegjzVIsHW-12742177419
> 21,1274511001873, responses,RS_1HQ4UG5BdOlAyuE-1274216757425,1274726323747,
> responses,RS_1Y7SbqSTsZrYe7a-1274328697838,1274478031930,
> responses,RS_1ZH5TB5OdW4BVLm-1274216239894,1274538267659,
> responses,RS_3BHc4KyoM3q72Yc-1274290546987,1274502062319,
> responses,RS_3ra9BaBMAXFAvbK-127421457
> 9958,1274381552543, responses,RS_6SDrGNuyyLd3oR6-1274219941155,1274385453586,
> responses,RS_8AGCEMWbI6mZuoQ-1274306857429,1274319602718,
> responses,RS_8C8T9DN47uwTG1S-1274215381765,1274289112817,
> responses,RS_8J5wmdmKmJXzK6g-1274299593861,1274494738952,
> responses,RS_8e5Sz0HeFPAdb6c-1274288
> 641459,1274495868557,
> responses,RS_8rjcnmBXPKzI896-1274306981684,1274403047940,
> responses,RS_9FS3VedcyrF0KX2-1274245971331,1274754745013,
> responses,RS_9oZgPtxO31npv3C-1274214027769,1274396489756,
> responses,RS_a3FdO2jhqWuy37C-1274209228660,1274399508186,
> responses,RS_a3LJVxwTj29MHVa-12742
> {noformat}
> Then you see the too many log files:
> {noformat}
> 2010-05-25 10:53:31,364 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested
> for region
> responses-index,--1274799047787--R_cBKrGxx0FdWjPso,1274804575862/783020138
> because: regionserver/192.168.0.81:60020.cacheFlusher
> 2010-05-25 10:53:32,364 WARN
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Region
> responses-index,--1274799047787--R_cBKrGxx0FdWjPso,1274804575862 has too many
> store files, putting it back at the end of the flush queue.
> {noformat}
> Which leads to this:
> {noformat}
> 2010-05-25 10:53:27,061 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Blocking updates for 'IPC Server handler 60 on 60020' on region
> responses-index,--1274799047787--R_cBKrGxx0FdWjPso,1274804575862: memstore
> size 128.0m is >= than blocking 128.0m size
> 2010-05-25 10:53:27,061 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Blocking updates for 'IPC Server handler 84 on 60020' on region
> responses-index,--1274799047787--R_cBKrGxx0FdWjPso,1274804575862: memstore
> size 128.0m is >= than blocking 128.0m size
> 2010-05-25 10:53:27,065 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Blocking updates for 'IPC Server handler 1 on 60020' on region
> responses-index,--1274799047787--R_cBKrGxx0FdWjPso,1274804575862: memstore
> size 128.0m is >= than blocking 128.0m size
> {noformat}
> Once the compaction / split is done a flush is able to happen which unblocks
> the IPC allowing writes to continue. Unfortunately this process can take
> upwards of 15+ minutes (the specific case shown here from our logs took about
> 4 min
[jira] [Updated] (HBASE-2631) Decide between "InMB" and "MB" as suffix for field names in ClusterStatus objects
[ https://issues.apache.org/jira/browse/HBASE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-2631: - Labels: noob (was: ) > Decide between "InMB" and "MB" as suffix for field names in ClusterStatus > objects > - > > Key: HBASE-2631 > URL: https://issues.apache.org/jira/browse/HBASE-2631 > Project: HBase > Issue Type: Improvement >Reporter: Jeff Hammerbacher > Labels: noob > > We vacillate between "InMB" (e.g. HServerLoad.getMemStoreSizeInMB(), > HServerLoad.getStorefileIndexSizeInMB()) and "MB" (e.g. > HServerLoad.getUsedHeapMB()) in the various ClusterStatus objects > (HServerInfo, HServerLoad, HServerLoad.RegionLoad). We should probably pick > one and stick to it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2675) Quick "smoke tests" testsuite
[
https://issues.apache.org/jira/browse/HBASE-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030021#comment-13030021
]
stack commented on HBASE-2675:
--
Is the following enough?
{code}
mvn clean test -Dtest=TestFromClientSide
{code}
It excercises the UI from the client side -- it starts up a minicluster --
doing about 40 odd little tests.
> Quick "smoke tests" testsuite
> -
>
> Key: HBASE-2675
> URL: https://issues.apache.org/jira/browse/HBASE-2675
> Project: HBase
> Issue Type: Test
>Reporter: Benoit Sigoure
>Priority: Minor
>
> It would be nice if there was a known subset of the tests that run fast (e.g.
> not more than a few seconds) and quickly help us check whether the code isn't
> horribly broken. This way one could run those tests at a frequent interval
> when iterating and only run the entire testsuite at the end, when they think
> they're done, since doing so is very time consuming.
> Someone would need to identify which tests really focus on the core
> functionality and add a target in the build system to just run those tests.
> As a bonus, it would be awesome++ if the core tests ran, say, 10x faster than
> they currently do. There's a lot of "sleep"-based "synchronization" in the
> tests and it would be nice to remove some of that where possible to make the
> tests run as fast as the machine can handle them.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-2765) Master should display currently unassigned regions in UI
[ https://issues.apache.org/jira/browse/HBASE-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reassigned HBASE-2765: Assignee: Todd Lipcon You have fixed this, haven't you Todd? I see RIT in master UI now. And we have the hbck tool. Can we close this issue out? > Master should display currently unassigned regions in UI > > > Key: HBASE-2765 > URL: https://issues.apache.org/jira/browse/HBASE-2765 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 0.90.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > > The master often knows that a region is unassigned, but thinks it's getting > assigned. It would be very useful to display the current set of unassigned > regions in the UI, as well as their current state (eg where it's supposed to > be getting assigned, how long ago it last heard from it, etc). This would be > useful in debugging/diagnosing cluster issues (eg I just noticed that a > client had hung, took me a while to figure out that I had a region that was > repeatedly failing to open with an NPE due to HBASE-2729) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme
[ https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3417: - Status: Patch Available (was: Open) This patch did not go in. Want me to massage it in? > CacheOnWrite is using the temporary output path for block names, need to use > a more consistent block naming scheme > -- > > Key: HBASE-3417 > URL: https://issues.apache.org/jira/browse/HBASE-3417 > Project: HBase > Issue Type: Bug > Components: io, regionserver >Affects Versions: 0.92.0 >Reporter: Jonathan Gray >Assignee: Jonathan Gray >Priority: Critical > Fix For: 0.92.0 > > Attachments: HBASE-3417-v1.patch, HBASE-3417-v2.patch, > HBASE-3417-v5.patch > > > Currently the block names used in the block cache are built using the > filesystem path. However, for cache on write, the path is a temporary output > file. > The original COW patch actually made some modifications to block naming stuff > to make it more consistent but did not do enough. Should add a separate > method somewhere for generating block names using some more easily mocked > scheme (rather than just raw path as we generate a random unique file name > twice, once for tmp and then again when moved into place). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3850) Log more details when a scanner lease expires
[
https://issues.apache.org/jira/browse/HBASE-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-3850:
-
Priority: Critical (was: Minor)
Tags: noob
> Log more details when a scanner lease expires
> -
>
> Key: HBASE-3850
> URL: https://issues.apache.org/jira/browse/HBASE-3850
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Benoit Sigoure
>Priority: Critical
>
> The message logged by the RegionServer when a Scanner lease expires isn't as
> useful as it could be. {{Scanner 4765412385779771089 lease expired}} - most
> clients don't log their scanner ID, so it's really hard to figure out what
> was going on. I think it would be useful to at least log the name of the
> region on which the Scanner was open, and it would be great to have the
> ip:port of the client that had that lease too.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3855) Performance degradation of memstore because reseek is linear
[ https://issues.apache.org/jira/browse/HBASE-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3855: - Resolution: Fixed Fix Version/s: (was: 0.94.0) 0.92.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to TRUNK. Thanks for the patch Dhruba. > Performance degradation of memstore because reseek is linear > > > Key: HBASE-3855 > URL: https://issues.apache.org/jira/browse/HBASE-3855 > Project: HBase > Issue Type: Improvement >Reporter: dhruba borthakur >Assignee: dhruba borthakur >Priority: Blocker > Fix For: 0.92.0 > > Attachments: memstoreReseek.txt, memstoreReseek2.txt > > > The scanner use reseek to find the next row (or next column) as part of a > scan. The reseek code iterates over a Set to position itself at the right > place. If there are many thousands of kvs that need to be skipped over, then > the time-cost is very high. In this case, a seek would be far lesser in cost > than a reseek. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3691) Add compressor support for 'snappy', google's compressor
[ https://issues.apache.org/jira/browse/HBASE-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nichole Treadway updated HBASE-3691: Attachment: (was: hbase-snappy-3691-trunk-003.patch) > Add compressor support for 'snappy', google's compressor > > > Key: HBASE-3691 > URL: https://issues.apache.org/jira/browse/HBASE-3691 > Project: HBase > Issue Type: Task >Reporter: stack >Priority: Critical > Fix For: 0.92.0 > > Attachments: hbase-snappy-3691-trunk-002.patch, > hbase-snappy-3691-trunk.patch > > > http://code.google.com/p/snappy/ is apache licensed. > bq. Snappy is a compression/decompression library. It does not aim for > maximum compression, or compatibility with any other compression library; > instead, it aims for very high speeds and reasonable compression. For > instance, compared to the fastest mode of zlib, Snappy is an order of > magnitude faster for most inputs, but the resulting compressed files are > anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in > 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses > at about 500 MB/sec or more. > bq. Snappy is widely used inside Google, in everything from BigTable and > MapReduce to our internal RPC systems. (Snappy has previously been referred > to as "Zippy" in some presentations and the likes.) > Lets get it in. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3691) Add compressor support for 'snappy', google's compressor
[ https://issues.apache.org/jira/browse/HBASE-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nichole Treadway updated HBASE-3691: Attachment: hbase-snappy-3691-trunk-003.patch Accidentally selected wrong license option. > Add compressor support for 'snappy', google's compressor > > > Key: HBASE-3691 > URL: https://issues.apache.org/jira/browse/HBASE-3691 > Project: HBase > Issue Type: Task >Reporter: stack >Priority: Critical > Fix For: 0.92.0 > > Attachments: hbase-snappy-3691-trunk-002.patch, > hbase-snappy-3691-trunk-003.patch, hbase-snappy-3691-trunk.patch > > > http://code.google.com/p/snappy/ is apache licensed. > bq. Snappy is a compression/decompression library. It does not aim for > maximum compression, or compatibility with any other compression library; > instead, it aims for very high speeds and reasonable compression. For > instance, compared to the fastest mode of zlib, Snappy is an order of > magnitude faster for most inputs, but the resulting compressed files are > anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in > 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses > at about 500 MB/sec or more. > bq. Snappy is widely used inside Google, in everything from BigTable and > MapReduce to our internal RPC systems. (Snappy has previously been referred > to as "Zippy" in some presentations and the likes.) > Lets get it in. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3691) Add compressor support for 'snappy', google's compressor
[ https://issues.apache.org/jira/browse/HBASE-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nichole Treadway updated HBASE-3691: Attachment: hbase-snappy-3691-trunk-003.patch Thanks for the patch...I made a few additional changes in HColumnDescriptor, and I updated the test files to include snappy. I noticed there are places in the hbase.avro classes where snappy support would need to be added in. Is it ok to add these changes in the patch, or do the avro classes need to be auto-generated somehow? > Add compressor support for 'snappy', google's compressor > > > Key: HBASE-3691 > URL: https://issues.apache.org/jira/browse/HBASE-3691 > Project: HBase > Issue Type: Task >Reporter: stack >Priority: Critical > Fix For: 0.92.0 > > Attachments: hbase-snappy-3691-trunk-002.patch, > hbase-snappy-3691-trunk.patch > > > http://code.google.com/p/snappy/ is apache licensed. > bq. Snappy is a compression/decompression library. It does not aim for > maximum compression, or compatibility with any other compression library; > instead, it aims for very high speeds and reasonable compression. For > instance, compared to the fastest mode of zlib, Snappy is an order of > magnitude faster for most inputs, but the resulting compressed files are > anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in > 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses > at about 500 MB/sec or more. > bq. Snappy is widely used inside Google, in everything from BigTable and > MapReduce to our internal RPC systems. (Snappy has previously been referred > to as "Zippy" in some presentations and the likes.) > Lets get it in. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3855) Performance degradation of memstore because reseek is linear
[ https://issues.apache.org/jira/browse/HBASE-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HBASE-3855: Attachment: memstoreReseek2.txt A minor improvement from the previous patch 1. invoke seek only if the getNext()call returned null and numIterReseek is zero. > Performance degradation of memstore because reseek is linear > > > Key: HBASE-3855 > URL: https://issues.apache.org/jira/browse/HBASE-3855 > Project: HBase > Issue Type: Improvement >Reporter: dhruba borthakur >Assignee: dhruba borthakur >Priority: Blocker > Fix For: 0.94.0 > > Attachments: memstoreReseek.txt, memstoreReseek2.txt > > > The scanner use reseek to find the next row (or next column) as part of a > scan. The reseek code iterates over a Set to position itself at the right > place. If there are many thousands of kvs that need to be skipped over, then > the time-cost is very high. In this case, a seek would be far lesser in cost > than a reseek. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
