[jira] [Created] (HBASE-11598) Add simple rpc throttling
Matteo Bertozzi created HBASE-11598: --- Summary: Add simple rpc throttling Key: HBASE-11598 URL: https://issues.apache.org/jira/browse/HBASE-11598 Project: HBase Issue Type: New Feature Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 1.0.0, 2.0.0 Add a simple version of rpc throttling. (by simple I mean something that requires less changes as possible to the core) The idea is to add a hbase:quota table to store the user/table quota information. Add a couple of API on the client like throttleUser() and throttleTable() and on the server side before executing the request we check the quota, if not an exception is thrown. The quota will be per-machine. There will be a flag QuotaScope that will be used in the future to specify the quota at cluster level instead of per machine. (A limit of 100req/min means that each machine can execute 100req/min with a scope per-machine). This will be the first cut, simple solution that requires verify few changes to the core. Later on we can make the client aware of the ThrottlingException and deal with it in a smarter way. Also we need to change a bit the RPC code to be able to yield the operation if the quota will be available not to far in the future, and avoid going back to the client for few seconds. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11599) BloomFilterFactory.IO_STOREFILE_DELETEFAMILY_BLOOM_ENABLED is never used
Jean-Marc Spaggiari created HBASE-11599: --- Summary: BloomFilterFactory.IO_STOREFILE_DELETEFAMILY_BLOOM_ENABLED is never used Key: HBASE-11599 URL: https://issues.apache.org/jira/browse/HBASE-11599 Project: HBase Issue Type: Bug Affects Versions: 0.98.4, 0.94.21, 1.0.0, 2.0.0 Reporter: Jean-Marc Spaggiari Priority: Minor BloomFilterFactory.IO_STOREFILE_DELETEFAMILY_BLOOM_ENABLED is not used anywhere. I think we should clean it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11322) SnapshotHFileCleaner makes the wrong check for lastModified time thus causing too many cache refreshes
[
https://issues.apache.org/jira/browse/HBASE-11322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl resolved HBASE-11322.
---
Resolution: Duplicate
Fix Version/s: (was: 0.94.22)
(was: 0.98.5)
(was: 0.99.0)
Marking as dup of HBASE-11360, which would this issue as well.
SnapshotHFileCleaner makes the wrong check for lastModified time thus causing
too many cache refreshes
--
Key: HBASE-11322
URL: https://issues.apache.org/jira/browse/HBASE-11322
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.19
Reporter: churro morales
Assignee: churro morales
Priority: Critical
Attachments: 11322.94.txt, HBASE-11322.patch
The SnapshotHFileCleaner calls the SnapshotFileCache if a particular HFile in
question is part of a snapshot.
If the HFile is not in the cache, we then refresh the cache and check again.
But the cache refresh checks to see if anything has been modified since the
last cache refresh but this logic is incorrect in certain scenarios.
The last modified time is done via this operation:
{code}
this.lastModifiedTime = Math.min(dirStatus.getModificationTime(),
tempStatus.getModificationTime());
{code}
and the check to see if the snapshot directories have been modified:
{code}
// if the snapshot directory wasn't modified since we last check, we are done
if (dirStatus.getModificationTime() = lastModifiedTime
tempStatus.getModificationTime() = lastModifiedTime) {
return;
}
{code}
Suppose the following happens:
dirStatus modified 6-1-2014
tempStatus modified 6-2-2014
lastModifiedTime = 6-1-2014
provided these two directories don't get modified again all subsequent checks
wont exit early, like they should.
In our cluster, this was a huge performance hit. The cleaner chain fell
behind, thus almost filling up dfs and our namenode heap.
Its a simple fix, instead of Math.min we use Math.max for the lastModified, I
believe that will be correct.
I'll apply a patch for you guys.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-11360) SnapshotFileCache causes too many cache refreshes
[ https://issues.apache.org/jira/browse/HBASE-11360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-11360. --- Resolution: Fixed Fix Version/s: (was: 0.99.0) Committed to 0.94 and 0.98. Holding off on 0.96 right. Not committed to trunk as it is not needed there. SnapshotFileCache causes too many cache refreshes - Key: HBASE-11360 URL: https://issues.apache.org/jira/browse/HBASE-11360 Project: HBase Issue Type: Bug Affects Versions: 0.94.19 Reporter: churro morales Assignee: churro morales Fix For: 0.98.5, 0.94.22 Attachments: 11360-0.98.txt, HBASE-11360-0.94.patch, HBASE-11360-0.96.patch Right now we decide whether to refresh the cache based on the lastModified timestamp of all the snapshots and those running snapshots which is located in the /hbase/.hbase-snapshot/.tmp/snapshot directory We ran a ExportSnapshot job which takes around 7 minutes between creating the directory and copying all the files. Thus the modified time for the /hbase/.hbase-snapshot/.tmp directory was 7 minutes earlier than the modified time of the /hbase/.hbase-snapshot/.tmp/snapshot directory Thus the cache refresh happens and doesn't pick up all the files but thinks its up to date as the modified time of the .tmp directory never changes. This is a bug as when the export job starts the cache never contains the files for the running snapshot and will fail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11600) DataInputputStream and DoubleOutputStream are no longer being used
Shengzhe Yao created HBASE-11600: Summary: DataInputputStream and DoubleOutputStream are no longer being used Key: HBASE-11600 URL: https://issues.apache.org/jira/browse/HBASE-11600 Project: HBase Issue Type: Task Components: io Reporter: Shengzhe Yao Assignee: Shengzhe Yao Priority: Trivial Fix For: 2.0.0 hbase-server/src/main/java/org/apache/hadoop/hbase/io/DataInputInputStream.java and hbase-server/src/main/java/org/apache/hadoop/hbase/io/DoubleOutputStream.java seems no longer being used. -- This message was sent by Atlassian JIRA (v6.2#6252)
How to validate HBase configuration (for region servers) from HBase admin page?
Hi all, We have set up an HBase cluster with 10 region nodes. We modified the default configurations to handle bucket cache in the hbase-site.xml (for all region servers) as follows: property namehbase.regionserver.handler.count/name value40/value /property property namehbase.bucketcache.ioengine/name valueoffheap/value /property property namehbase.bucketcache.percentage.in.combinedcache/name value0.8/value /property property namehbase.bucketcache.size/name value4/value /property However, when we access the HBase configuration page on the HBase Web admin, we are not able to see the updated configurations, even for the ones not related to bucket cache: propertynamehbase.regionserver.handler.count/namevalue*30*/valuesourcehbase-site.xml/source/property Can you give us a hint on why the configurations are not seen in the HBase configuration site, and how we can check out the HBase configuration for all region servers? Thanks, Gaby
Re: How to validate HBase configuration (for region servers) from HBase admin page?
bq. sourcehbase-site.xml/source Can you check the hbase-site.xml which is in classpath of the region server ? What release of hbase are you using ? Cheers On Mon, Jul 28, 2014 at 11:28 AM, gabriela montiel gabriela.mont...@oracle.com wrote: Hi all, We have set up an HBase cluster with 10 region nodes. We modified the default configurations to handle bucket cache in the hbase-site.xml (for all region servers) as follows: property namehbase.regionserver.handler.count/name value40/value /property property namehbase.bucketcache.ioengine/name valueoffheap/value /property property namehbase.bucketcache.percentage.in.combinedcache/name value0.8/value /property property namehbase.bucketcache.size/name value4/value /property However, when we access the HBase configuration page on the HBase Web admin, we are not able to see the updated configurations, even for the ones not related to bucket cache: propertynamehbase.regionserver.handler.count/ namevalue*30*/valuesourcehbase-site.xml/source/property Can you give us a hint on why the configurations are not seen in the HBase configuration site, and how we can check out the HBase configuration for all region servers? Thanks, Gaby
Re: How to validate HBase configuration (for region servers) from HBase admin page?
Hi Gaby, Did you restart your regionservers ? If not, HBase won't pick up the up-to-date configuration. Best Shengzhe On Mon, Jul 28, 2014 at 11:38 AM, Ted Yu yuzhih...@gmail.com wrote: bq. sourcehbase-site.xml/source Can you check the hbase-site.xml which is in classpath of the region server ? What release of hbase are you using ? Cheers On Mon, Jul 28, 2014 at 11:28 AM, gabriela montiel gabriela.mont...@oracle.com wrote: Hi all, We have set up an HBase cluster with 10 region nodes. We modified the default configurations to handle bucket cache in the hbase-site.xml (for all region servers) as follows: property namehbase.regionserver.handler.count/name value40/value /property property namehbase.bucketcache.ioengine/name valueoffheap/value /property property namehbase.bucketcache.percentage.in.combinedcache/name value0.8/value /property property namehbase.bucketcache.size/name value4/value /property However, when we access the HBase configuration page on the HBase Web admin, we are not able to see the updated configurations, even for the ones not related to bucket cache: propertynamehbase.regionserver.handler.count/ namevalue*30*/valuesourcehbase-site.xml/source/property Can you give us a hint on why the configurations are not seen in the HBase configuration site, and how we can check out the HBase configuration for all region servers? Thanks, Gaby
Re: How to validate HBase configuration (for region servers) from HBase admin page?
Hi Shengzhe, We executed a rolling restart on HBase through Cloudera 5, but after completing the site retrieved the same configurations. Thanks, Gaby Sent from my Samsung Mobile Original message From: yao yaosheng...@gmail.com Date:28/07/2014 13:43 (GMT-06:00) To: dev@hbase.apache.org Subject: Re: How to validate HBase configuration (for region servers) from HBase admin page? Hi Gaby, Did you restart your regionservers ? If not, HBase won't pick up the up-to-date configuration. Best Shengzhe On Mon, Jul 28, 2014 at 11:38 AM, Ted Yu yuzhih...@gmail.com wrote: bq. sourcehbase-site.xml/source Can you check the hbase-site.xml which is in classpath of the region server ? What release of hbase are you using ? Cheers On Mon, Jul 28, 2014 at 11:28 AM, gabriela montiel gabriela.mont...@oracle.com wrote: Hi all, We have set up an HBase cluster with 10 region nodes. We modified the default configurations to handle bucket cache in the hbase-site.xml (for all region servers) as follows: property namehbase.regionserver.handler.count/name value40/value /property property namehbase.bucketcache.ioengine/name valueoffheap/value /property property namehbase.bucketcache.percentage.in.combinedcache/name value0.8/value /property property namehbase.bucketcache.size/name value4/value /property However, when we access the HBase configuration page on the HBase Web admin, we are not able to see the updated configurations, even for the ones not related to bucket cache: propertynamehbase.regionserver.handler.count/ namevalue*30*/valuesourcehbase-site.xml/source/property Can you give us a hint on why the configurations are not seen in the HBase configuration site, and how we can check out the HBase configuration for all region servers? Thanks, Gaby
Re: How to validate HBase configuration (for region servers) from HBase admin page?
Probably a question for the cloudera hbase list? (Did you enter the block cache configs via CM safety valve mechanism?) Yours, St.Ack On Mon, Jul 28, 2014 at 11:54 AM, Gabriela Montiel gabriela.mont...@oracle.com wrote: Hi Shengzhe, We executed a rolling restart on HBase through Cloudera 5, but after completing the site retrieved the same configurations. Thanks, Gaby Sent from my Samsung Mobile Original message From: yao yaosheng...@gmail.com Date:28/07/2014 13:43 (GMT-06:00) To: dev@hbase.apache.org Subject: Re: How to validate HBase configuration (for region servers) from HBase admin page? Hi Gaby, Did you restart your regionservers ? If not, HBase won't pick up the up-to-date configuration. Best Shengzhe On Mon, Jul 28, 2014 at 11:38 AM, Ted Yu yuzhih...@gmail.com wrote: bq. sourcehbase-site.xml/source Can you check the hbase-site.xml which is in classpath of the region server ? What release of hbase are you using ? Cheers On Mon, Jul 28, 2014 at 11:28 AM, gabriela montiel gabriela.mont...@oracle.com wrote: Hi all, We have set up an HBase cluster with 10 region nodes. We modified the default configurations to handle bucket cache in the hbase-site.xml (for all region servers) as follows: property namehbase.regionserver.handler.count/name value40/value /property property namehbase.bucketcache.ioengine/name valueoffheap/value /property property namehbase.bucketcache.percentage.in.combinedcache/name value0.8/value /property property namehbase.bucketcache.size/name value4/value /property However, when we access the HBase configuration page on the HBase Web admin, we are not able to see the updated configurations, even for the ones not related to bucket cache: propertynamehbase.regionserver.handler.count/ namevalue*30*/valuesourcehbase-site.xml/source/property Can you give us a hint on why the configurations are not seen in the HBase configuration site, and how we can check out the HBase configuration for all region servers? Thanks, Gaby
Re: How to validate HBase configuration (for region servers) from HBase admin page?
And don't forget to set -XX:MaxDirectMemorySize (from 9.6.4.5.1. Enable SlabCache in the refguide -- http://hbase.apache.org/book.html). This you set where you pass your server JVM args. St.Ack On Mon, Jul 28, 2014 at 12:21 PM, Stack st...@duboce.net wrote: Probably a question for the cloudera hbase list? (Did you enter the block cache configs via CM safety valve mechanism?) Yours, St.Ack On Mon, Jul 28, 2014 at 11:54 AM, Gabriela Montiel gabriela.mont...@oracle.com wrote: Hi Shengzhe, We executed a rolling restart on HBase through Cloudera 5, but after completing the site retrieved the same configurations. Thanks, Gaby Sent from my Samsung Mobile Original message From: yao yaosheng...@gmail.com Date:28/07/2014 13:43 (GMT-06:00) To: dev@hbase.apache.org Subject: Re: How to validate HBase configuration (for region servers) from HBase admin page? Hi Gaby, Did you restart your regionservers ? If not, HBase won't pick up the up-to-date configuration. Best Shengzhe On Mon, Jul 28, 2014 at 11:38 AM, Ted Yu yuzhih...@gmail.com wrote: bq. sourcehbase-site.xml/source Can you check the hbase-site.xml which is in classpath of the region server ? What release of hbase are you using ? Cheers On Mon, Jul 28, 2014 at 11:28 AM, gabriela montiel gabriela.mont...@oracle.com wrote: Hi all, We have set up an HBase cluster with 10 region nodes. We modified the default configurations to handle bucket cache in the hbase-site.xml (for all region servers) as follows: property namehbase.regionserver.handler.count/name value40/value /property property namehbase.bucketcache.ioengine/name valueoffheap/value /property property namehbase.bucketcache.percentage.in.combinedcache/name value0.8/value /property property namehbase.bucketcache.size/name value4/value /property However, when we access the HBase configuration page on the HBase Web admin, we are not able to see the updated configurations, even for the ones not related to bucket cache: propertynamehbase.regionserver.handler.count/ namevalue*30*/valuesourcehbase-site.xml/source/property Can you give us a hint on why the configurations are not seen in the HBase configuration site, and how we can check out the HBase configuration for all region servers? Thanks, Gaby
[jira] [Created] (HBASE-11601) Parallelize Snapshot operations for 0.94
churro morales created HBASE-11601: -- Summary: Parallelize Snapshot operations for 0.94 Key: HBASE-11601 URL: https://issues.apache.org/jira/browse/HBASE-11601 Project: HBase Issue Type: Improvement Affects Versions: 0.94.21 Reporter: churro morales Although HBASE-11185 exists, it is geared towards the snapshot manifest code. We have used snapshots to ship our two largest tables across the country and while doing so found a few potential optimizations where doing things in parallel helped quite a bit. I can attach a patch containing changes I've made and we can discuss if these are changes worth getting pushed to 0.94. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11602) Kill itself when election znode is deleted
Yi Deng created HBASE-11602: --- Summary: Kill itself when election znode is deleted Key: HBASE-11602 URL: https://issues.apache.org/jira/browse/HBASE-11602 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.89-fb Reporter: Yi Deng Priority: Minor Fix For: 0.89-fb In case the election znode is deleted (manually for example), the active master should kill itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: DISCUSSION: 1.0.0
Did anyone get a chance to take a look at the patches? Regards, aditya... On Sun, Jul 20, 2014 at 8:37 PM, Aditya adityakish...@gmail.com wrote: The wrapper jar is part of the first patch, which is in git mailbox patch format. On Sat, Jul 19, 2014 at 2:03 AM, Ted Yu yuzhih...@gmail.com wrote: You may want to attach the wrapper jar to the JIRA directly. Cheers On Jul 19, 2014, at 1:52 AM, Aditya adityakish...@gmail.com wrote: Looks like the regular patch command skips any binary included in patches. On Sat, Jul 19, 2014 at 1:37 AM, Aditya adityakish...@gmail.com wrote: Thanks for taking a look Ted! Looks like the second patch created with git diff excluded the Gradle wrapper JAR from the patch. I would generate a new one which includes this this jar. In the meantime, you should be able to use the first patch attached to the JIRA which is in git-am format and that would let you build. On Fri, Jul 18, 2014 at 3:40 PM, Ted Yu yuzhih...@gmail.com wrote: Nice work, Aditya. Looks like the hbase-native-client profile requires gradle ? [exec] Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain Will take a look at your patch. Cheers On Fri, Jul 18, 2014 at 3:08 PM, Aditya adityakish...@gmail.com wrote: As requested, I have attached a combined patch to the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 and submitted it to jenkins. Would be great if someone could take a look and provide feedback. Regards, aditya... On Wed, Jul 9, 2014 at 7:05 PM, Aditya adityakish...@gmail.com wrote: I was hoping to get some initial comments before attaching patches for the build boat. I have broken the entire code into 5 patch sets, layered in a sequnce, each focusing on a particular area (public headers/JNI implementation/Examples+unit test, etc) for the ease of review. https://reviews.apache.org/r/23175/ https://reviews.apache.org/r/23176/ https://reviews.apache.org/r/23177/ https://reviews.apache.org/r/23178/ https://reviews.apache.org/r/23179/ These are also available as a sequence of patches as the pull request https://github.com/apache/hbase/pull/1. Only the last patch hooks everything to the HBase build process (optionally) and hence I was thinking of squashing these separate patches into a single patch to be submitted for build. On Wed, Jul 9, 2014 at 11:32 AM, Nick Dimiduk ndimi...@gmail.com wrote: This ticket has only open subtasks, ie nothing in 'patch available'. I assume you mean HBASE-10168. We'll see about getting you some reviews, but you should also go about formatting the patch for buildbot. Also, since your 3 reviews are individually 100+k, you should consider breaking them into three separate tickets. my 2¢ -n On Mon, Jul 7, 2014 at 12:01 PM, Aditya adityakish...@gmail.com wrote: Sorry about that. Here is the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 On Mon, Jul 7, 2014 at 10:05 AM, Nick Dimiduk ndimi...@gmail.com wrote: Would you mind including the JIRA numbers along with the request? Thanks, Nick On Mon, Jul 7, 2014 at 9:52 AM, Aditya adityakish...@gmail.com wrote: Do we want to have the C APIs part of 1.0.0 release. I had posted few patches and a set of review request sometime last week. On Fri, Jul 4, 2014 at 1:21 AM, Enis Söztutar enis@gmail.com wrote: On Thu, Jul 3, 2014 at 4:41 PM, Mikhail Antonov olorinb...@gmail.com wrote: Moved ZK watcher listener subtask out of scope HBASE-10909. Enis - with that, I guess HBASE-10909 can be marked in branch-1? Sounds good. HBASE-11464 - this is the jira where I'll capture tasks to abstract hbase client from ZK (mostly it would be post-1.0 work). Not sure whether we can make it fully backwards compatible with 1.0 clients. I guess we will see when the patches are done. Thanks, Mikhail 2014-07-03 12:52 GMT-07:00 Stack st...@duboce.net: On Thu, Jul 3, 2014 at 12:25 PM, Mikhail Antonov olorinb...@gmail.com wrote: Guys, getting back to ZK abstraction work w.r.t. release 1.0 and thereafter, some status update. So as we're getting closer to complete HBASE-10909, it looks like the steps may be like this: - there are 2 subtasks out there not closed yet, one of which is about log splitting (and Sergey S has submitted a patch for review), another is abstraction of ZK watcher (this is what I've been working on in the background; but after sketching the patch it seems like without being able to modify the control flows and some changes in the module structure, it'd be a lot of scaffolding code not really simplifying current code). So I'd propose to descope abstraction
Re: How to validate HBase configuration (for region servers) from HBase admin page?
Hi St.Ack, We are working with HBase 0.96.1.1-cdh5.0.1 using 10 region servers. We configured the bucket cache and set the -XX:MaxDirectMemorySize to 32g, heap size of 12g, and hbase.regionserver.count=40. The bucket cache configuration is the following (added in hbase-site.xml): property namehbase.regionserver.handler.count/name value40/value /property property namehbase.bucketcache.ioengine/name valueoffheap/value /property property namehbase.bucketcache.percentage.in.combinedcache/name value0.8/value /property property namehbase.bucketcache.size/name value4/value /property After restarting HBase, we checked out the HBase process from console we are able to set the -XX:MaxDirectMemorySize=32, as well as the memory heap size we configured. However, when we go to the http://hostname:60010/conf, the*hbase.regionserver.count* is still set up to 30 (presumably the default value) and the bucketcache parameters added (bucketcache.ioengine, bucketcache.percentage.in.combinedcache, bucketcache.size) are not showed up, so we are not sure if the parameters are taken into account or not. Is there a way to validate that the bucketcache parameters as well as the new value for the hbase.regionserver.count are really taken from the hbase-site.xml we changed? Thanks, Gaby On 28/07/2014 02:23 p.m., Stack wrote: And don't forget to set -XX:MaxDirectMemorySize (from 9.6.4.5.1. Enable SlabCache in the refguide -- http://hbase.apache.org/book.html). This you set where you pass your server JVM args. St.Ack On Mon, Jul 28, 2014 at 12:21 PM, Stack st...@duboce.net wrote: Probably a question for the cloudera hbase list? (Did you enter the block cache configs via CM safety valve mechanism?) Yours, St.Ack On Mon, Jul 28, 2014 at 11:54 AM, Gabriela Montiel gabriela.mont...@oracle.com wrote: Hi Shengzhe, We executed a rolling restart on HBase through Cloudera 5, but after completing the site retrieved the same configurations. Thanks, Gaby Sent from my Samsung Mobile Original message From: yao yaosheng...@gmail.com Date:28/07/2014 13:43 (GMT-06:00) To: dev@hbase.apache.org Subject: Re: How to validate HBase configuration (for region servers) from HBase admin page? Hi Gaby, Did you restart your regionservers ? If not, HBase won't pick up the up-to-date configuration. Best Shengzhe On Mon, Jul 28, 2014 at 11:38 AM, Ted Yu yuzhih...@gmail.com wrote: bq. sourcehbase-site.xml/source Can you check the hbase-site.xml which is in classpath of the region server ? What release of hbase are you using ? Cheers On Mon, Jul 28, 2014 at 11:28 AM, gabriela montiel gabriela.mont...@oracle.com wrote: Hi all, We have set up an HBase cluster with 10 region nodes. We modified the default configurations to handle bucket cache in the hbase-site.xml (for all region servers) as follows: property namehbase.regionserver.handler.count/name value40/value /property property namehbase.bucketcache.ioengine/name valueoffheap/value /property property namehbase.bucketcache.percentage.in.combinedcache/name value0.8/value /property property namehbase.bucketcache.size/name value4/value /property However, when we access the HBase configuration page on the HBase Web admin, we are not able to see the updated configurations, even for the ones not related to bucket cache: propertynamehbase.regionserver.handler.count/ namevalue*30*/valuesourcehbase-site.xml/source/property Can you give us a hint on why the configurations are not seen in the HBase configuration site, and how we can check out the HBase configuration for all region servers? Thanks, Gaby -- Regards, Gaby http://www.oracle.com Gabriela Montiel Moreno | Senior Member of Technical Staff Phone: +52 33 12531343 tel:+52%2033%2012531343 Oracle Semantic Technoliges ORACLE Mexico http://www.oracle.com/commitment Oracle is committed to developing practices and products that help protect the environment
[jira] [Created] (HBASE-11603) Apply version of HADOOP-8027 to our JMXJsonServlet
stack created HBASE-11603:
-
Summary: Apply version of HADOOP-8027 to our JMXJsonServlet
Key: HBASE-11603
URL: https://issues.apache.org/jira/browse/HBASE-11603
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: 11603.txt
Running master, I see below when I poll /jmx servlet:
{code}
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1345)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.UnsupportedOperationException: CollectionUsage threshold
is not supported
at
sun.management.MemoryPoolImpl.isCollectionUsageThresholdExceeded(MemoryPoolImpl.java:242)
at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at
com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)
at
com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)
at
com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)
at
com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at javax.management.StandardMBean.getAttribute(StandardMBean.java:372)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
{code}
This has been seen before over in HADOOP-8027. We get it since we brought
jetty server local from hadoop. I don't think it in branch-1, at least, will
wait till see it there.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-3288) CacheOnWrite and EvictOnClose should be configurable on a per-family basis
[ https://issues.apache.org/jira/browse/HBASE-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-3288. -- Resolution: Won't Fix Too hard just now. Our blockcache is by server, not by column family so configs per column family especially when blockcache is global to the server, would be messy to implement. Putting aside for now. Resolving as won't fix. CacheOnWrite and EvictOnClose should be configurable on a per-family basis -- Key: HBASE-3288 URL: https://issues.apache.org/jira/browse/HBASE-3288 Project: HBase Issue Type: Improvement Components: io, regionserver Reporter: Jonathan Gray HBASE-3287 introduced two new configuration parameters. These should both be optionally configurable at the family level. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11604) Disable co-locating meta/master by default
Jimmy Xiang created HBASE-11604: --- Summary: Disable co-locating meta/master by default Key: HBASE-11604 URL: https://issues.apache.org/jira/browse/HBASE-11604 Project: HBase Issue Type: Task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 1.0.0 To avoid possible confusing, it's better to keep the original deployment scheme in 1.0. ZK-less region assignment is off by default in 1.0 already. We should, by default, not assign any region to master or backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: How to validate HBase configuration (for region servers) from HBase admin page?
On Mon, Jul 28, 2014 at 1:50 PM, gabriela montiel gabriela.mont...@oracle.com wrote: Hi St.Ack, We are working with HBase 0.96.1.1-cdh5.0.1 using 10 region servers. We configured the bucket cache and set the -XX:MaxDirectMemorySize to 32g, heap size of 12g, and hbase.regionserver.count=40. The bucket cache configuration is the following (added in hbase-site.xml): property namehbase.regionserver.handler.count/name value40/value /property property namehbase.bucketcache.ioengine/name valueoffheap/value /property property namehbase.bucketcache.percentage.in.combinedcache/name value0.8/value /property property namehbase.bucketcache.size/name value4/value /property After restarting HBase, we checked out the HBase process from console we are able to set the -XX:MaxDirectMemorySize=32, as well as the memory heap size we configured. However, when we go to the http://hostname:60010/conf, the*hbase.regionserver.count* is still set up to 30 (presumably the default value) and the bucketcache parameters added (bucketcache.ioengine, bucketcache.percentage.in.combinedcache, bucketcache.size) are not showed up, so we are not sure if the parameters are taken into account or not. I do not know what hbase.regionserver.count is. Do you mean hbase.regionserver.handler.count? Is there a way to validate that the bucketcache parameters as well as the new value for the hbase.regionserver.count are really taken from the hbase-site.xml we changed? It is bit awkward in 0.96 but look in your logs. Look for ' LOG.info(Allocating LruBlockCache with maximum size + StringUtils.humanReadableInt(lruCacheSize));' Should be 0.2 * 12G. Also looking for LOG.info(Started bucket cache); If you turn on DEBUG even temporarily on the regionserver, you should see stats about your block cache emitted every 5 minutes or so: LOG.debug(BucketCache Stats: + failedBlockAdditions= + this.failedBlockAdditions.get() + , + total= + StringUtils.byteDesc(totalSize) + , + free= + StringUtils.byteDesc(freeSize) + , + usedSize= + StringUtils.byteDesc(usedSize) +, + cacheSize= + StringUtils.byteDesc(cacheSize) +, + accesses= + cacheStats.getRequestCount() + , + hits= + cacheStats.getHitCount() + , + IOhitsPerSecond= + cacheStats.getIOHitsPerSecond() + , + IOTimePerHit= + String.format(%.2f, cacheStats.getIOTimePerHit())+ , + hitRatio= + (cacheStats.getHitCount() == 0 ? 0, : (StringUtils.formatPercent(cacheStats.getHitRatio(), 2)+ , )) + cachingAccesses= + cacheStats.getRequestCachingCount() + , + cachingHits= + cacheStats.getHitCachingCount() + , + cachingHitsRatio= +(cacheStats.getHitCachingCount() == 0 ? 0, : (StringUtils.formatPercent(cacheStats.getHitCachingRatio(), 2)+ , )) + evictions= + cacheStats.getEvictionCount() + , + evicted= + cacheStats.getEvictedCount() + , + evictedPerRun= + cacheStats.evictedPerEviction()); St.Ack
[jira] [Created] (HBASE-11605) All classes that override equals should have a corresponding hashCode
Jeffrey Shmain created HBASE-11605: -- Summary: All classes that override equals should have a corresponding hashCode Key: HBASE-11605 URL: https://issues.apache.org/jira/browse/HBASE-11605 Project: HBase Issue Type: Bug Reporter: Jeffrey Shmain Priority: Trivial I was looking at this link: https://analysis.apache.org/drilldown/issues/71923?rule=findbugs%3AHE_EQUALS_USE_HASHCODErule_sev=CRITICAL And some classes that define equals do not define hashCode, which is not the best practice. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: DISCUSSION: 1.0.0
I started with https://reviews.apache.org/r/23175/ Will continue reviewing this week. On Mon, Jul 28, 2014 at 1:40 PM, Aditya adityakish...@gmail.com wrote: Did anyone get a chance to take a look at the patches? Regards, aditya... On Sun, Jul 20, 2014 at 8:37 PM, Aditya adityakish...@gmail.com wrote: The wrapper jar is part of the first patch, which is in git mailbox patch format. On Sat, Jul 19, 2014 at 2:03 AM, Ted Yu yuzhih...@gmail.com wrote: You may want to attach the wrapper jar to the JIRA directly. Cheers On Jul 19, 2014, at 1:52 AM, Aditya adityakish...@gmail.com wrote: Looks like the regular patch command skips any binary included in patches. On Sat, Jul 19, 2014 at 1:37 AM, Aditya adityakish...@gmail.com wrote: Thanks for taking a look Ted! Looks like the second patch created with git diff excluded the Gradle wrapper JAR from the patch. I would generate a new one which includes this this jar. In the meantime, you should be able to use the first patch attached to the JIRA which is in git-am format and that would let you build. On Fri, Jul 18, 2014 at 3:40 PM, Ted Yu yuzhih...@gmail.com wrote: Nice work, Aditya. Looks like the hbase-native-client profile requires gradle ? [exec] Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain Will take a look at your patch. Cheers On Fri, Jul 18, 2014 at 3:08 PM, Aditya adityakish...@gmail.com wrote: As requested, I have attached a combined patch to the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 and submitted it to jenkins. Would be great if someone could take a look and provide feedback. Regards, aditya... On Wed, Jul 9, 2014 at 7:05 PM, Aditya adityakish...@gmail.com wrote: I was hoping to get some initial comments before attaching patches for the build boat. I have broken the entire code into 5 patch sets, layered in a sequnce, each focusing on a particular area (public headers/JNI implementation/Examples+unit test, etc) for the ease of review. https://reviews.apache.org/r/23175/ https://reviews.apache.org/r/23176/ https://reviews.apache.org/r/23177/ https://reviews.apache.org/r/23178/ https://reviews.apache.org/r/23179/ These are also available as a sequence of patches as the pull request https://github.com/apache/hbase/pull/1. Only the last patch hooks everything to the HBase build process (optionally) and hence I was thinking of squashing these separate patches into a single patch to be submitted for build. On Wed, Jul 9, 2014 at 11:32 AM, Nick Dimiduk ndimi...@gmail.com wrote: This ticket has only open subtasks, ie nothing in 'patch available'. I assume you mean HBASE-10168. We'll see about getting you some reviews, but you should also go about formatting the patch for buildbot. Also, since your 3 reviews are individually 100+k, you should consider breaking them into three separate tickets. my 2¢ -n On Mon, Jul 7, 2014 at 12:01 PM, Aditya adityakish...@gmail.com wrote: Sorry about that. Here is the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 On Mon, Jul 7, 2014 at 10:05 AM, Nick Dimiduk ndimi...@gmail.com wrote: Would you mind including the JIRA numbers along with the request? Thanks, Nick On Mon, Jul 7, 2014 at 9:52 AM, Aditya adityakish...@gmail.com wrote: Do we want to have the C APIs part of 1.0.0 release. I had posted few patches and a set of review request sometime last week. On Fri, Jul 4, 2014 at 1:21 AM, Enis Söztutar enis@gmail.com wrote: On Thu, Jul 3, 2014 at 4:41 PM, Mikhail Antonov olorinb...@gmail.com wrote: Moved ZK watcher listener subtask out of scope HBASE-10909. Enis - with that, I guess HBASE-10909 can be marked in branch-1? Sounds good. HBASE-11464 - this is the jira where I'll capture tasks to abstract hbase client from ZK (mostly it would be post-1.0 work). Not sure whether we can make it fully backwards compatible with 1.0 clients. I guess we will see when the patches are done. Thanks, Mikhail 2014-07-03 12:52 GMT-07:00 Stack st...@duboce.net: On Thu, Jul 3, 2014 at 12:25 PM, Mikhail Antonov olorinb...@gmail.com wrote: Guys, getting back to ZK abstraction work w.r.t. release 1.0 and thereafter, some status update. So as we're getting closer to complete HBASE-10909, it looks like the steps may be like this: - there are 2 subtasks out there not closed yet, one of which is about log splitting (and Sergey S has submitted a patch for review), another is
[jira] [Created] (HBASE-11606) Enable ZK-less region assignment by default
Jimmy Xiang created HBASE-11606: --- Summary: Enable ZK-less region assignment by default Key: HBASE-11606 URL: https://issues.apache.org/jira/browse/HBASE-11606 Project: HBase Issue Type: Task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 2.0.0 Let's enable ZK-less region assignment by default in the master branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11607) Document HBase metrics
Misty Stanley-Jones created HBASE-11607: --- Summary: Document HBase metrics Key: HBASE-11607 URL: https://issues.apache.org/jira/browse/HBASE-11607 Project: HBase Issue Type: Sub-task Components: documentation, metrics Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11595) WAL files with encryption not flushed properly
[
https://issues.apache.org/jira/browse/HBASE-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-11595.
Resolution: Invalid
Fix Version/s: (was: 2.0.0)
(was: 0.98.5)
(was: 0.99.0)
WAL files with encryption not flushed properly
--
Key: HBASE-11595
URL: https://issues.apache.org/jira/browse/HBASE-11595
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Andrew Purtell
Priority: Critical
Reported using HBase 0.98.3 and HDFS 2.4.1
All data before failure has not yet been flushed so only exists in the WAL
files. During distributed splitting, the WAL has either not been written out
and synced in the same way as an unencrypted WAL or is unreadable:
{noformat}
2014-07-26 19:29:16,160 ERROR [RS_LOG_REPLAY_OPS-host1:60020-0]
codec.BaseDecoder: Partial cell read caused by EOF: java.io.IOException:
Premature EOF from inputStream
{noformat}
This file is still moved to oldWALs even though splitting failed.
Setting 'hbase.regionserver.wal.encryption' to false allows data recovery.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
About dfs.client-write-packet-size setting
The default dfs.client-write-packet-size value is 64k, at least it's in my Hadoop2 env. I did a benchmark about i via ycsb loading 2 million records(3*200 bytes): 1) dfs.client-write-packet-size=64k ygc count:399, ygct:4.208s 2) dfs.client-write-packet-size=8k ygc count:163, ygct:2.644s you see, it's about 40% benefit on gct:) It's because: in DFSOutputStream.Packet class, each Create a new packet operation, will call buf = new byte[PacketHeader.PKT_MAX_HEADER_LEN + pktSize];, here pktSize comes from dfs.client-write-packet-size setting, and in HBase write scenario, we sync WAL asap, so all the new packets are very small (in my ycsb testing, most of them were only hundreds of bytes, or a few kilo bytes), rarely reached to 64k, so always allocating 64k array is just a waste. It would be better that if we add it to refguide note:) ps; 8k just a test setting, we should set it according the real kv size pattern. Thanks,
