[jira] [Created] (HBASE-20551) Can't find bundle for base name org.apache.jasper.resources.LocalStrings, locale en_US
Aditya Pawaskar created HBASE-20551:
---
Summary: Can't find bundle for base name
org.apache.jasper.resources.LocalStrings, locale en_US
Key: HBASE-20551
URL: https://issues.apache.org/jira/browse/HBASE-20551
Project: HBase
Issue Type: Bug
Components: build, java, test
Affects Versions: 1.2.6
Environment: Ubuntu 18.04
OpenJDK version "1.8.0_162"
maven - 3.3.9
Reporter: Aditya Pawaskar
While building HBase 1.2.6 on Ubuntu 18.04, I observe build failure with error :
*Can't find bundle for base name org.apache.jasper.resources.LocalStrings,
locale en_US*
Steps to produce:
git clone git://github.com/apache/hbase.git
{color:#005cc5}cd{color} hbase
git checkout rel/1.2.6
mvn package
Error logs:
[INFO] Logging to
org.slf4j.impl.MavenSimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog
java.util.MissingResourceException: Can't find bundle for base name
org.apache.jasper.resources.LocalStrings, locale en_US
at
java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1573)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Updating the YCSB client for HBase
That's a great news Sean! Good to see some activity after a long time! Would you also be interested in accepting patches in the core module as well? Regards, aditya... On Tue, May 26, 2015 at 9:28 AM, Sean Busbey bus...@cloudera.com wrote: Hi Folks! I've recently become a maintainer on YCSB. We're looking to get a release out in June (our first in several years). We'll probably begin release candidates in a couple of weeks. I'd really like to improve the experience of folks using HBase; the current state of the datastore binding leaves much to be desired. Right now, people can't even reliably run against multiple versions of HBase. I *think* only using 1.0+ client jars works, regardless of old/new API usage. I'll find out for sure in RC testing. Unfortunately, my time on the project is quickly absorbed handling reviews and community management efforts. I know a few folks in the HBase community have made their own changes over the last few years. If folks could find some time to rebase their forks for the current state of the master branch and submit pull requests I'd be super grateful. -- Sean
Re: [ANNOUNCE] Apache HBase 1.0.0 is now available for download
A very important milestone indeed! Congratulations to everyone who have been part of this wonderful community. aditya... On Wed, Feb 25, 2015 at 12:22 AM, Lars George lars.geo...@gmail.com wrote: Great work everyone! Congratulations, this is the most awesome community to be in. Some coverage: - https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces72 - http://www.heise.de/developer/meldung/Big-Data-HBase-1-0-erschienen-2558708.html On Tue, Feb 24, 2015 at 9:30 AM, Enis Söztutar e...@apache.org wrote: The HBase Team is pleased to announce the immediate release of HBase 1.0.0. Download it from your favorite Apache mirror [1] or maven repository. HBase 1.0.0 is the next stable release, and the start of semantic versioned releases (See [2]). The 1.0.0 release has three goals: 1) to lay a stable foundation for future 1.x releases; 2) to stabilize running HBase cluster and its clients; and 3) make versioning and compatibility dimensions explicit Including previous (developer preview) 0.99.x releases, 1.0.0 contains over 1500 jiras resolved on top of 0.98.0. Some of the major changes are: API reorganization and changes HBase’s client level API has evolved over the years. To simplify the semantics and to support and make it extensible and easier to use in the future, we revisited the API before 1.0. To that end, 1.0.0 introduces new APIs, and deprecates some of the commonly-used client side APIs (HTableInterface, HTable and HBaseAdmin). We advise to update your application to use the new style of APIs, since deprecated APIs will be removed in future releases (2.x). See [3] and [4] for an overview of changes. All Client side API's are marked with InterfaceAudience.Public class, indicating that the class/method is an official client API for HBase (See “11.1.1. HBase API Surface” in the HBase Refguide[2] for more details on the Audience annotations) Going forward, all 1.x releases are planned to be API compatible for classes annotated as client public. Read availability using timeline consistent region replicas This release contains Phase 1 items for experimental Read availability using timeline consistent region replicas feature. A region can be hosted in multiple region servers in read-only mode. One of the replicas for the region will be primary, accepting writes, and other replicas will be sharing the same data files. Read requests can be done against any replica for the region with backup RPCs for high availability with timeline consistency guarantees. More information can be found at HBASE-10070. Online config change and other forward ports from 0.89-fb branch HBASE-12147 forward ported online config change which enables some of the configuration from the server to be reloaded without restarting the region servers. Master runs a Region Server as well Starting with 1.0.0, the HBase master server and backup master servers will also act as a region server. RPC port and info port for web UI is shared for the master and region server roles. Active master can host regions of defined tables if configured (disabled by default). Backup masters will not host regions. Other notable improvements in 1.0.0 are listed (but not limited to) below: - A new web skin in time for 1.0 (http://hbase.apache.org) - [HBASE-5349] - Automatic tuning of global memstore and block cache sizes - Various security, tags and visibility labels improvements - Bucket cache improvements (usability and compressed data blocks) - [HBASE-11367] - A new pluggable replication endpoint to plug in to HBase's inter-cluster replication to replicate to a custom data store - [HBASE-11885] - A Dockerfile to easily build and run HBase from source - [HBASE-8332] - Truncate table command - [HBASE-11059] - Region assignment to use hbase:meta table instead of zookeeper for faster region assignment (disabled by default) - Extensive documentation improvements - [HBASE-12511] - namespace permissions - add support from table creation privilege in a namespace 'C' - [HBASE-12568] - Adopt Semantic Versioning and document it in the book - [HBASE-12640] - Add Thrift-over-HTTPS and doAs support for Thrift Server - [HBASE-12651] - Backport HBASE-12559 'Provide LoadBalancer with online configuration capability' to branch-1 - [HBASE-10560] - Per cell TTLs - [HBASE-11997] - CopyTable with bulkload - [HBASE-11990] - Make setting the start and stop row for a specific prefix easier - [HBASE-12220] - Add hedgedReads and hedgedReadWins metrics - [HBASE-12032] - Script to stop regionservers via RPC - [HBASE-11907] - Use the joni byte[] regex engine in place of j.u.regex in RegexStringComparator - [HBASE-11796] - Add client support for atomic
[jira] [Created] (HBASE-13010) HFileOutputFormat2 partitioner's path is hard-coded as '/tmp'
Aditya Kishore created HBASE-13010:
--
Summary: HFileOutputFormat2 partitioner's path is hard-coded as
'/tmp'
Key: HBASE-13010
URL: https://issues.apache.org/jira/browse/HBASE-13010
Project: HBase
Issue Type: Bug
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Instead we should use {{$\{hadoop.tmp.dir\}}} which maps to
{{/tmp/hadoop-$\{user.name\}}} as the users may not have write access to
{{/tmp}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13009) HBase REST UI inaccessible
Aditya Kishore created HBASE-13009: -- Summary: HBase REST UI inaccessible Key: HBASE-13009 URL: https://issues.apache.org/jira/browse/HBASE-13009 Project: HBase Issue Type: Bug Components: REST Affects Versions: 0.98.10 Reporter: Aditya Kishore Assignee: Aditya Kishore Since the movement of HBase REST code into its own module, the UI is no longer accessible because the web.xml does not get bundled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Is it just me, or is maven now already rebuilding everything?
This is ubiquitous. With HBASE-11912[1], the maven compiler plugin version have been revised from 2.5.1 to 3.2, which suffers from this[2]. [1] https://github.com/apache/hbase/commit/70a312e53fc798c178f6abc95614a6673e9da244#diff-600376dffeb79835ede4a0b285078036L441 [2] https://jira.codehaus.org/browse/MCOMPILER-205 On Tue, Oct 28, 2014 at 9:10 PM, Ted Yu yuzhih...@gmail.com wrote: I tried 'mvn compile' and observed same behavior. FYI On Tue, Oct 28, 2014 at 9:01 PM, lars hofhansl la...@apache.org wrote: I just noticed that in HBase trunk/master every invocation of maven (compile, test, install, etc) seems to recompile everything.Pretty sure it did use to to do this before. Is anybody else seeing this, or did I mess up my env? Thanks. -- Lars
Re: Is it just me, or is maven now already rebuilding everything?
It is unfortunate that this critical bug in maven-compiler-plugin has remain unresolved for so long. In Apache Drill project, we are considering reverting to version 2.5.1 as lack of incremental compilation is a big inconvenience. On Wed, Oct 29, 2014 at 3:54 PM, Andrew Purtell apurt...@apache.org wrote: The compiler plugin upgrade was required for that change. On Wed, Oct 29, 2014 at 1:32 PM, Aditya adityakish...@gmail.com wrote: This is ubiquitous. With HBASE-11912[1], the maven compiler plugin version have been revised from 2.5.1 to 3.2, which suffers from this[2]. [1] https://github.com/apache/hbase/commit/70a312e53fc798c178f6abc95614a6673e9da244#diff-600376dffeb79835ede4a0b285078036L441 [2] https://jira.codehaus.org/browse/MCOMPILER-205 On Tue, Oct 28, 2014 at 9:10 PM, Ted Yu yuzhih...@gmail.com wrote: I tried 'mvn compile' and observed same behavior. FYI On Tue, Oct 28, 2014 at 9:01 PM, lars hofhansl la...@apache.org wrote: I just noticed that in HBase trunk/master every invocation of maven (compile, test, install, etc) seems to recompile everything.Pretty sure it did use to to do this before. Is anybody else seeing this, or did I mess up my env? Thanks. -- Lars -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-12230) User impersonation does not work in 'simple' mode.
Aditya Kishore created HBASE-12230:
--
Summary: User impersonation does not work in 'simple' mode.
Key: HBASE-12230
URL: https://issues.apache.org/jira/browse/HBASE-12230
Project: HBase
Issue Type: Bug
Components: REST, security
Affects Versions: 0.98.6.1
Reporter: Aditya Kishore
Assignee: Aditya Kishore
The [code responsible for initializing proxy
configuration|https://github.com/apache/hbase/blob/7cfdb38c9274e306ac37374c147a978c2cef31d6/hbase-server/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java#L54]
does not execute unless {{hadoop.security.authorization}} is set to true.
This is departure from other Hadoop components. Impersonation should not be
tied to authorization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12231) REST gateway authentication should be independent of Hadoop/HBase security configuration.
Aditya Kishore created HBASE-12231: -- Summary: REST gateway authentication should be independent of Hadoop/HBase security configuration. Key: HBASE-12231 URL: https://issues.apache.org/jira/browse/HBASE-12231 Project: HBase Issue Type: Improvement Components: REST Affects Versions: 0.98.6.1 Reporter: Aditya Kishore Assignee: Aditya Kishore It is reasonable that a user would want to secure the HBase REST interface without having to incur the cost of securing the entire HBase/Hadoop ecosystem. The current implementation does not allow that and needs to be modified. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12043) REST server should respond with FORBIDDEN(403) code on AccessDeniedException
Aditya Kishore created HBASE-12043: -- Summary: REST server should respond with FORBIDDEN(403) code on AccessDeniedException Key: HBASE-12043 URL: https://issues.apache.org/jira/browse/HBASE-12043 Project: HBase Issue Type: Bug Components: REST Affects Versions: 0.98.6.1 Reporter: Aditya Kishore Assignee: Aditya Kishore Currently it just throws a SecurityException with Unauthorized which does not get translated to 403 status code to the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12044) REST delete operation should not retry disableTable for DoNotRetryIOException
Aditya Kishore created HBASE-12044:
--
Summary: REST delete operation should not retry disableTable for
DoNotRetryIOException
Key: HBASE-12044
URL: https://issues.apache.org/jira/browse/HBASE-12044
Project: HBase
Issue Type: Bug
Components: REST
Affects Versions: 0.98.6.1
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Minor
At present, it will keep retrying 10 times even if the exception is of
{{DoNotRetryIOException}} type like {{TableNotFoundException}} or
{{TableNotEnabledException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12045) REST proxy users configuration in hbase-site.xml is ignored
Aditya Kishore created HBASE-12045:
--
Summary: REST proxy users configuration in hbase-site.xml is
ignored
Key: HBASE-12045
URL: https://issues.apache.org/jira/browse/HBASE-12045
Project: HBase
Issue Type: Bug
Reporter: Aditya Kishore
Assignee: Aditya Kishore
As described in [this comment on
HBASE-10439|https://issues.apache.org/jira/browse/HBASE-10439?focusedCommentId=14142709page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14142709],
the proxy configuration placed in {{hbase-site.xml}} is ignored.
The user must put this setting in {{core-site.xml}} for it to work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: DISCUSSION: 1.0.0
Thanks Ted! On Mon, Jul 28, 2014 at 4:37 PM, Ted Yu yuzhih...@gmail.com wrote: I started with https://reviews.apache.org/r/23175/ Will continue reviewing this week. On Mon, Jul 28, 2014 at 1:40 PM, Aditya adityakish...@gmail.com wrote: Did anyone get a chance to take a look at the patches? Regards, aditya... On Sun, Jul 20, 2014 at 8:37 PM, Aditya adityakish...@gmail.com wrote: The wrapper jar is part of the first patch, which is in git mailbox patch format. On Sat, Jul 19, 2014 at 2:03 AM, Ted Yu yuzhih...@gmail.com wrote: You may want to attach the wrapper jar to the JIRA directly. Cheers On Jul 19, 2014, at 1:52 AM, Aditya adityakish...@gmail.com wrote: Looks like the regular patch command skips any binary included in patches. On Sat, Jul 19, 2014 at 1:37 AM, Aditya adityakish...@gmail.com wrote: Thanks for taking a look Ted! Looks like the second patch created with git diff excluded the Gradle wrapper JAR from the patch. I would generate a new one which includes this this jar. In the meantime, you should be able to use the first patch attached to the JIRA which is in git-am format and that would let you build. On Fri, Jul 18, 2014 at 3:40 PM, Ted Yu yuzhih...@gmail.com wrote: Nice work, Aditya. Looks like the hbase-native-client profile requires gradle ? [exec] Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain Will take a look at your patch. Cheers On Fri, Jul 18, 2014 at 3:08 PM, Aditya adityakish...@gmail.com wrote: As requested, I have attached a combined patch to the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 and submitted it to jenkins. Would be great if someone could take a look and provide feedback. Regards, aditya... On Wed, Jul 9, 2014 at 7:05 PM, Aditya adityakish...@gmail.com wrote: I was hoping to get some initial comments before attaching patches for the build boat. I have broken the entire code into 5 patch sets, layered in a sequnce, each focusing on a particular area (public headers/JNI implementation/Examples+unit test, etc) for the ease of review. https://reviews.apache.org/r/23175/ https://reviews.apache.org/r/23176/ https://reviews.apache.org/r/23177/ https://reviews.apache.org/r/23178/ https://reviews.apache.org/r/23179/ These are also available as a sequence of patches as the pull request https://github.com/apache/hbase/pull/1. Only the last patch hooks everything to the HBase build process (optionally) and hence I was thinking of squashing these separate patches into a single patch to be submitted for build. On Wed, Jul 9, 2014 at 11:32 AM, Nick Dimiduk ndimi...@gmail.com wrote: This ticket has only open subtasks, ie nothing in 'patch available'. I assume you mean HBASE-10168. We'll see about getting you some reviews, but you should also go about formatting the patch for buildbot. Also, since your 3 reviews are individually 100+k, you should consider breaking them into three separate tickets. my 2¢ -n On Mon, Jul 7, 2014 at 12:01 PM, Aditya adityakish...@gmail.com wrote: Sorry about that. Here is the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 On Mon, Jul 7, 2014 at 10:05 AM, Nick Dimiduk ndimi...@gmail.com wrote: Would you mind including the JIRA numbers along with the request? Thanks, Nick On Mon, Jul 7, 2014 at 9:52 AM, Aditya adityakish...@gmail.com wrote: Do we want to have the C APIs part of 1.0.0 release. I had posted few patches and a set of review request sometime last week. On Fri, Jul 4, 2014 at 1:21 AM, Enis Söztutar enis@gmail.com wrote: On Thu, Jul 3, 2014 at 4:41 PM, Mikhail Antonov olorinb...@gmail.com wrote: Moved ZK watcher listener subtask out of scope HBASE-10909. Enis - with that, I guess HBASE-10909 can be marked in branch-1? Sounds good. HBASE-11464 - this is the jira where I'll capture tasks to abstract hbase client from ZK (mostly it would be post-1.0 work). Not sure whether we can make it fully backwards compatible with 1.0 clients. I guess we will see when the patches are done. Thanks, Mikhail 2014-07-03 12:52 GMT-07:00 Stack st...@duboce.net: On Thu, Jul 3, 2014 at 12:25 PM, Mikhail Antonov olorinb...@gmail.com wrote: Guys, getting back to ZK abstraction work w.r.t. release 1.0 and thereafter, some status update. So as we're getting closer to complete HBASE-10909, it looks like the steps may be like this: - there are 2 subtasks out there not closed yet, one of which is about log
Re: DISCUSSION: 1.0.0
Did anyone get a chance to take a look at the patches? Regards, aditya... On Sun, Jul 20, 2014 at 8:37 PM, Aditya adityakish...@gmail.com wrote: The wrapper jar is part of the first patch, which is in git mailbox patch format. On Sat, Jul 19, 2014 at 2:03 AM, Ted Yu yuzhih...@gmail.com wrote: You may want to attach the wrapper jar to the JIRA directly. Cheers On Jul 19, 2014, at 1:52 AM, Aditya adityakish...@gmail.com wrote: Looks like the regular patch command skips any binary included in patches. On Sat, Jul 19, 2014 at 1:37 AM, Aditya adityakish...@gmail.com wrote: Thanks for taking a look Ted! Looks like the second patch created with git diff excluded the Gradle wrapper JAR from the patch. I would generate a new one which includes this this jar. In the meantime, you should be able to use the first patch attached to the JIRA which is in git-am format and that would let you build. On Fri, Jul 18, 2014 at 3:40 PM, Ted Yu yuzhih...@gmail.com wrote: Nice work, Aditya. Looks like the hbase-native-client profile requires gradle ? [exec] Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain Will take a look at your patch. Cheers On Fri, Jul 18, 2014 at 3:08 PM, Aditya adityakish...@gmail.com wrote: As requested, I have attached a combined patch to the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 and submitted it to jenkins. Would be great if someone could take a look and provide feedback. Regards, aditya... On Wed, Jul 9, 2014 at 7:05 PM, Aditya adityakish...@gmail.com wrote: I was hoping to get some initial comments before attaching patches for the build boat. I have broken the entire code into 5 patch sets, layered in a sequnce, each focusing on a particular area (public headers/JNI implementation/Examples+unit test, etc) for the ease of review. https://reviews.apache.org/r/23175/ https://reviews.apache.org/r/23176/ https://reviews.apache.org/r/23177/ https://reviews.apache.org/r/23178/ https://reviews.apache.org/r/23179/ These are also available as a sequence of patches as the pull request https://github.com/apache/hbase/pull/1. Only the last patch hooks everything to the HBase build process (optionally) and hence I was thinking of squashing these separate patches into a single patch to be submitted for build. On Wed, Jul 9, 2014 at 11:32 AM, Nick Dimiduk ndimi...@gmail.com wrote: This ticket has only open subtasks, ie nothing in 'patch available'. I assume you mean HBASE-10168. We'll see about getting you some reviews, but you should also go about formatting the patch for buildbot. Also, since your 3 reviews are individually 100+k, you should consider breaking them into three separate tickets. my 2¢ -n On Mon, Jul 7, 2014 at 12:01 PM, Aditya adityakish...@gmail.com wrote: Sorry about that. Here is the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 On Mon, Jul 7, 2014 at 10:05 AM, Nick Dimiduk ndimi...@gmail.com wrote: Would you mind including the JIRA numbers along with the request? Thanks, Nick On Mon, Jul 7, 2014 at 9:52 AM, Aditya adityakish...@gmail.com wrote: Do we want to have the C APIs part of 1.0.0 release. I had posted few patches and a set of review request sometime last week. On Fri, Jul 4, 2014 at 1:21 AM, Enis Söztutar enis@gmail.com wrote: On Thu, Jul 3, 2014 at 4:41 PM, Mikhail Antonov olorinb...@gmail.com wrote: Moved ZK watcher listener subtask out of scope HBASE-10909. Enis - with that, I guess HBASE-10909 can be marked in branch-1? Sounds good. HBASE-11464 - this is the jira where I'll capture tasks to abstract hbase client from ZK (mostly it would be post-1.0 work). Not sure whether we can make it fully backwards compatible with 1.0 clients. I guess we will see when the patches are done. Thanks, Mikhail 2014-07-03 12:52 GMT-07:00 Stack st...@duboce.net: On Thu, Jul 3, 2014 at 12:25 PM, Mikhail Antonov olorinb...@gmail.com wrote: Guys, getting back to ZK abstraction work w.r.t. release 1.0 and thereafter, some status update. So as we're getting closer to complete HBASE-10909, it looks like the steps may be like this: - there are 2 subtasks out there not closed yet, one of which is about log splitting (and Sergey S has submitted a patch for review), another is abstraction of ZK watcher (this is what I've been working on in the background; but after sketching the patch it seems like without being able to modify the control flows and some changes in the module structure, it'd be a lot of scaffolding code not really simplifying current code). So I'd propose to descope abstraction
Re: hadoopqa -p0 patches
I did not mean that we enforce this, rather encourage. Would make committers' and reviewers' job easier. On Sun, Jul 20, 2014 at 4:06 AM, Ted Yu yuzhih...@gmail.com wrote: I agree with what Enis said. Cheers On Jul 19, 2014, at 11:03 PM, Enis Söztutar enis@gmail.com wrote: I don't think we should have much formatting restrictions around patches. git format patch is nice to use, and if the author provides it the committers can use, but I don't think we should force that for every patch. just my 2 cents. Enis On Sat, Jul 19, 2014 at 1:48 AM, Aditya adityakish...@gmail.com wrote: Now that the primary source repository for HBase is Git, shouldn't we move to git-am format patches? On Thu, Jul 17, 2014 at 5:39 PM, Enis Söztutar e...@apache.org wrote: Hi, It seems for some time, the hadoopqa script cannot apply the patches generated for -p0. I've emailed Giri about it, who recently moved the jenkins builds nodes in builds.apache.org. We believe that the issue maybe about the default shell not being bash, but waiting for him to come back from vacation. In the meantime I've disabled the smart handling of -p0 patches. The patch application should work now except for patches which just add new files. If you have such a patch, please make sure that it is generated for -p0. Enis
Re: DISCUSSION: 1.0.0
The wrapper jar is part of the first patch, which is in git mailbox patch format. On Sat, Jul 19, 2014 at 2:03 AM, Ted Yu yuzhih...@gmail.com wrote: You may want to attach the wrapper jar to the JIRA directly. Cheers On Jul 19, 2014, at 1:52 AM, Aditya adityakish...@gmail.com wrote: Looks like the regular patch command skips any binary included in patches. On Sat, Jul 19, 2014 at 1:37 AM, Aditya adityakish...@gmail.com wrote: Thanks for taking a look Ted! Looks like the second patch created with git diff excluded the Gradle wrapper JAR from the patch. I would generate a new one which includes this this jar. In the meantime, you should be able to use the first patch attached to the JIRA which is in git-am format and that would let you build. On Fri, Jul 18, 2014 at 3:40 PM, Ted Yu yuzhih...@gmail.com wrote: Nice work, Aditya. Looks like the hbase-native-client profile requires gradle ? [exec] Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain Will take a look at your patch. Cheers On Fri, Jul 18, 2014 at 3:08 PM, Aditya adityakish...@gmail.com wrote: As requested, I have attached a combined patch to the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 and submitted it to jenkins. Would be great if someone could take a look and provide feedback. Regards, aditya... On Wed, Jul 9, 2014 at 7:05 PM, Aditya adityakish...@gmail.com wrote: I was hoping to get some initial comments before attaching patches for the build boat. I have broken the entire code into 5 patch sets, layered in a sequnce, each focusing on a particular area (public headers/JNI implementation/Examples+unit test, etc) for the ease of review. https://reviews.apache.org/r/23175/ https://reviews.apache.org/r/23176/ https://reviews.apache.org/r/23177/ https://reviews.apache.org/r/23178/ https://reviews.apache.org/r/23179/ These are also available as a sequence of patches as the pull request https://github.com/apache/hbase/pull/1. Only the last patch hooks everything to the HBase build process (optionally) and hence I was thinking of squashing these separate patches into a single patch to be submitted for build. On Wed, Jul 9, 2014 at 11:32 AM, Nick Dimiduk ndimi...@gmail.com wrote: This ticket has only open subtasks, ie nothing in 'patch available'. I assume you mean HBASE-10168. We'll see about getting you some reviews, but you should also go about formatting the patch for buildbot. Also, since your 3 reviews are individually 100+k, you should consider breaking them into three separate tickets. my 2¢ -n On Mon, Jul 7, 2014 at 12:01 PM, Aditya adityakish...@gmail.com wrote: Sorry about that. Here is the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 On Mon, Jul 7, 2014 at 10:05 AM, Nick Dimiduk ndimi...@gmail.com wrote: Would you mind including the JIRA numbers along with the request? Thanks, Nick On Mon, Jul 7, 2014 at 9:52 AM, Aditya adityakish...@gmail.com wrote: Do we want to have the C APIs part of 1.0.0 release. I had posted few patches and a set of review request sometime last week. On Fri, Jul 4, 2014 at 1:21 AM, Enis Söztutar enis@gmail.com wrote: On Thu, Jul 3, 2014 at 4:41 PM, Mikhail Antonov olorinb...@gmail.com wrote: Moved ZK watcher listener subtask out of scope HBASE-10909. Enis - with that, I guess HBASE-10909 can be marked in branch-1? Sounds good. HBASE-11464 - this is the jira where I'll capture tasks to abstract hbase client from ZK (mostly it would be post-1.0 work). Not sure whether we can make it fully backwards compatible with 1.0 clients. I guess we will see when the patches are done. Thanks, Mikhail 2014-07-03 12:52 GMT-07:00 Stack st...@duboce.net: On Thu, Jul 3, 2014 at 12:25 PM, Mikhail Antonov olorinb...@gmail.com wrote: Guys, getting back to ZK abstraction work w.r.t. release 1.0 and thereafter, some status update. So as we're getting closer to complete HBASE-10909, it looks like the steps may be like this: - there are 2 subtasks out there not closed yet, one of which is about log splitting (and Sergey S has submitted a patch for review), another is abstraction of ZK watcher (this is what I've been working on in the background; but after sketching the patch it seems like without being able to modify the control flows and some changes in the module structure, it'd be a lot of scaffolding code not really simplifying current code). So I'd propose to descope abstraction of ZK watcher jira (HBASE-11073), namely: convert it to top-level JIRA and continue to work on it separately; rename HBASE-10909
Re: DISCUSSION: 1.0.0
Thanks for taking a look Ted! Looks like the second patch created with git diff excluded the Gradle wrapper JAR from the patch. I would generate a new one which includes this this jar. In the meantime, you should be able to use the first patch attached to the JIRA which is in git-am format and that would let you build. On Fri, Jul 18, 2014 at 3:40 PM, Ted Yu yuzhih...@gmail.com wrote: Nice work, Aditya. Looks like the hbase-native-client profile requires gradle ? [exec] Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain Will take a look at your patch. Cheers On Fri, Jul 18, 2014 at 3:08 PM, Aditya adityakish...@gmail.com wrote: As requested, I have attached a combined patch to the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 and submitted it to jenkins. Would be great if someone could take a look and provide feedback. Regards, aditya... On Wed, Jul 9, 2014 at 7:05 PM, Aditya adityakish...@gmail.com wrote: I was hoping to get some initial comments before attaching patches for the build boat. I have broken the entire code into 5 patch sets, layered in a sequnce, each focusing on a particular area (public headers/JNI implementation/Examples+unit test, etc) for the ease of review. https://reviews.apache.org/r/23175/ https://reviews.apache.org/r/23176/ https://reviews.apache.org/r/23177/ https://reviews.apache.org/r/23178/ https://reviews.apache.org/r/23179/ These are also available as a sequence of patches as the pull request https://github.com/apache/hbase/pull/1. Only the last patch hooks everything to the HBase build process (optionally) and hence I was thinking of squashing these separate patches into a single patch to be submitted for build. On Wed, Jul 9, 2014 at 11:32 AM, Nick Dimiduk ndimi...@gmail.com wrote: This ticket has only open subtasks, ie nothing in 'patch available'. I assume you mean HBASE-10168. We'll see about getting you some reviews, but you should also go about formatting the patch for buildbot. Also, since your 3 reviews are individually 100+k, you should consider breaking them into three separate tickets. my 2¢ -n On Mon, Jul 7, 2014 at 12:01 PM, Aditya adityakish...@gmail.com wrote: Sorry about that. Here is the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 On Mon, Jul 7, 2014 at 10:05 AM, Nick Dimiduk ndimi...@gmail.com wrote: Would you mind including the JIRA numbers along with the request? Thanks, Nick On Mon, Jul 7, 2014 at 9:52 AM, Aditya adityakish...@gmail.com wrote: Do we want to have the C APIs part of 1.0.0 release. I had posted few patches and a set of review request sometime last week. On Fri, Jul 4, 2014 at 1:21 AM, Enis Söztutar enis@gmail.com wrote: On Thu, Jul 3, 2014 at 4:41 PM, Mikhail Antonov olorinb...@gmail.com wrote: Moved ZK watcher listener subtask out of scope HBASE-10909. Enis - with that, I guess HBASE-10909 can be marked in branch-1? Sounds good. HBASE-11464 - this is the jira where I'll capture tasks to abstract hbase client from ZK (mostly it would be post-1.0 work). Not sure whether we can make it fully backwards compatible with 1.0 clients. I guess we will see when the patches are done. Thanks, Mikhail 2014-07-03 12:52 GMT-07:00 Stack st...@duboce.net: On Thu, Jul 3, 2014 at 12:25 PM, Mikhail Antonov olorinb...@gmail.com wrote: Guys, getting back to ZK abstraction work w.r.t. release 1.0 and thereafter, some status update. So as we're getting closer to complete HBASE-10909, it looks like the steps may be like this: - there are 2 subtasks out there not closed yet, one of which is about log splitting (and Sergey S has submitted a patch for review), another is abstraction of ZK watcher (this is what I've been working on in the background; but after sketching the patch it seems like without being able to modify the control flows and some changes in the module structure, it'd be a lot of scaffolding code not really simplifying current code). So I'd propose to descope abstraction of ZK watcher jira (HBASE-11073), namely: convert it to top-level JIRA and continue to work on it separately; rename HBASE-10909 to ZK abstraction: phase 1, and mark it as closed as soon as log splitting jira is completed. This way HBASE-10909 fits to branch-1. Sounds good to me. - secondly, in the discussion to the CatalogTracker patch, we started talking about modifying client to not know about ZK, but rather keep the location of current masters and talk to them using RPC calls. This work can
Re: hadoopqa -p0 patches
Now that the primary source repository for HBase is Git, shouldn't we move to git-am format patches? On Thu, Jul 17, 2014 at 5:39 PM, Enis Söztutar e...@apache.org wrote: Hi, It seems for some time, the hadoopqa script cannot apply the patches generated for -p0. I've emailed Giri about it, who recently moved the jenkins builds nodes in builds.apache.org. We believe that the issue maybe about the default shell not being bash, but waiting for him to come back from vacation. In the meantime I've disabled the smart handling of -p0 patches. The patch application should work now except for patches which just add new files. If you have such a patch, please make sure that it is generated for -p0. Enis
Re: DISCUSSION: 1.0.0
Looks like the regular patch command skips any binary included in patches. On Sat, Jul 19, 2014 at 1:37 AM, Aditya adityakish...@gmail.com wrote: Thanks for taking a look Ted! Looks like the second patch created with git diff excluded the Gradle wrapper JAR from the patch. I would generate a new one which includes this this jar. In the meantime, you should be able to use the first patch attached to the JIRA which is in git-am format and that would let you build. On Fri, Jul 18, 2014 at 3:40 PM, Ted Yu yuzhih...@gmail.com wrote: Nice work, Aditya. Looks like the hbase-native-client profile requires gradle ? [exec] Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain Will take a look at your patch. Cheers On Fri, Jul 18, 2014 at 3:08 PM, Aditya adityakish...@gmail.com wrote: As requested, I have attached a combined patch to the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 and submitted it to jenkins. Would be great if someone could take a look and provide feedback. Regards, aditya... On Wed, Jul 9, 2014 at 7:05 PM, Aditya adityakish...@gmail.com wrote: I was hoping to get some initial comments before attaching patches for the build boat. I have broken the entire code into 5 patch sets, layered in a sequnce, each focusing on a particular area (public headers/JNI implementation/Examples+unit test, etc) for the ease of review. https://reviews.apache.org/r/23175/ https://reviews.apache.org/r/23176/ https://reviews.apache.org/r/23177/ https://reviews.apache.org/r/23178/ https://reviews.apache.org/r/23179/ These are also available as a sequence of patches as the pull request https://github.com/apache/hbase/pull/1. Only the last patch hooks everything to the HBase build process (optionally) and hence I was thinking of squashing these separate patches into a single patch to be submitted for build. On Wed, Jul 9, 2014 at 11:32 AM, Nick Dimiduk ndimi...@gmail.com wrote: This ticket has only open subtasks, ie nothing in 'patch available'. I assume you mean HBASE-10168. We'll see about getting you some reviews, but you should also go about formatting the patch for buildbot. Also, since your 3 reviews are individually 100+k, you should consider breaking them into three separate tickets. my 2¢ -n On Mon, Jul 7, 2014 at 12:01 PM, Aditya adityakish...@gmail.com wrote: Sorry about that. Here is the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 On Mon, Jul 7, 2014 at 10:05 AM, Nick Dimiduk ndimi...@gmail.com wrote: Would you mind including the JIRA numbers along with the request? Thanks, Nick On Mon, Jul 7, 2014 at 9:52 AM, Aditya adityakish...@gmail.com wrote: Do we want to have the C APIs part of 1.0.0 release. I had posted few patches and a set of review request sometime last week. On Fri, Jul 4, 2014 at 1:21 AM, Enis Söztutar enis@gmail.com wrote: On Thu, Jul 3, 2014 at 4:41 PM, Mikhail Antonov olorinb...@gmail.com wrote: Moved ZK watcher listener subtask out of scope HBASE-10909. Enis - with that, I guess HBASE-10909 can be marked in branch-1? Sounds good. HBASE-11464 - this is the jira where I'll capture tasks to abstract hbase client from ZK (mostly it would be post-1.0 work). Not sure whether we can make it fully backwards compatible with 1.0 clients. I guess we will see when the patches are done. Thanks, Mikhail 2014-07-03 12:52 GMT-07:00 Stack st...@duboce.net: On Thu, Jul 3, 2014 at 12:25 PM, Mikhail Antonov olorinb...@gmail.com wrote: Guys, getting back to ZK abstraction work w.r.t. release 1.0 and thereafter, some status update. So as we're getting closer to complete HBASE-10909, it looks like the steps may be like this: - there are 2 subtasks out there not closed yet, one of which is about log splitting (and Sergey S has submitted a patch for review), another is abstraction of ZK watcher (this is what I've been working on in the background; but after sketching the patch it seems like without being able to modify the control flows and some changes in the module structure, it'd be a lot of scaffolding code not really simplifying current code). So I'd propose to descope abstraction of ZK watcher jira (HBASE-11073), namely: convert it to top-level JIRA and continue to work on it separately; rename HBASE-10909 to ZK abstraction: phase 1, and mark it as closed as soon as log splitting jira is completed. This way HBASE-10909 fits to branch-1. Sounds good to me. - secondly, in the discussion to the CatalogTracker patch, we started talking about
Re: build a hbase client only packge
I think it might be useful to add a maven assembly descriptor which builds such a client-only package. The native client patch that I have submitted https://reviews.apache.org/r/23179/diff/#index_header, builds a separate tarball which only includes the necessary files required to use the native client library. Jerry, mind creating a JIRA and submitting a patch for the Java client? On Wed, Jul 16, 2014 at 8:14 PM, Jerry He jerry...@gmail.com wrote: Hi, Nick I was just looking into what is being done for Hadoop over there. If I build hadoop at the top level, I see there is a hadoop-client-2.2.0.tar.gz created in hadoop-client/target. It looks like something of a client tarball that packages the client jars with dependencies ... [bdvm102 target]$ ls antrun dist-maketar.sh hadoop-client-2.2.0-javadoc.jar javadoc-bundle-options test-dir archive-tmp hadoop-client-2.2.0 hadoop-client-2.2.0.tar.gz maven-archiver dist-copynativelibs.sh hadoop-client-2.2.0.jar hadoop-client-2.2.0-tests.jartest $tar tzf hadoop-client-2.2.0.tar.gz hadoop-client-2.2.0/ hadoop-client-2.2.0/share/ hadoop-client-2.2.0/share/hadoop/ hadoop-client-2.2.0/share/hadoop/client/ hadoop-client-2.2.0/share/hadoop/client/hadoop-client-2.2.0-tests.jar hadoop-client-2.2.0/share/hadoop/client/lib/ hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-auth-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-yarn-common-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/zookeeper-3.4.5.jar hadoop-client-2.2.0/share/hadoop/client/lib/log4j-1.2.17.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-cli-1.2.jar hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-annotations-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/xmlenc-0.52.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-codec-1.4.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-lang-2.5.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-logging-1.1.1.jar hadoop-client-2.2.0/share/hadoop/client/lib/jsr305-1.3.9.jar hadoop-client-2.2.0/share/hadoop/client/lib/slf4j-log4j12-1.7.5.jar hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-mapreduce-client-jobclient-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-common-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/snappy-java-1.0.4.1.jar hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-yarn-client-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/protobuf-java-2.5.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-compress-1.4.1.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-configuration-1.6.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-math-2.2.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-io-2.1.jar hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-yarn-server-common-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-mapreduce-client-common-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-collections-3.2.1.jar hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-mapreduce-client-core-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/jackson-core-asl-1.8.8.jar hadoop-client-2.2.0/share/hadoop/client/lib/xz-1.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-hdfs-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-beanutils-1.8.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-yarn-api-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/paranamer-2.3.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-net-3.1.jar hadoop-client-2.2.0/share/hadoop/client/lib/slf4j-api-1.7.5.jar hadoop-client-2.2.0/share/hadoop/client/lib/avro-1.7.4.jar hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-mapreduce-client-shuffle-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-digester-1.8.jar hadoop-client-2.2.0/share/hadoop/client/lib/guava-11.0.2.jar hadoop-client-2.2.0/share/hadoop/client/lib/hadoop-mapreduce-client-app-2.2.0.jar hadoop-client-2.2.0/share/hadoop/client/lib/commons-httpclient-3.1.jar hadoop-client-2.2.0/share/hadoop/client/lib/jackson-mapper-asl-1.8.8.jar hadoop-client-2.2.0/share/hadoop/client/lib/jetty-util-6.1.26.jar hadoop-client-2.2.0/share/hadoop/client/hadoop-client-2.2.0.jar On Wed, Jul 16, 2014 at 4:45 PM, Nick Dimiduk ndimi...@gmail.com wrote: hbase-shell is a separate module in the java build (as is hbase-client). Are you thinking in terms java or of bigtop packaging here? On Wed, Jul 16, 2014 at 4:18 PM, Jerry He jerry...@gmail.com wrote: Hi, Nick If there is a way to include the hbase shell in a 'client only' package/assembly, that will be great. For the REST/Thrift gateways, there are probably people who want to have separate packages for them to install them. But not on my mind yet. On Wed, Jul 16, 2014 at 3:44 PM, Nick Dimiduk ndimi...@gmail.com wrote: What does client package include? Just
Re: DISCUSSION: 1.0.0
As requested, I have attached a combined patch to the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 and submitted it to jenkins. Would be great if someone could take a look and provide feedback. Regards, aditya... On Wed, Jul 9, 2014 at 7:05 PM, Aditya adityakish...@gmail.com wrote: I was hoping to get some initial comments before attaching patches for the build boat. I have broken the entire code into 5 patch sets, layered in a sequnce, each focusing on a particular area (public headers/JNI implementation/Examples+unit test, etc) for the ease of review. https://reviews.apache.org/r/23175/ https://reviews.apache.org/r/23176/ https://reviews.apache.org/r/23177/ https://reviews.apache.org/r/23178/ https://reviews.apache.org/r/23179/ These are also available as a sequence of patches as the pull request https://github.com/apache/hbase/pull/1. Only the last patch hooks everything to the HBase build process (optionally) and hence I was thinking of squashing these separate patches into a single patch to be submitted for build. On Wed, Jul 9, 2014 at 11:32 AM, Nick Dimiduk ndimi...@gmail.com wrote: This ticket has only open subtasks, ie nothing in 'patch available'. I assume you mean HBASE-10168. We'll see about getting you some reviews, but you should also go about formatting the patch for buildbot. Also, since your 3 reviews are individually 100+k, you should consider breaking them into three separate tickets. my 2¢ -n On Mon, Jul 7, 2014 at 12:01 PM, Aditya adityakish...@gmail.com wrote: Sorry about that. Here is the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 On Mon, Jul 7, 2014 at 10:05 AM, Nick Dimiduk ndimi...@gmail.com wrote: Would you mind including the JIRA numbers along with the request? Thanks, Nick On Mon, Jul 7, 2014 at 9:52 AM, Aditya adityakish...@gmail.com wrote: Do we want to have the C APIs part of 1.0.0 release. I had posted few patches and a set of review request sometime last week. On Fri, Jul 4, 2014 at 1:21 AM, Enis Söztutar enis@gmail.com wrote: On Thu, Jul 3, 2014 at 4:41 PM, Mikhail Antonov olorinb...@gmail.com wrote: Moved ZK watcher listener subtask out of scope HBASE-10909. Enis - with that, I guess HBASE-10909 can be marked in branch-1? Sounds good. HBASE-11464 - this is the jira where I'll capture tasks to abstract hbase client from ZK (mostly it would be post-1.0 work). Not sure whether we can make it fully backwards compatible with 1.0 clients. I guess we will see when the patches are done. Thanks, Mikhail 2014-07-03 12:52 GMT-07:00 Stack st...@duboce.net: On Thu, Jul 3, 2014 at 12:25 PM, Mikhail Antonov olorinb...@gmail.com wrote: Guys, getting back to ZK abstraction work w.r.t. release 1.0 and thereafter, some status update. So as we're getting closer to complete HBASE-10909, it looks like the steps may be like this: - there are 2 subtasks out there not closed yet, one of which is about log splitting (and Sergey S has submitted a patch for review), another is abstraction of ZK watcher (this is what I've been working on in the background; but after sketching the patch it seems like without being able to modify the control flows and some changes in the module structure, it'd be a lot of scaffolding code not really simplifying current code). So I'd propose to descope abstraction of ZK watcher jira (HBASE-11073), namely: convert it to top-level JIRA and continue to work on it separately; rename HBASE-10909 to ZK abstraction: phase 1, and mark it as closed as soon as log splitting jira is completed. This way HBASE-10909 fits to branch-1. Sounds good to me. - secondly, in the discussion to the CatalogTracker patch, we started talking about modifying client to not know about ZK, but rather keep the location of current masters and talk to them using RPC calls. This work can not go into branch-1, as it involves invasive changes in client including new RPC. As I understand the branching schema now, those changes can go to master branch, we just don't merge them to branch-1, and depending on their completeness we can pull them to 1.1 release or so. You have it right Mikhail. St.Ack -- Thanks, Michael Antonov
Re: DISCUSSION: 1.0.0
I was hoping to get some initial comments before attaching patches for the build boat. I have broken the entire code into 5 patch sets, layered in a sequnce, each focusing on a particular area (public headers/JNI implementation/Examples+unit test, etc) for the ease of review. https://reviews.apache.org/r/23175/ https://reviews.apache.org/r/23176/ https://reviews.apache.org/r/23177/ https://reviews.apache.org/r/23178/ https://reviews.apache.org/r/23179/ These are also available as a sequence of patches as the pull request https://github.com/apache/hbase/pull/1. Only the last patch hooks everything to the HBase build process (optionally) and hence I was thinking of squashing these separate patches into a single patch to be submitted for build. On Wed, Jul 9, 2014 at 11:32 AM, Nick Dimiduk ndimi...@gmail.com wrote: This ticket has only open subtasks, ie nothing in 'patch available'. I assume you mean HBASE-10168. We'll see about getting you some reviews, but you should also go about formatting the patch for buildbot. Also, since your 3 reviews are individually 100+k, you should consider breaking them into three separate tickets. my 2¢ -n On Mon, Jul 7, 2014 at 12:01 PM, Aditya adityakish...@gmail.com wrote: Sorry about that. Here is the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 On Mon, Jul 7, 2014 at 10:05 AM, Nick Dimiduk ndimi...@gmail.com wrote: Would you mind including the JIRA numbers along with the request? Thanks, Nick On Mon, Jul 7, 2014 at 9:52 AM, Aditya adityakish...@gmail.com wrote: Do we want to have the C APIs part of 1.0.0 release. I had posted few patches and a set of review request sometime last week. On Fri, Jul 4, 2014 at 1:21 AM, Enis Söztutar enis@gmail.com wrote: On Thu, Jul 3, 2014 at 4:41 PM, Mikhail Antonov olorinb...@gmail.com wrote: Moved ZK watcher listener subtask out of scope HBASE-10909. Enis - with that, I guess HBASE-10909 can be marked in branch-1? Sounds good. HBASE-11464 - this is the jira where I'll capture tasks to abstract hbase client from ZK (mostly it would be post-1.0 work). Not sure whether we can make it fully backwards compatible with 1.0 clients. I guess we will see when the patches are done. Thanks, Mikhail 2014-07-03 12:52 GMT-07:00 Stack st...@duboce.net: On Thu, Jul 3, 2014 at 12:25 PM, Mikhail Antonov olorinb...@gmail.com wrote: Guys, getting back to ZK abstraction work w.r.t. release 1.0 and thereafter, some status update. So as we're getting closer to complete HBASE-10909, it looks like the steps may be like this: - there are 2 subtasks out there not closed yet, one of which is about log splitting (and Sergey S has submitted a patch for review), another is abstraction of ZK watcher (this is what I've been working on in the background; but after sketching the patch it seems like without being able to modify the control flows and some changes in the module structure, it'd be a lot of scaffolding code not really simplifying current code). So I'd propose to descope abstraction of ZK watcher jira (HBASE-11073), namely: convert it to top-level JIRA and continue to work on it separately; rename HBASE-10909 to ZK abstraction: phase 1, and mark it as closed as soon as log splitting jira is completed. This way HBASE-10909 fits to branch-1. Sounds good to me. - secondly, in the discussion to the CatalogTracker patch, we started talking about modifying client to not know about ZK, but rather keep the location of current masters and talk to them using RPC calls. This work can not go into branch-1, as it involves invasive changes in client including new RPC. As I understand the branching schema now, those changes can go to master branch, we just don't merge them to branch-1, and depending on their completeness we can pull them to 1.1 release or so. You have it right Mikhail. St.Ack -- Thanks, Michael Antonov
Re: DISCUSSION: 1.0.0
Do we want to have the C APIs part of 1.0.0 release. I had posted few patches and a set of review request sometime last week. On Fri, Jul 4, 2014 at 1:21 AM, Enis Söztutar enis@gmail.com wrote: On Thu, Jul 3, 2014 at 4:41 PM, Mikhail Antonov olorinb...@gmail.com wrote: Moved ZK watcher listener subtask out of scope HBASE-10909. Enis - with that, I guess HBASE-10909 can be marked in branch-1? Sounds good. HBASE-11464 - this is the jira where I'll capture tasks to abstract hbase client from ZK (mostly it would be post-1.0 work). Not sure whether we can make it fully backwards compatible with 1.0 clients. I guess we will see when the patches are done. Thanks, Mikhail 2014-07-03 12:52 GMT-07:00 Stack st...@duboce.net: On Thu, Jul 3, 2014 at 12:25 PM, Mikhail Antonov olorinb...@gmail.com wrote: Guys, getting back to ZK abstraction work w.r.t. release 1.0 and thereafter, some status update. So as we're getting closer to complete HBASE-10909, it looks like the steps may be like this: - there are 2 subtasks out there not closed yet, one of which is about log splitting (and Sergey S has submitted a patch for review), another is abstraction of ZK watcher (this is what I've been working on in the background; but after sketching the patch it seems like without being able to modify the control flows and some changes in the module structure, it'd be a lot of scaffolding code not really simplifying current code). So I'd propose to descope abstraction of ZK watcher jira (HBASE-11073), namely: convert it to top-level JIRA and continue to work on it separately; rename HBASE-10909 to ZK abstraction: phase 1, and mark it as closed as soon as log splitting jira is completed. This way HBASE-10909 fits to branch-1. Sounds good to me. - secondly, in the discussion to the CatalogTracker patch, we started talking about modifying client to not know about ZK, but rather keep the location of current masters and talk to them using RPC calls. This work can not go into branch-1, as it involves invasive changes in client including new RPC. As I understand the branching schema now, those changes can go to master branch, we just don't merge them to branch-1, and depending on their completeness we can pull them to 1.1 release or so. You have it right Mikhail. St.Ack -- Thanks, Michael Antonov
Re: DISCUSSION: 1.0.0
Sorry about that. Here is the umbrella JIRA https://issues.apache.org/jira/browse/HBASE-1015 On Mon, Jul 7, 2014 at 10:05 AM, Nick Dimiduk ndimi...@gmail.com wrote: Would you mind including the JIRA numbers along with the request? Thanks, Nick On Mon, Jul 7, 2014 at 9:52 AM, Aditya adityakish...@gmail.com wrote: Do we want to have the C APIs part of 1.0.0 release. I had posted few patches and a set of review request sometime last week. On Fri, Jul 4, 2014 at 1:21 AM, Enis Söztutar enis@gmail.com wrote: On Thu, Jul 3, 2014 at 4:41 PM, Mikhail Antonov olorinb...@gmail.com wrote: Moved ZK watcher listener subtask out of scope HBASE-10909. Enis - with that, I guess HBASE-10909 can be marked in branch-1? Sounds good. HBASE-11464 - this is the jira where I'll capture tasks to abstract hbase client from ZK (mostly it would be post-1.0 work). Not sure whether we can make it fully backwards compatible with 1.0 clients. I guess we will see when the patches are done. Thanks, Mikhail 2014-07-03 12:52 GMT-07:00 Stack st...@duboce.net: On Thu, Jul 3, 2014 at 12:25 PM, Mikhail Antonov olorinb...@gmail.com wrote: Guys, getting back to ZK abstraction work w.r.t. release 1.0 and thereafter, some status update. So as we're getting closer to complete HBASE-10909, it looks like the steps may be like this: - there are 2 subtasks out there not closed yet, one of which is about log splitting (and Sergey S has submitted a patch for review), another is abstraction of ZK watcher (this is what I've been working on in the background; but after sketching the patch it seems like without being able to modify the control flows and some changes in the module structure, it'd be a lot of scaffolding code not really simplifying current code). So I'd propose to descope abstraction of ZK watcher jira (HBASE-11073), namely: convert it to top-level JIRA and continue to work on it separately; rename HBASE-10909 to ZK abstraction: phase 1, and mark it as closed as soon as log splitting jira is completed. This way HBASE-10909 fits to branch-1. Sounds good to me. - secondly, in the discussion to the CatalogTracker patch, we started talking about modifying client to not know about ZK, but rather keep the location of current masters and talk to them using RPC calls. This work can not go into branch-1, as it involves invasive changes in client including new RPC. As I understand the branching schema now, those changes can go to master branch, we just don't merge them to branch-1, and depending on their completeness we can pull them to 1.1 release or so. You have it right Mikhail. St.Ack -- Thanks, Michael Antonov
Re: protobuf version in 0.94
The protobuf Java stubs checked into the source tree are compiled with 2.4.x and are not binary compatible with 2.5 (and vice versa) so we would need two versions of stubs. The wire format, however, is compatible in either direction. On Sun, Apr 6, 2014 at 3:51 PM, lars hofhansl la...@apache.org wrote: I didn't say that right. What I meant was: If I added Hadoop 2.2, 2.3, and 2.4 profiles, and in those profiles override the protobuf version... There should be no issue with that, correct? The default protobuf version would remove 2.4.x. -- Lars From: Jean-Marc Spaggiari jean-m...@spaggiari.org To: dev dev@hbase.apache.org Sent: Wednesday, April 2, 2014 5:03 AM Subject: Re: protobuf version in 0.94 Hum, that means Hadoop 1.0.x and 1.2.x branches will fail... 2014-04-02 2:53 GMT-04:00 Esteban Gutierrez este...@cloudera.com: For HBase itself is not a problem :) it compiles, but users running HBase 0.94.x in clusters pre HADOOP-9845 they will hit issues during rolling restarts specially if the 2.4.0 jar is present. For instance, that can break MR jobs. esteban. -- Cloudera, Inc. On Tue, Apr 1, 2014 at 9:31 PM, lars hofhansl la...@apache.org wrote: Is anybody aware of any reason why we wouldn't just update the protobuf version in 0.94 to 2.5.0? I know 2.5.0 is backward compatible with 2.4.0, but is the reverse true as well? I.e. if we do not change any .proto files will 2.4.0 be able to decoded message encoded with 2.5.0? If yes, we should just upgrade. If not we can't upgrade since it would potentially break server-server compatibility and hence rolling upgrades. -- Lars
Re: [ANNOUNCE] HBase 0.98.1 is now available for download
Thanks Andrew! The release is missing a tag (http://svn.apache.org/viewvc/hbase/tags/)
Re: Introducing libhbase (C APIs for Apache HBase)
No, it does not. It uses a modified version of AsyncHBase libraryhttp://goo.gl/aikZDnover JNI. On Tue, Mar 18, 2014 at 12:31 AM, haosdent haosd...@gmail.com wrote: Hi, Aditya. Thank you for your great job. I am very exciting about these issues. If libhbase depends on Thrift? On Tue, Mar 18, 2014 at 9:25 AM, Aditya adityakish...@gmail.com wrote: Hi, Pursuant to the JIRAs HBASE-10168https://issues.apache.org/jira/browse/HBASE-10168, HBASE-9977 https://issues.apache.org/jira/browse/HBASE-9977 and HBASE-9835https://issues.apache.org/jira/browse/HBASE-9835I am happy to announce that the first draft of a JNI based implementation of C APIs for HBase is now available for your review. The source and instructions to build and use is available at MapR's Github repository http://goo.gl/dE5tzB. A slide from my presentation on the same can be downloaded from the meetup site http://goo.gl/nfXx9f. Would put the patches on the respective JIRA shortly. Regards, aditya... https://issues.apache.org/jira/browse/HBASE-9835 -- Best Regards, Haosdent Huang
Re: Introducing libhbase (C APIs for Apache HBase)
There is some penalty of transition from native code to JNI and back but I would expect it definitely performs better than a Thrift based client. Will publish performance (comparison) figures soon. On Tue, Mar 18, 2014 at 1:36 AM, haosdent haosd...@gmail.com wrote: Cool. So it have a better peformance? On Tue, Mar 18, 2014 at 4:09 PM, Aditya adityakish...@gmail.com wrote: No, it does not. It uses a modified version of AsyncHBase libraryhttp://goo.gl/aikZDnover JNI. On Tue, Mar 18, 2014 at 12:31 AM, haosdent haosd...@gmail.com wrote: Hi, Aditya. Thank you for your great job. I am very exciting about these issues. If libhbase depends on Thrift? On Tue, Mar 18, 2014 at 9:25 AM, Aditya adityakish...@gmail.com wrote: Hi, Pursuant to the JIRAs HBASE-10168https://issues.apache.org/jira/browse/HBASE-10168, HBASE-9977 https://issues.apache.org/jira/browse/HBASE-9977 and HBASE-9835https://issues.apache.org/jira/browse/HBASE-9835I am happy to announce that the first draft of a JNI based implementation of C APIs for HBase is now available for your review. The source and instructions to build and use is available at MapR's Github repository http://goo.gl/dE5tzB. A slide from my presentation on the same can be downloaded from the meetup site http://goo.gl/nfXx9f. Would put the patches on the respective JIRA shortly. Regards, aditya... https://issues.apache.org/jira/browse/HBASE-9835 -- Best Regards, Haosdent Huang -- Best Regards, Haosdent Huang
Introducing libhbase (C APIs for Apache HBase)
Hi, Pursuant to the JIRAs HBASE-10168https://issues.apache.org/jira/browse/HBASE-10168, HBASE-9977 https://issues.apache.org/jira/browse/HBASE-9977 and HBASE-9835https://issues.apache.org/jira/browse/HBASE-9835I am happy to announce that the first draft of a JNI based implementation of C APIs for HBase is now available for your review. The source and instructions to build and use is available at MapR's Github repository http://goo.gl/dE5tzB. A slide from my presentation on the same can be downloaded from the meetup site http://goo.gl/nfXx9f. Would put the patches on the respective JIRA shortly. Regards, aditya... https://issues.apache.org/jira/browse/HBASE-9835
Re: Are we no longer giving credit in SVN commit messages?
This might be easier if and when we move to Git as the system of record in which case git am will take care of credit provided the patch is submitted in git format. On Mon, Jan 20, 2014 at 8:43 PM, Stack st...@duboce.net wrote: I have never added the contributor. Let me do it from here on out. Let me write up the above agreement into the refguide. St.Ack On Mon, Jan 20, 2014 at 7:51 PM, ramkrishna vasudevan ramkrishna.s.vasude...@gmail.com wrote: +1. I used to follow the above said practice. But I add my own name also if its my own patch. so any case same format. Good to follow that. On Tue, Jan 21, 2014 at 7:28 AM, Andrew Purtell apurt...@apache.org wrote: On Mon, Jan 20, 2014 at 5:38 PM, lars hofhansl la...@apache.org wrote: What do people do about their own contributions? No additional attribution needed was the discussion. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-10272) Cluster becomes in-operational if the node hosting the active Master AND ROOT/META table goes offline
Aditya Kishore created HBASE-10272:
--
Summary: Cluster becomes in-operational if the node hosting the
active Master AND ROOT/META table goes offline
Key: HBASE-10272
URL: https://issues.apache.org/jira/browse/HBASE-10272
Project: HBase
Issue Type: Bug
Components: IPC/RPC
Affects Versions: 0.94.15
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Critical
Since HBASE-6364, HBase client caches a connection failure to a server and any
subsequent attempt to connect to the server throws a {{FailedServerException}}
Now if a node which hosted the active Master AND ROOT/META table goes offline,
the newly anointed Master's initial attempt to connect to the dead region
server will fail with {{NoRouteToHostException}} which it handles but since on
second attempt crashes with {{FailedServerException}}
Here is the log from one such occurance
{noformat}
2013-11-20 10:58:00,161 FATAL org.apache.hadoop.hbase.master.HMaster: Master
server abort: loaded coprocessors are: []
2013-11-20 10:58:00,161 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled
exception. Starting shutdown.
org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is
in the failed servers list: xxx02/192.168.1.102:60020
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:425)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at $Proxy9.getProtocolVersion(Unknown Source)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1335)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1294)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1281)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:506)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:383)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:445)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnection(CatalogTracker.java:464)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:624)
at
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:684)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:560)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:376)
at java.lang.Thread.run(Thread.java:662)
2013-11-20 10:58:00,162 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2013-11-20 10:58:00,162 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server
on 6
{noformat}
Each of the backup master will crash with same error and restarting them will
have the same effect. Once this happens, the cluster will remain in-operational
until the node with region server is brought online (or the Zookeeper node
containing the root region server and/or META entry from the ROOT table is
deleted).
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: [VOTE] The 1st hbase 0.94.15 release candidate is available for download
Would you like to consider https://issues.apache.org/jira/browse/HBASE-10193for 0.94.15? On Thu, Dec 19, 2013 at 12:57 PM, lars hofhansl la...@apache.org wrote: +1 - unpacked - checked docs - ran IntegrationTestBigLinkedList and IntegrationTestLoadAndVerify in local mode many times - loaded 100m rows, scanned, flushed, scanned, compacted, scanned. All good - did various performance tests, no regressions - tested with both JDK6 and JDK7 From: lars hofhansl la...@apache.org To: hbase-dev dev@hbase.apache.org Sent: Wednesday, December 18, 2013 3:59 PM Subject: [VOTE] The 1st hbase 0.94.15 release candidate is available for download The 1st 0.94.15 RC is available for download at http://people.apache.org/~larsh/hbase-0.94.15-rc0/ Signed with my code signing key: C7CFE328 HBase 0.94.15 is a bug fix release along with some performance improvements: [HBASE-7886] - [replication] hlog zk node will not be deleted if client roll hlog [HBASE-9485] - TableOutputCommitter should implement recovery if we don't want jobs to start from 0 on RM restart [HBASE-9995] - Not stopping ReplicationSink when using custom implementation for the ReplicationSink [HBASE-10014] - HRegion#doMiniBatchMutation rollbacks the memstore even if there is nothing to rollback. [HBASE-10015] - Replace intrinsic locking with explicit locks in StoreScanner [HBASE-10026] - HBaseAdmin#createTable could fail if region splits too fast [HBASE-10046] - Unmonitored HBase service could accumulate Status objects and OOM [HBASE-10057] - TestRestoreFlushSnapshotFromClient and TestRestoreSnapshotFromClient fail to finish occasionally [HBASE-10061] - TableMapReduceUtil.findOrCreateJar calls updateMap(null, ) resulting in thrown NPE [HBASE-10064] - AggregateClient.validateParameters can throw NPE [HBASE-10089] - Metrics intern table names cause eventual permgen OOM in 0.94 [HBASE-10111] - Verify that a snapshot is not corrupted before restoring it [HBASE-10112] - Hbase rest query params for maxVersions and maxValues are not parsed [HBASE-10117] - Avoid synchronization in HRegionScannerImpl.isFilterDone [HBASE-10120] - start-hbase.sh doesn't respect --config in non-distributed mode [HBASE-10179] - HRegionServer underreports readRequestCounts by 1 under certain conditions [HBASE-10181] - HBaseObjectWritable.readObject catches DoNotRetryIOException and wraps it back in a regular IOException [HBASE-9931] - Optional setBatch for CopyTable to copy large rows in batches [HBASE-10001] - Add a coprocessor to help testing the performances without taking into account the i/o [HBASE-10007] - PerformanceEvaluation: Add sampling and latency collection to randomRead test [HBASE-10010] - eliminate the put latency spike on the new log file beginning [HBASE-10048] - Add hlog number metric in regionserver [HBASE-10049] - Small improvments in region_mover.rb [HBASE-10093] - Unregister ReplicationSource metric bean when the replication source thread is terminated [HBASE-9047] - Tool to handle finishing replication when the cluster is offline [HBASE-10119] - Allow HBase coprocessors to clean up when they fail [HBASE-9927] - ReplicationLogCleaner#stop() calls HConnectionManager#deleteConnection() unnecessarily [HBASE-9986] - Incorporate HTTPS support for HBase (0.94 port) [HBASE-10058] - Test for HBASE-9915 (avoid reading index blocks) [HBASE-10189] - Intermittent TestReplicationSyncUpTool failure The list of changes is also available here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753version=12325559 Here're the jenkins runs for this RC: https://builds.apache.org/job/HBase-0.94.15/2/ and https://builds.apache.org/job/HBase-0.94.15-security/1/ Please try out the RC, check out the doc, take it for a spin, etc, and vote +1/-1 by EOD December 27th on whether we should release this as 0.94.15. (9 days because of the holidays) Thanks. -- Lars
[jira] [Created] (HBASE-10193) Cleanup HRegion if one of the store fails to open at region initialization
Aditya Kishore created HBASE-10193: -- Summary: Cleanup HRegion if one of the store fails to open at region initialization Key: HBASE-10193 URL: https://issues.apache.org/jira/browse/HBASE-10193 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.14, 0.96.1 Reporter: Aditya Kishore Assignee: Aditya Kishore Priority: Critical While investigating a different issue, I realized that the fix for HBASE-9737 is not sufficient to prevent resource leak if a region fails to open for some reason, say a corrupt HFile. The region may have, by then, opened other good HFiles in that store or other stores if it has more than one column family and their streams may leak if not closed. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HBASE-10168) JNI implementation of HBase C APIs for 0.94.x releases
Aditya Kishore created HBASE-10168: -- Summary: JNI implementation of HBase C APIs for 0.94.x releases Key: HBASE-10168 URL: https://issues.apache.org/jira/browse/HBASE-10168 Project: HBase Issue Type: Task Components: Client Affects Versions: 0.94.14 Reporter: Aditya Kishore Assignee: Aditya Kishore -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HBASE-10046) Unmonitored HBase service could accumulate Status objects and OOM
Aditya Kishore created HBASE-10046:
--
Summary: Unmonitored HBase service could accumulate Status objects
and OOM
Key: HBASE-10046
URL: https://issues.apache.org/jira/browse/HBASE-10046
Project: HBase
Issue Type: Bug
Components: monitoring
Affects Versions: 0.94.14, 0.96.0
Reporter: Aditya Kishore
Assignee: Aditya Kishore
This was observed in a cluster where HBase Master UI was not monitored for
quite a while. During this period, a distributed log spitting task failed in an
in-completable way and Master kept recreating the task over and over again.
And each such task would create a monitored status...
{code:title=SplitLogManager.java}
...
public long splitLogDistributed(final ListPath logDirs, PathFilter filter)
throws IOException {
MonitoredTask status = TaskMonitor.get().createStatus(
Doing distributed log split in + logDirs);
...
{code}
...which kept accumulating on heap.
Now these monitored tasks get cleaned only when someone looks at the service UI
otherwise they keep growing boundless.
The postmortem of the heap dump showed that these task object occupied 99% of
HBase master heap accumulated over a week.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-9986) Incorporate HTTPS support for HBase (0.94 port)
Aditya Kishore created HBASE-9986: - Summary: Incorporate HTTPS support for HBase (0.94 port) Key: HBASE-9986 URL: https://issues.apache.org/jira/browse/HBASE-9986 Project: HBase Issue Type: Task Reporter: Aditya Kishore Assignee: Ted Yu Fix For: 0.98.0, 0.96.1 Attachments: 9954-v1.txt, 9954-v2.txt, HBASE-9954_0.94.patch In various classes, http://; is hard coded. This JIRA adds support for using HBASE web UI via HTTPS -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9835) Define C language interface of HBase Client synchronous APIs
Aditya Kishore created HBASE-9835: - Summary: Define C language interface of HBase Client synchronous APIs Key: HBASE-9835 URL: https://issues.apache.org/jira/browse/HBASE-9835 Project: HBase Issue Type: Sub-task Components: Client Reporter: Aditya Kishore Assignee: Aditya Kishore Creating this as a sub task of HBASE-1015 to define Define C language interface of HBase Client synchronous APIs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9783) Validate column families early in o.a.h.h.regionserver.HRegion.mutateRowsWithLocks()
Aditya Kishore created HBASE-9783:
-
Summary: Validate column families early in
o.a.h.h.regionserver.HRegion.mutateRowsWithLocks()
Key: HBASE-9783
URL: https://issues.apache.org/jira/browse/HBASE-9783
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.12
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Minor
h6. 0.94 branch only
Calling {{HTable.mutateRow()}} with non-existent column families result in a
NPE on the region server which is wrapped into {{RemoteException}} and returned
to the client. Since NPE is not a {{DoNotRetryIOException}}, client keeps
repeating it until the number of retries are exhausted.
Negative case, hence creating as minor.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-9745) Append HBASE_CLASSPATH to end of Java classpath and use another env var for prefix
Aditya Kishore created HBASE-9745: - Summary: Append HBASE_CLASSPATH to end of Java classpath and use another env var for prefix Key: HBASE-9745 URL: https://issues.apache.org/jira/browse/HBASE-9745 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.94.11, 0.95.2, 0.98.0 Reporter: Aditya Kishore Assignee: Aditya Kishore HBASE-9097 changed the behavior to prefix HBASE_CLASSPATH to end of Java classpath instead of appending it. This break existing behavior (read more on HBASE-9097). We should revert to existing behavior and provide another way to prefix certain jars to the classpath. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9737) Resource leak if a corrupt HFile presents itself in a region leading to Region Server OOM
Aditya Kishore created HBASE-9737:
-
Summary: Resource leak if a corrupt HFile presents itself in a
region leading to Region Server OOM
Key: HBASE-9737
URL: https://issues.apache.org/jira/browse/HBASE-9737
Project: HBase
Issue Type: Bug
Components: HFile
Affects Versions: 0.94.12
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Critical
One of our customer was recently hit with OOM error on almost all of the region
servers.
Postmortem of the issue reveled that a corrupt HFile had made its way into one
of the regions which resulted into the region brought offline immediately which
is as per design.
What happened next reveals two issues:\\
\\
* As soon as the region was offlined, Master noticed this and tried to assign
the region to another region server which of course failed (again due to the
corrupt HFile) and then Master tried to assign this to another and so on. So
this region kept bouncing from one server to another and this went unnoticed
for few hours and all region servers log were filled with thousands of this
message:{noformat}org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Failed open of
region=userdata,50743646010,1378139055806.318c533716869574f10615703269497f.,
starting to roll back the global memstore size.
java.io.IOException: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
Trailer from file
/hbase/userdata/318c533716869574f10615703269497f/data/a3e2ae39f71441ac92a6563479fb976e
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:550)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:463)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3835)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3783)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
Trailer from file
/hbase/userdata/318c533716869574f10615703269497f/data/a3e2ae39f71441ac92a6563479fb976e
at
org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:404)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:257)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3017)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:525)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:523)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
{noformat} For situation like this, the region should be marked
offlined_with_error or something similar so that Master does not try to
assign it to another server without user fixing the issue. I will create a
separate JIRA for that.
* The second problem and the scope of this JIRA is that the function
{{org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion()}} throws exception
without closing the {{FSDataInputStream}} objects even if closeIStream is set
to true. This lead to orphan filesystem streams accumulating in region server
and it eventually died of OOM.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-9740) A corrupt HFile could cause endless attempts to assign the region without a chance of success
Aditya Kishore created HBASE-9740: - Summary: A corrupt HFile could cause endless attempts to assign the region without a chance of success Key: HBASE-9740 URL: https://issues.apache.org/jira/browse/HBASE-9740 Project: HBase Issue Type: Bug Reporter: Aditya Kishore Assignee: Aditya Kishore As described in HBASE-9737, a corrupt HFile in a region could lead to an assignment storm in the cluster since the Master will keep trying to assign the region to each region server one after another and obviously none will succeed. The region server, upon detecting such a scenario should mark the region as RS_ZK_REGION_FAILED_ERROR (or something to the effect) in the Zookeeper which should indicate the Master to stop assigning the region until the error has been resolved (via an HBase shell command, probably assign?) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9727) DELETE scanner operation is a no-op
Aditya Kishore created HBASE-9727: - Summary: DELETE scanner operation is a no-op Key: HBASE-9727 URL: https://issues.apache.org/jira/browse/HBASE-9727 Project: HBase Issue Type: Bug Components: REST Reporter: Aditya Kishore Assignee: Aditya Kishore -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9577) Log process environment variable on HBase service startup
Aditya Kishore created HBASE-9577: - Summary: Log process environment variable on HBase service startup Key: HBASE-9577 URL: https://issues.apache.org/jira/browse/HBASE-9577 Project: HBase Issue Type: Improvement Affects Versions: 0.94.11, 0.95.2 Reporter: Aditya Kishore Assignee: Aditya Kishore Priority: Minor Fix For: 0.96.0 HBase services already logs information related to JVM properties and command line arguments used to start the services which have been immensely helpful to investigate issues. One thing that they do not log is the environment variables and an unintended variable in the environment could lead to a scenario not reproducible anywhere else including its original location if the service is restarted differently. We should log environment variables (excluding those which may contains credentials) on service startup with option to disable this logging. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9581) Few tests broken since HBASE-9551
Aditya Kishore created HBASE-9581:
-
Summary: Few tests broken since HBASE-9551
Key: HBASE-9581
URL: https://issues.apache.org/jira/browse/HBASE-9581
Project: HBase
Issue Type: Bug
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Few test cases, for example
{{org.apache.hadoop.hbase.client.TestSnapshotFromClient}} hangs since the
commit for HBASE-9551
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9482) Do not enforce secure Hadoop for secure HBase
Aditya Kishore created HBASE-9482:
-
Summary: Do not enforce secure Hadoop for secure HBase
Key: HBASE-9482
URL: https://issues.apache.org/jira/browse/HBASE-9482
Project: HBase
Issue Type: Bug
Components: security
Reporter: Aditya Kishore
Assignee: Aditya Kishore
We should recommend and not enforce secure Hadoop underneath as a requirement
to run secure HBase.
Few of our customers have HBase clusters which expose only HBase services to
outside the physical network and no other services (including ssh) are
accessible from outside of such cluster.
However they are forced to setup secure Hadoop and incur the penalty of
security overhead at filesystem layer even if they do not need to.
The following code tests for both secure HBase and secure Hadoop.
{code:title=org.apache.hadoop.hbase.security.User|borderStyle=solid}
/**
* Returns whether or not secure authentication is enabled for HBase. Note
that
* HBase security requires HDFS security to provide any guarantees, so this
requires that
* both codehbase.security.authentication/code and
codehadoop.security.authentication/code
* are set to codekerberos/code.
*/
public static boolean isHBaseSecurityEnabled(Configuration conf) {
return kerberos.equalsIgnoreCase(conf.get(HBASE_SECURITY_CONF_KEY))
kerberos.equalsIgnoreCase(
conf.get(CommonConfigurationKeys.HADOOP_SECURITY_AUTHENTICATION));
}
{code}
What is worse that if {{hadoop.security.authentication}} is not set to
{{kerberos}} (undocumented at http://hbase.apache.org/book/security.html),
all other configuration have no impact and HBase RPCs silently switch back to
unsecured mode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9385) Log HBase Master command line arguments on startup
Aditya Kishore created HBASE-9385: - Summary: Log HBase Master command line arguments on startup Key: HBASE-9385 URL: https://issues.apache.org/jira/browse/HBASE-9385 Project: HBase Issue Type: Improvement Components: master Reporter: Aditya Kishore Assignee: Aditya Kishore Priority: Minor Fix For: 0.96.0 Region server already log JVM information (name, version, vendor, command line arguments, etc). Would be useful to have Master do it too. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9031) ImmutableBytesWritable.toString() should downcast the bytes before converting to hex string
Aditya Kishore created HBASE-9031:
-
Summary: ImmutableBytesWritable.toString() should downcast the
bytes before converting to hex string
Key: HBASE-9031
URL: https://issues.apache.org/jira/browse/HBASE-9031
Project: HBase
Issue Type: Bug
Components: io
Affects Versions: 0.94.9, 0.95.1
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Minor
Fix For: 0.95.2
The attached patch addresses few issues.
# We need only (3*this.length) capacity in ByteBuffer and not
(3*this.bytes.length).
# Do not calculate (offset + length) at every iteration.
# No test is required at every iteration to add space (' ') before every byte
other than the first one. Uses {{sb.substring(1)}} instead.
# Finally and most importantly (the original issue of this report), downcast
the promoted int (the parameter to {{Integer.toHexString()}}) to byte range.
Without #4, the byte array \{54,125,64, -1, -45\} is transformed to 36 7d 40
ffd3 instead of 36 7d 40 ff d3.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9032) Result.getBytes() returns null if backed by KeyValue array
Aditya Kishore created HBASE-9032: - Summary: Result.getBytes() returns null if backed by KeyValue array Key: HBASE-9032 URL: https://issues.apache.org/jira/browse/HBASE-9032 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.94.9 Reporter: Aditya Kishore Assignee: Aditya Kishore Fix For: 0.94.10 Attachments: HBASE-9032.patch This applies only to 0.94 (and earlier) branch. If the Result object was constructed using either of Result(KeyValue[]) or Result(ListKeyValue), calling Result.getBytes() returns null instead of the serialized ImmutableBytesWritable object. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [ANNOUNCE] HBase 0.94.9 is available for download
Thanks Lars. It looks like HBASE-8656 https://issues.apache.org/jira/browse/HBASE-8656, though part of the tarball, is not included in 0.94.9 tag. Regards, Aditya... On Thu, Jul 4, 2013 at 10:58 AM, lars hofhansl la...@apache.org wrote: The HBase Team is pleased to announce the immediate release of HBase 0.94.9. Download it from your favorite Apache mirror [1]. As usual, all previous 0.92.x and 0.94.x releases can upgraded to 0.94.9 via a rolling upgrade without downtime, intermediary versions can be skipped. 0.94.9 is the current stable release of HBase. It fixes the following 26 issues: [HBASE-8453] - TestImportExport failing again due to configuration issues [HBASE-8494] - TestRemoteAdmin#testClusterStatus should not assume 'requests' does not change [HBASE-8522] - Archived hfiles and old hlogs may be deleted immediately by HFileCleaner, LogCleaner in HMaster [HBASE-8555] - FilterList correctness may be affected by random ordering of sub-filter(list) [HBASE-8590] - [0.94] BlockingMetaScannerVisitor should check for parent meta entry while waiting for split daughter [HBASE-8639] - Poor performance of htable#getscanner in multithreaded environment due to DNS.getDefaultHost() being called in ScannerCallable#prepare() [HBASE-8640] - ServerName in master may not initialize with the configured ipc address of hbase.master.ipc.address [HBASE-8655] - Backport to 94 - HBASE-8346(Prefetching .META. rows in case only when useCache is set to true) [HBASE-8656] - Rpc call may not be notified in SecureClient [HBASE-8671] - Per-region WAL breaks CP backwards compatibility in 0.94 for non-enabled case [HBASE-8684] - Table Coprocessor can't access external HTable by default [HBASE-8700] - IntegrationTestBigLinkedList can fail due to random number collision [HBASE-8724] - [0.94] ExportSnapshot should not use hbase.tmp.dir as a staging dir on hdfs [HBASE-8742] - HTableDescriptor Properties not preserved when cloning [HBASE-8743] - upgrade hadoop-23 version to 0.23.7 [HBASE-8749] - Potential race condition between FSUtils.renameAndSetModifyTime() and HFile/LogCleaner [HBASE-8762] - Performance/operational penalty when calling HTable.get with a list of one Get [HBASE-8783] - RSSnapshotManager.ZKProcedureMemberRpcs may be initialized with the wrong server name [HBASE-5083] - Backup HMaster should have http infoport open with link to the active master [HBASE-8609] - Make the CopyTable support startRow, stopRow options [HBASE-8636] - Backport KeyValue Codec to 0.94 (HBASE-7413) [HBASE-8683] - Add major compaction support in CompactionTool [HBASE-8692] - [AccessController] Restrict HTableDescriptor enumeration [HBASE-8702] - Make WALEditCodec pluggable [HBASE-8504] - HTable.getRegionsInRange() should provide a non-cached API [HBASE-8603] - Backport HBASE-6921 to 0.94 See also the full release notes [2]. Thanks to the 18 individuals who contributed to this release! Yours, The HBase Team 1. http://www.apache.org/dyn/closer.cgi/hbase/ 2. https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753version=12324431
Re: [ANNOUNCE} The 2nd hbase 0.94.5 release candidate is available for download
Thanks Andrew, you are too kind :) And yes, this is first of the many bottled tests we run to validate all HBase builds included in our distribution. Regards, Aditya... On Thu, Feb 14, 2013 at 11:45 AM, Andrew Purtell apurt...@apache.orgwrote: That's impressive. Can we bottle that test? :-) +1 from me too, but I was currently only able to check the basics on a single node. On Thu, Feb 14, 2013 at 11:30 AM, Aditya adityakish...@gmail.com wrote: +1 11 nodes cluster, 50 tables (1GiB-2TiB), 5K regions. Cluster upgraded through 0.90.6 - 0.92.2 - 0.94.5-rc1 YCSB, PE, with induced fault. Everything seem to be in order. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: [ANNOUNCE} The 2nd hbase 0.94.5 release candidate is available for download
+1 11 nodes cluster, 50 tables (1GiB-2TiB), 5K regions. Cluster upgraded through 0.90.6 - 0.92.2 - 0.94.5-rc1 YCSB, PE, with induced fault. Everything seem to be in order. Regards, Aditya... -- Email adityakish...@gmail.com WWW http://www.adityakishore.com/ LinkedIn http://www.linkedin.com/in/adityakishore/ On Thu, Feb 14, 2013 at 10:51 AM, Elliott Clark ecl...@apache.org wrote: +1 tested locally. Tried an upgrade from earlier 0.94 everything worked. TestAcid passed. On Wed, Feb 13, 2013 at 10:26 AM, Stack st...@duboce.net wrote: +1 Downloaded, checked out docs, and layout (the sbin dir should probably go). Started it up, did standlone PE and RowCounter. Numbers come out right. Looking in logs, nothing untoward. St.Ack On Fri, Feb 8, 2013 at 12:45 PM, lars hofhansl la...@apache.org wrote: The 2nd 0.94.5 RC is available for download at http://people.apache.org/~larsh/hbase-0.94.5-rc1/ Signed with my code signing key: C7CFE328 0.94.5 is a bug fix release. This RC has 76 issues resolved against it, contributed by 230 individuals. These are mostly bug fixes. Also fixed is a long standing bug in replication along with lots of random test failures. The full list of changes is available here: https://issues.apache.org/jira/browse/HBASE/fixforversion/12323874 Please try out the RC, check out the doc, take it for a spin, etc, and vote +1/-1 by February 15th on whether we should release this as 0.94.5. The release testing spreadsheet is here: https://docs.google.com/spreadsheet/ccc?key=0AvLqcVIqNtlTdFhTZlI5QWhOUzBneHRCazk0Y1FOSUE#gid=0 (if you test this release, please add your test to this spreadsheet, so that we can gauge the coverage) Thanks. -- Lars
Re: ANN: HBase 0.94.2 is available for download
Thanks Lars! I could not find a 0.94.2 tag in SVN though. Kind regards, Aditya Kishore On Thu, Oct 11, 2012 at 10:10 PM, lars hofhansl lhofha...@yahoo.com wrote: The HBase Team is pleased to announce the release of HBase 0.94.2. Download it from your favorite Apache mirror [1]. HBase 0.94.2 is a bug fix release and has 117 issues resolved against it, including some performance improvements. 0.94.2 is the current stable release of HBase. All previous 0.92.x and 0.94.x releases can upgraded to 0.94.2 via a rolling upgrade - in place, without downtime. For a complete list of changes, see release notes [2]. Yours, The HBase Team P.S. Thanks goes out to the over 40 individuals who contributed to this release! 1. http://www.apache.org/dyn/closer.cgi/hbase/ 2. https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753version=12321884
Re: ANN: The second hbase 0.92.2 release candidate is available for download
I tested 0.92.2 this weekend under heavy workload and induced master/region server fail-over. No issues were observed. Kind regards, Aditya Kishore On Tue, Sep 11, 2012 at 11:43 AM, Shrijeet Paliwal shrij...@rocketfuel.comwrote: 0.92.2 has been running for a couple of weeks on all of our production data centers. No issues yet. -Shrijeet On Fri, Sep 7, 2012 at 11:00 AM, Ted Yu yuzhih...@gmail.com wrote: A gentle reminder that Sept. 12th is approaching. I am +1 on this RC: I have done some basic checking in a dev cluster and everything was working. Have a nice weekend. On Fri, Aug 31, 2012 at 8:58 AM, Ted Yu yuzhih...@gmail.com wrote: Hi, The second 0.92.2 RC is available for download here: http://people.apache.org/~tedyu/hbase-0.92.2-rc1/ http://people.apache.org/%7Etedyu/hbase-0.92.2-rc0/ I signed the tarball. My gpg key (signed by stack) is available from pgp.mit.edu. Key id: F48B08A4 http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0xCBC29E86F48B08A4 HBase 0.92 is the stable release. You can do a rolling restart to get your 0.92.x HBase up on this 0.92.2RC. The full list of changes is available here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12319888styleName=TextprojectId=12310753Create=Createatl_token=A5KQ-2QAV-T4JA-FDED|20dd5912cca928036b1ca12dd6edb463220c0a85|lin https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753version=12316419 Please take this RC for a spin, check out the doc, etc, and vote +1/-1 by Sept. 12th on whether we should release this as 0.92.2. Thanks. Ted Yu
Re: ANN: The 2nd hbase 0.94.1 release candidate is available for download
Non-binding +1 Installed in a 5 node cluster, ran some standard and some custom load tests. No issues so far. Regards, Aditya... On Tue, Aug 7, 2012 at 10:16 AM, Stack st...@duboce.net wrote: On Wed, Jul 25, 2012 at 5:09 AM, lars hofhansl lhofha...@yahoo.com wrote: The 2nd 0.94.1 RC is available for download here: http://people.apache.org/~larsh/hbase-0.94.1-rc1/ +1 Downloaded it. Started a standalone instance. Loaded up a couple million of rows. Verified count. Checked logs for errors. Looked good except for the below which is not enough to sink a release. Docs look fine. Tarball should have an md5 published alongside it -- see the hadoop how to release doc. St.Ack 2012-08-07 16:58:43,226 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /127.0.0.1:60142 2012-08-07 16:58:43,232 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x13901f7792d0009 with negotiated timeout 4 for client /127.0.0.1:60142 2012-08-07 16:59:16,777 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13901f7792d0009, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224) at java.lang.Thread.run(Thread.java:662)
Re: Cutting an HBase 0.90.7
Thanks Jon! Do we have a target date yet? Regards, Aditya... On Mon, Jun 25, 2012 at 4:57 PM, Jonathan Hsieh j...@cloudera.com wrote: That's the plan. Wish me luck. :) Jon. On Mon, Jun 25, 2012 at 8:46 AM, Stack st...@duboce.net wrote: On Mon, Jun 25, 2012 at 3:17 PM, Jonathan Hsieh j...@cloudera.com wrote: Hi all, I'd like to start cutting a 0.90.7 release for HBase. +1 I'd suggest that while we can volunteer issues for inclusion, you should be the one to determine what is a blocker or not. Good on you Jon, St.Ack -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com
HBase accessing Tables
HI all, Atlast I am able to install the hadoop/Hbase and was ableto create some tables. when I login to http://10.10.20.1:60010/master.jsp I am able to see all the tables I have created. b(from webpage) But when I try to see from HBASE shell noting is happening. === hbase(main):021:0* list TABLE 11/12/14 18:49:57 FATAL zookeeper.ZKConfig: The server in zoo.cfg cannot be set to localhost in a fully-distributed setup because it won't be reachable. See Getting Started for more information. ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase Here is some help for this command: List all tables in hbase. Optional regular expression parameter could be used to filter the output. Examples: == how can I see the list of tables? how to fix the Issue I am having? Can you please guide me. -Adi --2120775178-2132875201-1323918267=:17723 Content-Type: text/html; charset=us-ascii htmlbodydiv style=color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12ptdivdiv style=padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; margin-top: 8px; margin-right: 0px; margin-bottom: 0px; margin-left: 12px; font-family: arial, helvetica, sans-serif; div style=font-size: 13px; HI all,/divdiv style=font-size: 13px; br/divdiv style=font-size: 13px; Atlast I am able to install the hadoop/Hbase and was ableto create some tables./divdiv style=font-size: 13px; br/divdiv style=font-size: 13px; when I login to http://10.10.20.1:60010/master.jsp/divdiv style=font-size: 13px; I am able to see all the tables I have created. b(from webpage)/divdiv style=font-size: 13px; br/divdiv style=font-size: 13px; But when I try to see from HBASE shell noting is happening./divdiv style=font-size: 13px; br/divdiv style=font-size: 13px; ===/divdiv style=font-size: 13px; hbase(main):021:0* listbr/divdivdivfont class=Apple-style-span size=2TABLE/font/divdivfont class=Apple-style-span size=211/12/14 18:49:57 FATAL zookeeper.ZKConfig: The server in zoo.cfg cannot be set to localhost in a fully-distributed setup because it won't be reachable. See Getting Started for more information./font/divdivfont class=Apple-style-span size=2br/font/divdivfont class=Apple-style-span size=2ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/font/divdivfont class=Apple-style-span size=2br/font/divdivfont class=Apple-style-span size=2Here is some help for this command:/font/divdivfont class=Apple-style-span size=2List all tables in hbase. Optional regular expression parameter could/font/divdivfont class=Apple-style-span size=2be used to filter the output. Examples:/font/divdivfont class=Apple-style-span size=2==/font/divdivfont class=Apple-style-span size=2br/font/divdivfont class=Apple-style-span size=2how can I see the list of tables?/font/divdiv style=font-size: 13px; how to fix the Issue I am having?/divdiv style=font-size: 13px; br/divdiv style=font-size: 13px; Can you please guide me./divdiv style=font-size: 13px; br/divdiv style=font-size: 13px; -Adi/div/div/div/div /div/body/html --2120775178-2132875201-1323918267=:17723--
problem starting HBase regional server
Hi, I have installed the Hadoop/Hbase. Can you please clarify me the following questions 1) I am having my hmaster running. when I start my regional server on the other host. I am getting the following Exception. can you please let me know what I am missing and also how to debug this? == sudo /sbin/service hadoop-hbase-regionserver start starting regionserver, logging to /var/log/hbase/hbase-hbase-regionserver-host123.out Exception in thread main java.lang.NoClassDefFoundError: org/apache/zookeeper/KeeperException at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2442) at java.lang.Class.getMethod0(Class.java:2685) at java.lang.Class.getMethod(Class.java:1620) at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:484) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:476) Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) == 2) My Hadoop is up and running. But when I try to connect to the Hbase from my BRoswer, I keep getting connecting in the status Bar and no response. I tried: http://10.10.10.11:60010 when I checked Hmaster it is running on 10.10.10.11. but the GUI does not come 3) I have a Hadoop and Hbase installed in 5 hosts. if I want this to be given to 25 applications for accessing the DB, what are the configuration changes that I should be taken into consideration like: ( in some applications max handles , threads) similarly what all I should be taken into consideration. 4) when I try to connect to HBase shell it stops and does not proceed. = hbase(main):006:0* list TABLE 11/12/11 18:25:57 FATAL zookeeper.ZKConfig: The server in zoo.cfg cannot be set to localhost in a fully-distributed setup because it won't be reachable. See Getting Started for more information. = Thanks again in advance. -Adi
Re: what is fuse used for?
Thank you David for the clarification. - Original Message - From: David Engfer david.eng...@gmail.com To: dev@hbase.apache.org dev@hbase.apache.org Cc: Sent: Saturday, December 10, 2011 10:27 PM Subject: Re: what is fuse used for? Hadoop fuse allows you to mount HDFS as a virtual unix file system (see the FUSE libraries (filesystem in userspace)) so that normal unix commands can be used to navigate and edit the files and directories. But this is not needed and not suggested (ESP for production); use the hdfs CLI or API instead. I believe it was orig created to have a simple way in development to browse hdfs, or for apps that had to use a Linux filesystem for input/output. With Cloudera's Hue utility, you can now more easily browse hdfs. Thanks, _ David Engfer [sent via iPhone] On Dec 11, 2011, at 12:06 AM, Aditya Kumar adityakumar...@yahoo.com wrote: HI, I am trying to install hadoop/hbase on my new cluster. I came across this rpm hadoop-fuse? what is hadoop-fuse? do we really need it?if so when ? I came across rpm - hadoop-zookeeper and hadoop-zookeeper-server? what are these 2 rpms? (I tough only zookeper server is sufficent to install) Regards, Adi
Re: problem starting HBase regional server
thank you for the response. I do have zookeeper.jar? locate zookeeper.jar /usr/lib/hbase/lib/zookeeper.jar /usr/lib/zookeeper/zookeeper.jar what else I should see to resolve issue (1) - Original Message - From: Stack st...@duboce.net To: dev@hbase.apache.org; Aditya Kumar adityakumar...@yahoo.com Cc: Sent: Sunday, December 11, 2011 8:27 PM Subject: Re: problem starting HBase regional server On Sun, Dec 11, 2011 at 6:29 PM, Aditya Kumar adityakumar...@yahoo.com wrote: 1) I am having my hmaster running. when I start my regional server on the other host. I am getting the following Exception. can you please let me know what I am missing and also how to debug this? == sudo /sbin/service hadoop-hbase-regionserver start starting regionserver, logging to /var/log/hbase/hbase-hbase-regionserver-host123.out Exception in thread main java.lang.NoClassDefFoundError: org/apache/zookeeper/KeeperException at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2442) at java.lang.Class.getMethod0(Class.java:2685) at java.lang.Class.getMethod(Class.java:1620) at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:484) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:476) Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) You are missing the zookeeper jar it would seem. == 2) My Hadoop is up and running. But when I try to connect to the Hbase from my BRoswer, I keep getting connecting in the status Bar and no response. I tried: http://10.10.10.11:60010 when I checked Hmaster it is running on 10.10.10.11. but the GUI does not come Check master logs. If it can't find zk jar, its not going to come up. 3) I have a Hadoop and Hbase installed in 5 hosts. if I want this to be given to 25 applications for accessing the DB, what are the configuration changes that I should be taken into consideration like: ( in some applications max handles , threads) similarly what all I should be taken into consideration. Read the manual that is up on hbase.org. 4) when I try to connect to HBase shell it stops and does not proceed. = hbase(main):006:0* list TABLE 11/12/11 18:25:57 FATAL zookeeper.ZKConfig: The server in zoo.cfg cannot be set to localhost in a fully-distributed setup because it won't be reachable. See Getting Started for more information. = What it says. If listening on localhost, no one remote will be able to connect. Again, check out aforementioned manual. Good luck, St.Ack Thanks again in advance. -Adi
Re: what is fuse used for?
HI, I am trying to install hadoop/hbase on my new cluster. I came across this rpm hadoop-fuse? what is hadoop-fuse? do we really need it?if so when ? I came across rpm - hadoop-zookeeper and hadoop-zookeeper-server? what are these 2 rpms? (I tough only zookeper server is sufficent to install) Regards, Adi
Re: Amazon s3 and Hadoop/Hbase
Thanks Ram. More details: I want to keep the files for now. when you say Analyses what type of analysis need HBase? Can you please mention them.. As I am ina very early state I might not need any other thing than Keeping, BUT if it is needed for Analysies can you please let me know what it is From: Ramkrishna S Vasudevan ramkrishna.vasude...@huawei.com To: dev@hbase.apache.org; 'Aditya Kumar' adityakumar...@yahoo.com Sent: Thursday, November 24, 2011 9:07 PM Subject: RE: Amazon s3 and Hadoop/Hbase Hi If your use case is like just store some files particularly if you need only a big storage medium then S3 should be fine. HBase is a database where you want your data to be added from some specific source like a social site and you want to do some analysis. Regards Ram -Original Message- From: Aditya Kumar [mailto:adityakumar...@yahoo.com] Sent: Friday, November 25, 2011 9:26 AM To: dev@hbase.apache.org Subject: Re: Amazon s3 and Hadoop/Hbase Thanks Sonal for the excellent reply. I am looking for a use case: where in I want to store 1 files and they will be accessed and modified and kept back in the S3. so there will be modifications of Files on S3 and trying to access them. For a use case like this do I need Hbase on S3? (or) if I just use S3, create bucket and keep on adding files and modify them on the need bases From: Sonal Goyal sonalgoy...@gmail.com To: dev@hbase.apache.org; Aditya Kumar adityakumar...@yahoo.com Sent: Thursday, November 24, 2011 7:11 PM Subject: Re: Amazon s3 and Hadoop/Hbase Aditya, S3 is a file storage service. It does provide versioning and key based file access, but it does not provide any processing paradigm, like Map Reduce for Hadoop. HBase provides random access to data - its more like a database in that you can have a key based access for records, and it is a columnar store. S3 can be compared with HDFS, the hadoop distributed file system, and in fact one can spin Hadoop clusters off S3 file system. Maybe if you explain your problem in detail, we can help you further on the mailing list. You can also read up the fundamentals on the Hadoop and HBase wiki, there is a high level introduction which can clear the basic doubts. Hope that helps. Best Regards, Sonal Crux: Reporting for HBase https://github.com/sonalgoyal/crux Nube Technologies http://www.nubetech.co http://in.linkedin.com/in/sonalgoyal On Fri, Nov 25, 2011 at 7:10 AM, Aditya Kumar adityakumar...@yahoo.comwrote: Thanks Li Pi. so, does it mean that s3 is hadoop? can you please let me know what I will be missing if I use s3 only with out Hbase? 2) In s3 there is delete operation, does it really delete the key and there by value? From: Li Pi l...@idle.li To: Aditya Kumar adityakumar...@yahoo.com; dev@hbase.apache.org Sent: Thursday, November 24, 2011 5:29 PM Subject: Re: Amazon s3 and Hadoop/Hbase You can run hbase against s3. On Nov 24, 2011 5:11 PM, Aditya Kumar adityakumar...@yahoo.com wrote: Hi, I am looking at Amazon S3. is it like HDFS/Hadoop? . If it is like Hadoop can we add only data to the tables.. than can we use Hbase? I tried to read online, all I see that S3 will not delete the data and will do versioing (something similar to Hadoop) TIA, Adi
Amazon s3 and Hadoop/Hbase
Hi, I am looking at Amazon S3. is it like HDFS/Hadoop? . If it is like Hadoop can we add only data to the tables.. than can we use Hbase? I tried to read online, all I see that S3 will not delete the data and will do versioing (something similar to Hadoop) TIA, Adi
Re: Amazon s3 and Hadoop/Hbase
Thanks Li Pi. so, does it mean that s3 is hadoop? can you please let me know what I will be missing if I use s3 only with out Hbase? 2) In s3 there is delete operation, does it really delete the key and there by value? From: Li Pi l...@idle.li To: Aditya Kumar adityakumar...@yahoo.com; dev@hbase.apache.org Sent: Thursday, November 24, 2011 5:29 PM Subject: Re: Amazon s3 and Hadoop/Hbase You can run hbase against s3. On Nov 24, 2011 5:11 PM, Aditya Kumar adityakumar...@yahoo.com wrote: Hi, I am looking at Amazon S3. is it like HDFS/Hadoop? . If it is like Hadoop can we add only data to the tables.. than can we use Hbase? I tried to read online, all I see that S3 will not delete the data and will do versioing (something similar to Hadoop) TIA, Adi
Re: Amazon s3 and Hadoop/Hbase
Thanks Sonal for the excellent reply. I am looking for a use case: where in I want to store 1 files and they will be accessed and modified and kept back in the S3. so there will be modifications of Files on S3 and trying to access them. For a use case like this do I need Hbase on S3? (or) if I just use S3, create bucket and keep on adding files and modify them on the need bases From: Sonal Goyal sonalgoy...@gmail.com To: dev@hbase.apache.org; Aditya Kumar adityakumar...@yahoo.com Sent: Thursday, November 24, 2011 7:11 PM Subject: Re: Amazon s3 and Hadoop/Hbase Aditya, S3 is a file storage service. It does provide versioning and key based file access, but it does not provide any processing paradigm, like Map Reduce for Hadoop. HBase provides random access to data - its more like a database in that you can have a key based access for records, and it is a columnar store. S3 can be compared with HDFS, the hadoop distributed file system, and in fact one can spin Hadoop clusters off S3 file system. Maybe if you explain your problem in detail, we can help you further on the mailing list. You can also read up the fundamentals on the Hadoop and HBase wiki, there is a high level introduction which can clear the basic doubts. Hope that helps. Best Regards, Sonal Crux: Reporting for HBase https://github.com/sonalgoyal/crux Nube Technologies http://www.nubetech.co http://in.linkedin.com/in/sonalgoyal On Fri, Nov 25, 2011 at 7:10 AM, Aditya Kumar adityakumar...@yahoo.comwrote: Thanks Li Pi. so, does it mean that s3 is hadoop? can you please let me know what I will be missing if I use s3 only with out Hbase? 2) In s3 there is delete operation, does it really delete the key and there by value? From: Li Pi l...@idle.li To: Aditya Kumar adityakumar...@yahoo.com; dev@hbase.apache.org Sent: Thursday, November 24, 2011 5:29 PM Subject: Re: Amazon s3 and Hadoop/Hbase You can run hbase against s3. On Nov 24, 2011 5:11 PM, Aditya Kumar adityakumar...@yahoo.com wrote: Hi, I am looking at Amazon S3. is it like HDFS/Hadoop? . If it is like Hadoop can we add only data to the tables.. than can we use Hbase? I tried to read online, all I see that S3 will not delete the data and will do versioing (something similar to Hadoop) TIA, Adi
Hbase -question
Hi, I am new to Hbase and trying to understand the internals. I read some documents and still confused. can any one please clarify me : what is HLog,storefile.Hfile and store? I am looking for a simple flow to understand what is written to these files. looking for a simple PUT and GET I was aware of the WAL file in general. your clarifications will be really appreciated and also sorry for asking the basic question -Adi
Re: Hbase -question
Thanks for the reply and explanation. Can you explain me the following: HFile is having the raw data blocks and also the indices to find the the blocks about the file. i.e, It has the raw data blocks and indices of these blocks? will it not be heavy to have multiple copies? I was of the understanding that : Hfile will have only the filename with the indices of the blocks. can you please clarify me more in detail why we need to have the raw data also? From: Stack st...@duboce.net To: dev@hbase.apache.org; Aditya Kumar adityakumar...@yahoo.com Sent: Monday, November 14, 2011 10:37 PM Subject: Re: Hbase -question On Mon, Nov 14, 2011 at 10:25 PM, Aditya Kumar adityakumar...@yahoo.com wrote: can any one please clarify me : what is HLog,storefile.Hfile and store? HLog is the name of our WAL implementation (see in code the wal package, see how the WAL implementation is done in the HLog class). HFile is about raw data blocks and indices to find those blocks as well as metadata about the file. Ideally it knows nothing about the upper hbase tiers. StoreFile then is the glue between hbase and its raw storage HFile. Store is the implementation of a Column Family (The Store class should have been called ColumnFamily) Sorry our naming is a little less clear than it should be. St.Ack
Hadoop real time
Hi, I am reading an article on Hadoop and hbase. Can any one explain me the following statements.. - Hadoop is not used for real time processing? I think we should be able to use it for Real time processing also... Can you please tell me why Hadoop is said not to be used for Real time processing of data? -- Hadoop Augments Existing data bases? I as thinking we can replace the DB with Hdoop...I do not see any issue?
