[jira] [Commented] (HBASE-4920) We need a mascot, a totem
[ https://issues.apache.org/jira/browse/HBASE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932744#comment-13932744 ] Roman Shaposhnik commented on HBASE-4920: - [~saint@gmail.com] yup either google docs or survey monkey. And good call on combinations. > We need a mascot, a totem > - > > Key: HBASE-4920 > URL: https://issues.apache.org/jira/browse/HBASE-4920 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: HBase Orca Logo.jpg, Orca_479990801.jpg, Screen shot > 2011-11-30 at 4.06.17 PM.png, apache hbase orca logo_Proof 3.pdf, apache > logo_Proof 8.pdf, krake.zip, more_orcas.png, more_orcas2.png, photo (2).JPG, > plus_orca.png > > > We need a totem for our t-shirt that is yet to be printed. O'Reilly owns the > Clyesdale. We need something else. > We could have a fluffy little duck that quacks 'hbase!' when you squeeze it > and we could order boxes of them from some off-shore sweatshop that > subcontracts to a contractor who employs child labor only. > Or we could have an Orca (Big!, Fast!, Killer!, and in a poem that Marcy from > Salesforce showed me, that was a bit too spiritual for me to be seen quoting > here, it had the Orca as the 'Guardian of the Cosmic Memory': i.e. in > translation, bigdata). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10681) Allow Hbase artifacts to deploy to remote repo other than apache
[
https://issues.apache.org/jira/browse/HBASE-10681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932192#comment-13932192
]
Roman Shaposhnik commented on HBASE-10681:
--
[~rguo] here's what I'm thinking:
# this is already implemented for snapshots in a parent Apache pom: just pass
-DdistMgmtSnapshotsUrl=XXX to the build
# you can always utilize
-DaltDeploymentRepository/altReleaseDeploymentRepository/altSnapshotDeploymentRepository
to redirect the deployment
The trouble that I see with your approach is that once you add the override for
a non-snapshot repo to the HBase pom, I don't think there's any way to keep the
one coming from the parent pom as the default.
> Allow Hbase artifacts to deploy to remote repo other than apache
>
>
> Key: HBASE-10681
> URL: https://issues.apache.org/jira/browse/HBASE-10681
> Project: HBase
> Issue Type: Improvement
> Components: build
>Reporter: Guo Ruijing
>
> pom.xml in hbase write as
>
>
> hbase.apache.org
> HBase Website at hbase.apache.org
>
> file:///tmp
>
> We expect pom.xml in hbase like hadoop can be:
>
>
> ${distMgmtStagingId}
> ${distMgmtStagingName}
> ${distMgmtStagingUrl}
>
>
> ${distMgmtSnapshotsId}
> ${distMgmtSnapshotsName}
> ${distMgmtSnapshotsUrl}
>
>
> hbase.apache.org
> HBase Website at hbase.apache.org
>
> file:///tmp
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9442) HLogKey(walKey) constructor needs to be either removed, deprecated or fixed
[ https://issues.apache.org/jira/browse/HBASE-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13759243#comment-13759243 ] Roman Shaposhnik commented on HBASE-9442: - [~jdcryans] my thoughts exactly! And given that it has been broken for quite some time now I doubt there's anybody actually using it. Hence it should be safe to remove. > HLogKey(walKey) constructor needs to be either removed, deprecated or fixed > --- > > Key: HBASE-9442 > URL: https://issues.apache.org/jira/browse/HBASE-9442 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Roman Shaposhnik > Attachments: 9442.txt, 9442.txt, 9442v2.txt > > > It is currently impossible to use HLogKey(walKey) because of the NPE that > occurs due to constructor itself not initializing clusterIds but instead > immediately calling readFieldsFromPb which then references it: > > https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java#L475 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9443) ReplicationProtbufUtil.toHLogEntries needs to be either removed or deprecated
Roman Shaposhnik created HBASE-9443: --- Summary: ReplicationProtbufUtil.toHLogEntries needs to be either removed or deprecated Key: HBASE-9443 URL: https://issues.apache.org/jira/browse/HBASE-9443 Project: HBase Issue Type: Improvement Components: Replication Affects Versions: 0.96.0 Reporter: Roman Shaposhnik Currently, due to HBASE-9442 ReplicationProtbufUtil.toHLogEntries is rendered useless. Given that it isn't really used by HBase itself, perhaps it needs to be removed/deprecated? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9442) HLogKey(walKey) constructor needs to be either removed, deprecated or fixed
Roman Shaposhnik created HBASE-9442: --- Summary: HLogKey(walKey) constructor needs to be either removed, deprecated or fixed Key: HBASE-9442 URL: https://issues.apache.org/jira/browse/HBASE-9442 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: Roman Shaposhnik It is currently impossible to use HLogKey(walKey) because of the NPE that occurs due to constructor itself not initializing clusterIds but instead immediately calling readFieldsFromPb which then references it: https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java#L475 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8604) improve reporting of incorrect peer address in replication
Roman Shaposhnik created HBASE-8604: --- Summary: improve reporting of incorrect peer address in replication Key: HBASE-8604 URL: https://issues.apache.org/jira/browse/HBASE-8604 Project: HBase Issue Type: Improvement Components: Replication Affects Versions: 0.94.6 Reporter: Roman Shaposhnik Priority: Minor I was running some replication code that incorrectly advertised the peer address for replication. HBase complained that the format of the record was NOT what it was expecting but it didn't include what it saw in the exception message. Including that string would help cutting down the time it takes to debug issues like that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497468#comment-13497468 ] Roman Shaposhnik commented on HBASE-6929: - A couple of points # I was saying that if HBase project would not want to get into the business of certifying that HBase is known to work against Hadoop 2 it can safely NOT publish the artifacts and expect downstream projects to build local copies of HBase # Bigtop 0.4.0 and 0.5.0 do fair amount of testing of the HBase 0.94.X and Hadoop 2.0.X combination # If you keep #1 and #2 in mind -- you might be interested in outsourcing the management of the hadoop2 HBase artifact to the Bigtop project. If that is an attractive option -- lets talk. > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481850#comment-13481850 ] Roman Shaposhnik commented on HBASE-6929: - @stack if HBASE-4850 is not an option I believe you guys are on the right track. You definitely don't want to use classifier for this so that leaves mucking with the version string. As long as you come up with a reasonable encoding strategy for pointing at various artifacts (e.g. hadoop2-secure, etc.) you can append that to the version string. Watch out for things in the pom.xml that need to be tweaked for each of these combinations -- I got bitten by default settings in specially versioned pom files a couple of times. > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4850) hbase tests need to be made Hadoop version agnostic
[ https://issues.apache.org/jira/browse/HBASE-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481846#comment-13481846 ] Roman Shaposhnik commented on HBASE-4850: - if that's the case I think this JIRA needs to be closed as won't fix :-( > hbase tests need to be made Hadoop version agnostic > --- > > Key: HBASE-4850 > URL: https://issues.apache.org/jira/browse/HBASE-4850 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.92.0, 0.92.1, 0.94.0 >Reporter: Roman Shaposhnik >Priority: Critical > > Currently it is possible to have a single hbase jar that can work with > multiple versions of Hadoop. It would be nice if hbase-test.jar also followed > the suit. For now I'm aware of the following problems (but there could be > more): > 1. org.apache.hadoop.hbase.mapreduce.NMapInputFormat is failing because > org.apache.hadoop.mapreduce.JobContext is either class or interface > depending on which version of Hadoop you compile it against. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478566#comment-13478566 ] Roman Shaposhnik commented on HBASE-6929: - bq. If between hadoop1 and hadoop2, then I don't think we can resolve this easily It is not super trivial, but it isn't insurmountable either. Besides, we can borrow generously from some of the projects that did some of it already. In short -- it definitely can be done, but it requires more than just yours truly ;-) > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478474#comment-13478474 ] Roman Shaposhnik commented on HBASE-6929: - Long time ago I filed HBASE-4850 but haven't had much time to devote to it since then. Is there any chance that somebody can help me working on it? > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478354#comment-13478354 ] Roman Shaposhnik commented on HBASE-6929: - So, isn't the lack of secure jars as much a problem as the lack of the ones compiled against different versions of Hadoop? I'm just trying to understand the scope of different 'twists' of HBase that one would have to know/care about. > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478309#comment-13478309 ] Roman Shaposhnik commented on HBASE-6929: - Aren't you guys already publishing secure/unsecure bits? Adding a dedicated version would make it 4 then. > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6803) script hbase should add JAVA_LIBRARY_PATH to LD_LIBRARY_PATH
[
https://issues.apache.org/jira/browse/HBASE-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457887#comment-13457887
]
Roman Shaposhnik commented on HBASE-6803:
-
Looks good, but I'd recommend quoting it like this:
{noformat}
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH"
{noformat}
> script hbase should add JAVA_LIBRARY_PATH to LD_LIBRARY_PATH
>
>
> Key: HBASE-6803
> URL: https://issues.apache.org/jira/browse/HBASE-6803
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: trunk-6803.patch
>
>
> Snappy SO fails to load properly if LD_LIBRARY_PATH does not include the path
> where snappy SO is.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6567) make memory locking configuration of regioservers more flexible
[ https://issues.apache.org/jira/browse/HBASE-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433304#comment-13433304 ] Roman Shaposhnik commented on HBASE-6567: - It would definitely address the issue that is lurking (mainly the fact that the proposed variable serves too purposes -- enabling the feature to begin with and telling the system what is the target user). That said, I feel that HBASE_REGIONSERVER_MLOCK_USER is also not ideal, since it doesn't really communicate clearly what is happening. In my initial proposal I was trying to minimize the # of variables that we would have to introduce but perhaps we should just bite the bullet and have two: * HBASE_REGIONSERVER_MLOCK (binary: empty/not set -- disabled, non empty -- enabled) * HBASE_REGIONSERVER_UID (string: a target user for the running process regardless of any other feature that is enabled) > make memory locking configuration of regioservers more flexible > --- > > Key: HBASE-6567 > URL: https://issues.apache.org/jira/browse/HBASE-6567 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 0.96.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 0.96.0 > > > The current implementation of the memory locking feature of regisoservers has > a downside of not being flexible to configure for permanent use. Sure there > is a --mlock flag but that needs to be explicitly passed on every invocation > and thus require extra steps to be configured for permanent use (IOW, there's > not a single env variable I can set to have a desired effect). The only other > alternative -- the explicit setting of HBASE_REGIONSERVER_OPTS -- has a > downside of being pretty cryptic to the novice user and has a killer problem > of not explicitly telling higher level scripts (like init.d or upstart ones) > which user the initial hbase process should be executed as. > I propose a very simple solution (which is essentially making --mlock setting > into an env. variable): add a variable called HBASE_REGIONSERVER_MLOCK that > can be set in hbase-env.sh and has the following semantics: >* [default] not set: mlocking feature is disabled >* set but empty: mlocking feature is enabled and the target user is hbase >* set and not empty: mlocking feature is enabled and the target user is > the value of the variable > Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6567) make memory locking configuration of regioservers more flexible
Roman Shaposhnik created HBASE-6567: --- Summary: make memory locking configuration of regioservers more flexible Key: HBASE-6567 URL: https://issues.apache.org/jira/browse/HBASE-6567 Project: HBase Issue Type: Improvement Components: scripts Affects Versions: 0.96.0 Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik Fix For: 0.96.0 The current implementation of the memory locking feature of regisoservers has a downside of not being flexible to configure for permanent use. Sure there is a --mlock flag but that needs to be explicitly passed on every invocation and thus require extra steps to be configured for permanent use (IOW, there's not a single env variable I can set to have a desired effect). The only other alternative -- the explicit setting of HBASE_REGIONSERVER_OPTS -- has a downside of being pretty cryptic to the novice user and has a killer problem of not explicitly telling higher level scripts (like init.d or upstart ones) which user the initial hbase process should be executed as. I propose a very simple solution (which is essentially making --mlock setting into an env. variable): add a variable called HBASE_REGIONSERVER_MLOCK that can be set in hbase-env.sh and has the following semantics: * [default] not set: mlocking feature is disabled * set but empty: mlocking feature is enabled and the target user is hbase * set and not empty: mlocking feature is enabled and the target user is the value of the variable Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6332) improve HBase's pom file for better integration with downstream ivy projects
[ https://issues.apache.org/jira/browse/HBASE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HBASE-6332: Attachment: HBASE-6332.patch.txt Attaching a patch with two simple workarounds that are harmless to the HBase itself, but extremely useful to the downstream. > improve HBase's pom file for better integration with downstream ivy projects > > > Key: HBASE-6332 > URL: https://issues.apache.org/jira/browse/HBASE-6332 > Project: HBase > Issue Type: Improvement > Components: build >Affects Versions: 0.94.1 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 0.94.1 > > Attachments: HBASE-6332.patch.txt > > > Currently there are 2 issues affecting the downstream ivy projects: ># no default value for slf4j.version ># dependency on a non-standard junit artifact > I suggest we correct both of these in order to ensure the smooth upgrade path > for things like Sqoop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6332) improve HBase's pom file for better integration with downstream ivy projects
[ https://issues.apache.org/jira/browse/HBASE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HBASE-6332: Status: Patch Available (was: Open) > improve HBase's pom file for better integration with downstream ivy projects > > > Key: HBASE-6332 > URL: https://issues.apache.org/jira/browse/HBASE-6332 > Project: HBase > Issue Type: Improvement > Components: build >Affects Versions: 0.94.1 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 0.94.1 > > Attachments: HBASE-6332.patch.txt > > > Currently there are 2 issues affecting the downstream ivy projects: ># no default value for slf4j.version ># dependency on a non-standard junit artifact > I suggest we correct both of these in order to ensure the smooth upgrade path > for things like Sqoop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6332) improve HBase's pom file for better integration with downstream ivy projects
Roman Shaposhnik created HBASE-6332: --- Summary: improve HBase's pom file for better integration with downstream ivy projects Key: HBASE-6332 URL: https://issues.apache.org/jira/browse/HBASE-6332 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.1 Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik Fix For: 0.94.1 Currently there are 2 issues affecting the downstream ivy projects: # no default value for slf4j.version # dependency on a non-standard junit artifact I suggest we correct both of these in order to ensure the smooth upgrade path for things like Sqoop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4920) We need a mascot, a totem
[ https://issues.apache.org/jira/browse/HBASE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402498#comment-13402498 ] Roman Shaposhnik commented on HBASE-4920: - @Jonathan "Obviously, You're Not a Golfer" (tm) ;-) > We need a mascot, a totem > - > > Key: HBASE-4920 > URL: https://issues.apache.org/jira/browse/HBASE-4920 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: HBase Orca Logo.jpg, Orca_479990801.jpg, Screen shot > 2011-11-30 at 4.06.17 PM.png, apache hbase orca logo_Proof 3.pdf, apache > logo_Proof 8.pdf, krake.zip, more_orcas.png, more_orcas2.png, photo (2).JPG, > plus_orca.png > > > We need a totem for our t-shirt that is yet to be printed. O'Reilly owns the > Clyesdale. We need something else. > We could have a fluffy little duck that quacks 'hbase!' when you squeeze it > and we could order boxes of them from some off-shore sweatshop that > subcontracts to a contractor who employs child labor only. > Or we could have an Orca (Big!, Fast!, Killer!, and in a poem that Marcy from > Salesforce showed me, that was a bit too spiritual for me to be seen quoting > here, it had the Orca as the 'Guardian of the Cosmic Memory': i.e. in > translation, bigdata). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6201) HBase integration/system tests
[
https://issues.apache.org/jira/browse/HBASE-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396383#comment-13396383
]
Roman Shaposhnik commented on HBASE-6201:
-
@Enis,
in that case I'd strongly encourage the community to take a serious look at
BIGTOP-635. This is exactly what we're hoping to achieve with it.
> HBase integration/system tests
> --
>
> Key: HBASE-6201
> URL: https://issues.apache.org/jira/browse/HBASE-6201
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> Integration and general system tests have been discussed previously, and the
> conclusion is that we need to unify how we do "release candidate" testing
> (HBASE-6091).
> In this issue, I would like to discuss and agree on a general plan, and open
> subtickets for execution so that we can carry out most of the tests in
> HBASE-6091 automatically.
> Initially, here is what I have in mind:
> 1. Create hbase-it (or hbase-tests) containing forward port of HBASE-4454
> (without any tests). This will allow integration test to be run with
> {code}
> mvn verify
> {code}
> 2. Add ability to run all integration/system tests on a given cluster. Smt
> like:
> {code}
> mvn verify -Dconf=/etc/hbase/conf/
> {code}
> should run the test suite on the given cluster. (Right now we can launch some
> of the tests (TestAcidGuarantees) from command line). Most of the system
> tests will be client side, and interface with the cluster through public
> APIs. We need a tool on top of MiniHBaseCluster or improve
> HBaseTestingUtility, so that tests can interface with the mini cluster or the
> actual cluster uniformly.
> 3. Port candidate unit tests to the integration tests module. Some of the
> candidates are:
> - TestAcidGuarantees / TestAtomicOperation
> - TestRegionBalancing (HBASE-6053)
> - TestFullLogReconstruction
> - TestMasterFailover
> - TestImportExport
> - TestMultiVersions / TestKeepDeletes
> - TestFromClientSide
> - TestShell and src/test/ruby
> - TestRollingRestart
> - Test**OnCluster
> - Balancer tests
> These tests should continue to be run as unit tests w/o any change in
> semantics. However, given an actual cluster, they should use that, instead of
> spinning a mini cluster.
> 4. Add more tests, especially, long running ingestion tests (goraci, BigTop's
> TestLoadAndVerify, LoadTestTool), and chaos monkey style fault tests.
> All suggestions welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6201) HBase integration/system tests
[
https://issues.apache.org/jira/browse/HBASE-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396346#comment-13396346
]
Roman Shaposhnik commented on HBASE-6201:
-
Feel free to chime in: BIGTOP-635
This is meant to cover HBase and HDFS use cases to begin with, but also should
be generic enough to tackle just about anything. Given the level of ambition --
the more code we can reuse the better. Hence anybody familiar with real Cluster
Management frameworks are welcome to chime in and give us feedback on how
realistic such a reuse really is.
> HBase integration/system tests
> --
>
> Key: HBASE-6201
> URL: https://issues.apache.org/jira/browse/HBASE-6201
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> Integration and general system tests have been discussed previously, and the
> conclusion is that we need to unify how we do "release candidate" testing
> (HBASE-6091).
> In this issue, I would like to discuss and agree on a general plan, and open
> subtickets for execution so that we can carry out most of the tests in
> HBASE-6091 automatically.
> Initially, here is what I have in mind:
> 1. Create hbase-it (or hbase-tests) containing forward port of HBASE-4454
> (without any tests). This will allow integration test to be run with
> {code}
> mvn verify
> {code}
> 2. Add ability to run all integration/system tests on a given cluster. Smt
> like:
> {code}
> mvn verify -Dconf=/etc/hbase/conf/
> {code}
> should run the test suite on the given cluster. (Right now we can launch some
> of the tests (TestAcidGuarantees) from command line). Most of the system
> tests will be client side, and interface with the cluster through public
> APIs. We need a tool on top of MiniHBaseCluster or improve
> HBaseTestingUtility, so that tests can interface with the mini cluster or the
> actual cluster uniformly.
> 3. Port candidate unit tests to the integration tests module. Some of the
> candidates are:
> - TestAcidGuarantees / TestAtomicOperation
> - TestRegionBalancing (HBASE-6053)
> - TestFullLogReconstruction
> - TestMasterFailover
> - TestImportExport
> - TestMultiVersions / TestKeepDeletes
> - TestFromClientSide
> - TestShell and src/test/ruby
> - TestRollingRestart
> - Test**OnCluster
> - Balancer tests
> These tests should continue to be run as unit tests w/o any change in
> semantics. However, given an actual cluster, they should use that, instead of
> spinning a mini cluster.
> 4. Add more tests, especially, long running ingestion tests (goraci, BigTop's
> TestLoadAndVerify, LoadTestTool), and chaos monkey style fault tests.
> All suggestions welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6201) HBase integration/system tests
[
https://issues.apache.org/jira/browse/HBASE-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396329#comment-13396329
]
Roman Shaposhnik commented on HBASE-6201:
-
bq. I think your categorization, and my comments above are telling the same
thing, no confusion there.
Great to hear -- in that case the implementation is the next level to go to.
bq. At this point, we only need starting / stopping deamons kind of
functionality, assuming the cluster is already deployed.
bq. On the other side, if we provide a "mvn verify" in hbase-it module to run
the tests on the actual cluster, I assume BigTop can leverage this to carry out
the tests.
Basically, all that Bigtop needs is a maven artifact that we can plug in our
infrastructure. We won't be using the 'mvn verify' as it is implemented in
HBase's pom.xml but rather hooking the very same artifact to our Maven
execution framework.
Are you saying that you would like the tests themeselves to get involved in the
lifecycle of each service? Like bringing them up and down, etc?
This an area I'm really interested in providing a framework for in Bigtop. I'm
about to open up a JIRA for that with the hope that we can cook something
usable up rather quickly. The key point is that I don't want tests to have any
logic that concerns itself with ssh/etc. It needs to be a sort of an
agent-based framework that allows tests to query the topology of the cluster
and also perform actions on the nodes of that topology in a most generic sense.
> HBase integration/system tests
> --
>
> Key: HBASE-6201
> URL: https://issues.apache.org/jira/browse/HBASE-6201
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> Integration and general system tests have been discussed previously, and the
> conclusion is that we need to unify how we do "release candidate" testing
> (HBASE-6091).
> In this issue, I would like to discuss and agree on a general plan, and open
> subtickets for execution so that we can carry out most of the tests in
> HBASE-6091 automatically.
> Initially, here is what I have in mind:
> 1. Create hbase-it (or hbase-tests) containing forward port of HBASE-4454
> (without any tests). This will allow integration test to be run with
> {code}
> mvn verify
> {code}
> 2. Add ability to run all integration/system tests on a given cluster. Smt
> like:
> {code}
> mvn verify -Dconf=/etc/hbase/conf/
> {code}
> should run the test suite on the given cluster. (Right now we can launch some
> of the tests (TestAcidGuarantees) from command line). Most of the system
> tests will be client side, and interface with the cluster through public
> APIs. We need a tool on top of MiniHBaseCluster or improve
> HBaseTestingUtility, so that tests can interface with the mini cluster or the
> actual cluster uniformly.
> 3. Port candidate unit tests to the integration tests module. Some of the
> candidates are:
> - TestAcidGuarantees / TestAtomicOperation
> - TestRegionBalancing (HBASE-6053)
> - TestFullLogReconstruction
> - TestMasterFailover
> - TestImportExport
> - TestMultiVersions / TestKeepDeletes
> - TestFromClientSide
> - TestShell and src/test/ruby
> - TestRollingRestart
> - Test**OnCluster
> - Balancer tests
> These tests should continue to be run as unit tests w/o any change in
> semantics. However, given an actual cluster, they should use that, instead of
> spinning a mini cluster.
> 4. Add more tests, especially, long running ingestion tests (goraci, BigTop's
> TestLoadAndVerify, LoadTestTool), and chaos monkey style fault tests.
> All suggestions welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6201) HBase integration/system tests
[
https://issues.apache.org/jira/browse/HBASE-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396272#comment-13396272
]
Roman Shaposhnik commented on HBASE-6201:
-
Sorry for chiming in late, but here's how I see it after quite a bit of
internal discussions with some of the HBase and HDFS devs. First of all, lets
not got caught up in terminology of what is a unit test, what is a functional
test, etc. If we assume this stance than all the tests, essentially, fall under
the 3 main categories:
# tests that muck about with the internals of the particular single project
(HDFS, HBase, etc) using things like private APIs (or sometimes even things
like reflections, etc to really get into the guts of the system)
# tests that concern themselves with a single project (HDFS, HBase, etc) but
use only public APIs AND don't use
# tests that concern themselves with multiple projects at the same time
(imagine a test that submits an Oozie workflow that has some Pig and Hive
actions actively manipulating data in HBase) but only using public APIs
It is pretty clear that #3 definitely belongs to Bigtop while #1 definitely
belongs to individual projects. For quite some time I was thinking that #2
belongs to Bigtop testbase as well, but I've changed my mind. I now believe
that such tests should reside in individual projects and:
# be clearly marked as such not to be confused with class #1 (test suites,
test lists, naming convention work, etc)
# be written/refactored in such a way that doesn't tie them to a particular
deployment strategy. IOW they should assume the subsystem to be deployed.
# be hooked up to the project build's system in such a way that takes care of
deploying the least amount of a system to make them run (e.g. MiniDFS, MiniMR,
etc.)
Thus if HBase can follow these rules and have a subset of tests that can be
executed in different envs. both HBase core devs and bigger Bigtop dev
community win, since we can leverage each other's work.
Makes sense? If it does I can help with re-factoring.
> HBase integration/system tests
> --
>
> Key: HBASE-6201
> URL: https://issues.apache.org/jira/browse/HBASE-6201
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> Integration and general system tests have been discussed previously, and the
> conclusion is that we need to unify how we do "release candidate" testing
> (HBASE-6091).
> In this issue, I would like to discuss and agree on a general plan, and open
> subtickets for execution so that we can carry out most of the tests in
> HBASE-6091 automatically.
> Initially, here is what I have in mind:
> 1. Create hbase-it (or hbase-tests) containing forward port of HBASE-4454
> (without any tests). This will allow integration test to be run with
> {code}
> mvn verify
> {code}
> 2. Add ability to run all integration/system tests on a given cluster. Smt
> like:
> {code}
> mvn verify -Dconf=/etc/hbase/conf/
> {code}
> should run the test suite on the given cluster. (Right now we can launch some
> of the tests (TestAcidGuarantees) from command line). Most of the system
> tests will be client side, and interface with the cluster through public
> APIs. We need a tool on top of MiniHBaseCluster or improve
> HBaseTestingUtility, so that tests can interface with the mini cluster or the
> actual cluster uniformly.
> 3. Port candidate unit tests to the integration tests module. Some of the
> candidates are:
> - TestAcidGuarantees / TestAtomicOperation
> - TestRegionBalancing (HBASE-6053)
> - TestFullLogReconstruction
> - TestMasterFailover
> - TestImportExport
> - TestMultiVersions / TestKeepDeletes
> - TestFromClientSide
> - TestShell and src/test/ruby
> - TestRollingRestart
> - Test**OnCluster
> - Balancer tests
> These tests should continue to be run as unit tests w/o any change in
> semantics. However, given an actual cluster, they should use that, instead of
> spinning a mini cluster.
> 4. Add more tests, especially, long running ingestion tests (goraci, BigTop's
> TestLoadAndVerify, LoadTestTool), and chaos monkey style fault tests.
> All suggestions welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4821) A fully automated comprehensive distributed integration test for HBase
[ https://issues.apache.org/jira/browse/HBASE-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261981#comment-13261981 ] Roman Shaposhnik commented on HBASE-4821: - @Enis, @Jon, yes you'd have to provide knobs in the tests as to what the desired data size is. TestLoadAndVerify already does that and all the others should follow. > A fully automated comprehensive distributed integration test for HBase > -- > > Key: HBASE-4821 > URL: https://issues.apache.org/jira/browse/HBASE-4821 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Critical > > To properly verify that a particular version of HBase is good for production > deployment we need a better way to do real cluster testing after incremental > changes. Running unit tests is good, but we also need to deploy HBase to a > cluster, run integration tests, load tests, Thrift server tests, kill some > region servers, kill the master, and produce a report. All of this needs to > happen in 20-30 minutes with minimal manual intervention. I think this way we > can combine agile development with high stability of the codebase. I am > envisioning a high-level framework written in a scripting language (e.g. > Python) that would abstract external operations such as "deploy to test > cluster", "kill a particular server", "run load test A", "run load test B" > (we already have a few kinds of load tests implemented in Java, and we could > write a Thrift load test in Python). This tool should also produce > intermediate output, allowing to catch problems early and restart the test. > No implementation has yet been done. Any ideas or suggestions are welcome. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4821) A fully automated comprehensive distributed integration test for HBase
[ https://issues.apache.org/jira/browse/HBASE-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261980#comment-13261980 ] Roman Shaposhnik commented on HBASE-4821: - @Enis, I'm not sure what API are yous asking for. Bigtop currently provides a way of quickly deploying a fully distributed clusters using puppet code. You really don't want to burden your tests with deployment logic -- hence a choice of a full fledged CM system: puppet. We're already using that for nightly testing of the entire Hadoop stack on ~5 nodes fully distributed clusters that we sping on-demand on EC2. Here's a job that does that: http://bigtop01.cloudera.org:8080/view/Test/job/DeployCluster/ And here's how a test runs looks like (bear with me while I fix a couple of things that got broken after Bigtop's trunk migration to Hadoop 2.X): http://bigtop01.cloudera.org:8080/view/Test/job/SmokeCluster/ > A fully automated comprehensive distributed integration test for HBase > -- > > Key: HBASE-4821 > URL: https://issues.apache.org/jira/browse/HBASE-4821 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Critical > > To properly verify that a particular version of HBase is good for production > deployment we need a better way to do real cluster testing after incremental > changes. Running unit tests is good, but we also need to deploy HBase to a > cluster, run integration tests, load tests, Thrift server tests, kill some > region servers, kill the master, and produce a report. All of this needs to > happen in 20-30 minutes with minimal manual intervention. I think this way we > can combine agile development with high stability of the codebase. I am > envisioning a high-level framework written in a scripting language (e.g. > Python) that would abstract external operations such as "deploy to test > cluster", "kill a particular server", "run load test A", "run load test B" > (we already have a few kinds of load tests implemented in Java, and we could > write a Thrift load test in Python). This tool should also produce > intermediate output, allowing to catch problems early and restart the test. > No implementation has yet been done. Any ideas or suggestions are welcome. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4821) A fully automated comprehensive distributed integration test for HBase
[ https://issues.apache.org/jira/browse/HBASE-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261885#comment-13261885 ] Roman Shaposhnik commented on HBASE-4821: - @Enis, I had a really nice chat with Jon yesterday and we arrived at a common understanding that the tests in general fall into 3 distinct categories (please note that this categories classify test *implementation* and not whether they are used as part of the mvn test, mvn verify or Bigtop's test infra -- more on that later): # pure unit tests -- they reach into the guts of the implementation and use non-public APIs. There's absolutely no way to run that testcode on anything but MiniHBase/MiniDFS/MiniMR. Bigtop has no role to play in helping HBase community with developing/maintaining/executing those tests. # HBase-specific functional tests -- these are the tests that only use public APIs and don't muck about with internals. They are, however, only concerned with HBase itself. IOW, a test that wants to verify that you can submit an Oozie workflow that has Hive->HBASE->Pig pipeline does not fall into this category # Integration tests -- these are the multi-component tests that exercise not just HBase but a # of different components. An above example of the Oozie workflow falls into this category. Here's how an ideal situation looks from Bigtop's perspective: * you guys totally take care of #1 and you implement it as usual unit tests. * Bigtop (with your help) takes care of #3. It simply makes no sense to reproduce the same infra at the HBase level. * A proposal on #2 is this -- these tests belong to HBase. However, they have to be clearly marked as belonging to the functional class AND they have to utilize a very thin shim layer so you can use them in an mvn verify context and we can reuse them in Bigtop running against a fully distributed beefy clusters. At this point I'm convinced that TestLoadAndVerify should be the first example of this class and it should reside in HBase codebase (yet be available to Bigtop). Let me know if this makes sense. > A fully automated comprehensive distributed integration test for HBase > -- > > Key: HBASE-4821 > URL: https://issues.apache.org/jira/browse/HBASE-4821 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin >Priority: Critical > > To properly verify that a particular version of HBase is good for production > deployment we need a better way to do real cluster testing after incremental > changes. Running unit tests is good, but we also need to deploy HBase to a > cluster, run integration tests, load tests, Thrift server tests, kill some > region servers, kill the master, and produce a report. All of this needs to > happen in 20-30 minutes with minimal manual intervention. I think this way we > can combine agile development with high stability of the codebase. I am > envisioning a high-level framework written in a scripting language (e.g. > Python) that would abstract external operations such as "deploy to test > cluster", "kill a particular server", "run load test A", "run load test B" > (we already have a few kinds of load tests implemented in Java, and we could > write a Thrift load test in Python). This tool should also produce > intermediate output, allowing to catch problems early and restart the test. > No implementation has yet been done. Any ideas or suggestions are welcome. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5858) When HBase gets executed against Hadoop 2.X "SLF4j:CLASSPATH contains multiple SLF4j binding" warning is displayed
[ https://issues.apache.org/jira/browse/HBASE-5858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260185#comment-13260185 ] Roman Shaposhnik commented on HBASE-5858: - Praveen, if what you're getting is an *error* you must be having a different problem. I get the very same messages but they are harmless warnings. > When HBase gets executed against Hadoop 2.X "SLF4j:CLASSPATH contains > multiple SLF4j binding" warning is displayed > -- > > Key: HBASE-5858 > URL: https://issues.apache.org/jira/browse/HBASE-5858 > Project: HBase > Issue Type: Bug > Components: scripts >Affects Versions: 0.92.0, 0.92.1 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik >Priority: Minor > Attachments: error.JPG > > > Since HBase tries to find locations of the Hadoop jars from the environment > in certain cases it ends up with extra SLF4j jars coming from Hadoop > classpath. When that happens "SLF4j:CLASSPATH contains multiple SLF4j > binding" warning gets displayed. > The good news here is that the warning is just that -- a warning. The bad > news is that it seems this issue will be tricky to fix properly. On one hand > we would like Hadoop itself to give us transitive closure of the set of jar > files required (computing that in the HBase script is a maintenance > nightmare). On the other hand that set of jar files could contain some of the > very same jars shipped with HBase. > This problem existed for quiet some time. SLF4j is just the first component > to complain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5858) When HBase gets executed against Hadoop 2.X "SLF4j:CLASSPATH contains multiple SLF4j binding" warning is displayed
Roman Shaposhnik created HBASE-5858: --- Summary: When HBase gets executed against Hadoop 2.X "SLF4j:CLASSPATH contains multiple SLF4j binding" warning is displayed Key: HBASE-5858 URL: https://issues.apache.org/jira/browse/HBASE-5858 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.92.1, 0.92.0 Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik Priority: Minor Since HBase tries to find locations of the Hadoop jars from the environment in certain cases it ends up with extra SLF4j jars coming from Hadoop classpath. When that happens "SLF4j:CLASSPATH contains multiple SLF4j binding" warning gets displayed. The good news here is that the warning is just that -- a warning. The bad news is that it seems this issue will be tricky to fix properly. On one hand we would like Hadoop itself to give us transitive closure of the set of jar files required (computing that in the HBase script is a maintenance nightmare). On the other hand that set of jar files could contain some of the very same jars shipped with HBase. This problem existed for quiet some time. SLF4j is just the first component to complain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it
[ https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113859#comment-13113859 ] Roman Shaposhnik commented on HBASE-4209: - @stack > At least log the exception. > What is the exception we are suppressing? Should we at least log it here too? Basically, the only root cause of possible exceptions here is the code in suppressHdfsShutdownHook() (which is quite fragile and overly aggressive in throwing exceptions if things don't look quite right). Now, if you look at how this code is executed in a non-standalone case -- HBase does bail on exceptions thrown from there. We can adopt the same approach in a standalone and unit testing case, but I wasn't sure it was the right thing to do. After all, NOT bailing out (and lets say simply logging these things) will be no worse than what we currently have (not calling shutdown hooks at all). On the other hand -- if we do bail out aggressively we get a better chance of catching incompatibilities between suppressHdfsShutdownHook() logic and future hadoop releases during unit tests. So perhaps -- bailing out would make the most sense of all. > The HBase hbase-daemon.sh SIGKILLs master when stopping it > -- > > Key: HBASE-4209 > URL: https://issues.apache.org/jira/browse/HBASE-4209 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: HBASE-4209.patch.txt > > > There's a bit of code in hbase-daemon.sh that makes HBase master being > SIGKILLed when stopping it rather than trying SIGTERM (like it does for other > daemons). When HBase is executed in a standalone mode (and the only daemon > you need to run is master) that causes newly created tables to go missing as > unflushed data is thrown out. If there was not a good reason to kill master > with SIGKILL perhaps we can take that special case out and rely on SIGTERM. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4427) It would help to run a standalone HBase's ZK on a different port
[
https://issues.apache.org/jira/browse/HBASE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113625#comment-13113625
]
Roman Shaposhnik commented on HBASE-4427:
-
@ Jean-Daniel Cryans
I'm sorry to drag it longer than it should be -- but you didn't answer my
question. Your concerned is answered in the comment #4 (look for
embeddedClientPort or miniZKclientPort). Of course all of this is totally
predicated on all the clients always paying attention to hbase-default.xml.
Whether they always do or not, was, in fact, my question ;-)
P.S. Sorry for not mentioning embeddedClientPort or miniZKclientPort in the
description of the proposed change, I thought it would be assumed, but I should
have been more explicit.
> It would help to run a standalone HBase's ZK on a different port
>
>
> Key: HBASE-4427
> URL: https://issues.apache.org/jira/browse/HBASE-4427
> Project: HBase
> Issue Type: Improvement
> Components: zookeeper
>Affects Versions: 0.90.4
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
>
> It would be extremely helpful to have standalone HBase default to a
> non-standard port for running its embedded ZK. This would help to run HBase
> on the same host where a legitimate fully distributed ZK server, etc.
> It seems that the following addition to hbase-default.xml would be enough to
> make it happen:
> {noformat}
> +
> +hbase.zookeeper.property.clientPort
> +4181
> +
> {noformat}
> This will take care of the master/client for HBase and can be overridden in
> hbase-site if needed.
> Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4427) It would help to run a standalone HBase's ZK on a different port
[
https://issues.apache.org/jira/browse/HBASE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113591#comment-13113591
]
Roman Shaposhnik commented on HBASE-4427:
-
@Jean-Daniel Cryans
> the idea of setting hbase.zookeeper.property.clientPort to a new default
> breaks about every setup I know of.
this is not correct. everything that reads hbase-default.xml will continue to
work as expected (hbase shell for example). I'm still to new to HBase to know
whether 3d party clients that are built on top of HBase APIs are forced to read
hbase-default.xml or not. Would appreciate if somebody let me know.
In general, however, I think the consensus here is pretty clear. Thanks for the
feedback and if somebody can answer the question above (on hbase-default.xml) I
think we can safely close this JIRA.
> It would help to run a standalone HBase's ZK on a different port
>
>
> Key: HBASE-4427
> URL: https://issues.apache.org/jira/browse/HBASE-4427
> Project: HBase
> Issue Type: Improvement
> Components: zookeeper
>Affects Versions: 0.90.4
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
>
> It would be extremely helpful to have standalone HBase default to a
> non-standard port for running its embedded ZK. This would help to run HBase
> on the same host where a legitimate fully distributed ZK server, etc.
> It seems that the following addition to hbase-default.xml would be enough to
> make it happen:
> {noformat}
> +
> +hbase.zookeeper.property.clientPort
> +4181
> +
> {noformat}
> This will take care of the master/client for HBase and can be overridden in
> hbase-site if needed.
> Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4427) It would help to run a standalone HBase's ZK on a different port
[
https://issues.apache.org/jira/browse/HBASE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113035#comment-13113035
]
Roman Shaposhnik commented on HBASE-4427:
-
@Jean-Daniel Cryans
If by local client you mean things like hbase shell -- that works quite nicely
with the proposed change.
> It would help to run a standalone HBase's ZK on a different port
>
>
> Key: HBASE-4427
> URL: https://issues.apache.org/jira/browse/HBASE-4427
> Project: HBase
> Issue Type: Improvement
> Components: zookeeper
>Affects Versions: 0.90.4
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
>
> It would be extremely helpful to have standalone HBase default to a
> non-standard port for running its embedded ZK. This would help to run HBase
> on the same host where a legitimate fully distributed ZK server, etc.
> It seems that the following addition to hbase-default.xml would be enough to
> make it happen:
> {noformat}
> +
> +hbase.zookeeper.property.clientPort
> +4181
> +
> {noformat}
> This will take care of the master/client for HBase and can be overridden in
> hbase-site if needed.
> Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4427) It would help to run a standalone HBase's ZK on a different port
[
https://issues.apache.org/jira/browse/HBASE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112983#comment-13112983
]
Roman Shaposhnik commented on HBASE-4427:
-
@Jean-Daniel Cryans
> but then if you have an external client it means you have to
> change its default config in order to talk to a cluster that has a managed
> ZK,
how "external" of a client are we talking here? running on the same host or
remote?
> Something to consider would be refusing to start HBase if while starting it's
> own ZK it gets a port binding exception.
I believe that's the current behavior, which is, in fact, part of the problem
with out-of-the-box experience on nodes where there's an already running ZK
> It would help to run a standalone HBase's ZK on a different port
>
>
> Key: HBASE-4427
> URL: https://issues.apache.org/jira/browse/HBASE-4427
> Project: HBase
> Issue Type: Improvement
> Components: zookeeper
>Affects Versions: 0.90.4
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
>
> It would be extremely helpful to have standalone HBase default to a
> non-standard port for running its embedded ZK. This would help to run HBase
> on the same host where a legitimate fully distributed ZK server, etc.
> It seems that the following addition to hbase-default.xml would be enough to
> make it happen:
> {noformat}
> +
> +hbase.zookeeper.property.clientPort
> +4181
> +
> {noformat}
> This will take care of the master/client for HBase and can be overridden in
> hbase-site if needed.
> Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4427) It would help to run a standalone HBase's ZK on a different port
[
https://issues.apache.org/jira/browse/HBASE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109854#comment-13109854
]
Roman Shaposhnik commented on HBASE-4427:
-
@stack, I was hoping other members of the community would chime in (one way or
the other ;-)) but I guess it is just you and me.
So here's my attempt at convincing you that this is the right thing to do: at
the end of the day I believe that only an explicitly managed version of ZK can
occupy a well-known client port of 2181. Every other application that uses an
embedded ZK to manage some kind of an ensemble for its own purposes has to run
its copy of ZK on a different port. Otherwise it is a bit like having an
embedded version of Jetty that would alway bind to 8080 -- not clean and source
of miscellaneous gotchas downstream when a real application server starts.
Now, perhaps, to make things extra clear we should introduce an extra property
called embeddedClientPort or miniZKclientPort and let the existing be.
If you agree with this reasoning, I can attach a patch to make it happen
according to the scenario I've outlined. If not -- that's also fine, but please
let me know either way.
> It would help to run a standalone HBase's ZK on a different port
>
>
> Key: HBASE-4427
> URL: https://issues.apache.org/jira/browse/HBASE-4427
> Project: HBase
> Issue Type: Improvement
> Components: zookeeper
>Affects Versions: 0.90.4
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
>
> It would be extremely helpful to have standalone HBase default to a
> non-standard port for running its embedded ZK. This would help to run HBase
> on the same host where a legitimate fully distributed ZK server, etc.
> It seems that the following addition to hbase-default.xml would be enough to
> make it happen:
> {noformat}
> +
> +hbase.zookeeper.property.clientPort
> +4181
> +
> {noformat}
> This will take care of the master/client for HBase and can be overridden in
> hbase-site if needed.
> Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it
[
https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109824#comment-13109824
]
Roman Shaposhnik commented on HBASE-4209:
-
I was asked yesterday how this can be tested. I'm not quite sure I can come up
with a traditional unit test for this change, but here's how to verify it
manually:
{noformat}
$ bin/hbase master start & PID=$! ; sleep 15 ; kill $PID
{noformat}
And look for LOG messages of the following nature:
{noformat}
INFO regionserver.ShutdownHook: Shutdown hook starting;
hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-18,5,main]
..
INFO regionserver.ShutdownHook: Starting fs shutdown hook thread.
INFO regionserver.ShutdownHook: Shutdown hook finished.
{noformat}
Without the patch -- the HBase master simply dies (as in -- no messages get
produced)
> The HBase hbase-daemon.sh SIGKILLs master when stopping it
> --
>
> Key: HBASE-4209
> URL: https://issues.apache.org/jira/browse/HBASE-4209
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
> Attachments: HBASE-4209.patch.txt
>
>
> There's a bit of code in hbase-daemon.sh that makes HBase master being
> SIGKILLed when stopping it rather than trying SIGTERM (like it does for other
> daemons). When HBase is executed in a standalone mode (and the only daemon
> you need to run is master) that causes newly created tables to go missing as
> unflushed data is thrown out. If there was not a good reason to kill master
> with SIGKILL perhaps we can take that special case out and rely on SIGTERM.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4427) It would help to run a standalone HBase's ZK on a different port
[
https://issues.apache.org/jira/browse/HBASE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107204#comment-13107204
]
Roman Shaposhnik commented on HBASE-4427:
-
It is mostly out-of-the-box experience kind of question and you're absolutely
right -- I can set it in my hbase-site.xml. However, I would like to understand
a little bit better why do you think it'll be that disruptive. Suppose we
change hbase-default.xml. The scenarious I see are these:
* standalone mode -- will work perfectly
* pseudo-distributed mode (hbase.cluster.distributed == true) with embedded
ZK -- will work perfectly
* pseudo-distributed mode (hbase.cluster.distributed == true) with
standalone ZK (hbase.zookeeper.quorum != nul) -- will work perfectly since, as
far as I understand, hbase.zookeeper.quorum will override
hbase.zookeeper.property.clientPort.
* fully-distributed mode with standalone ZK -- see above. Little difference
between where the processes run, they will still require the same configs as
far as hbase-site.xml is concerned.
Am I missing anything?
> It would help to run a standalone HBase's ZK on a different port
>
>
> Key: HBASE-4427
> URL: https://issues.apache.org/jira/browse/HBASE-4427
> Project: HBase
> Issue Type: Improvement
> Components: zookeeper
>Affects Versions: 0.90.4
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
>
> It would be extremely helpful to have standalone HBase default to a
> non-standard port for running its embedded ZK. This would help to run HBase
> on the same host where a legitimate fully distributed ZK server, etc.
> It seems that the following addition to hbase-default.xml would be enough to
> make it happen:
> {noformat}
> +
> +hbase.zookeeper.property.clientPort
> +4181
> +
> {noformat}
> This will take care of the master/client for HBase and can be overridden in
> hbase-site if needed.
> Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4427) It would help to run a standalone HBase's ZK on a different port
It would help to run a standalone HBase's ZK on a different port
Key: HBASE-4427
URL: https://issues.apache.org/jira/browse/HBASE-4427
Project: HBase
Issue Type: Improvement
Components: zookeeper
Affects Versions: 0.90.4
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
Priority: Minor
It would be extremely helpful to have standalone HBase default to a
non-standard port for running its embedded ZK. This would help to run HBase on
the same host where a legitimate fully distributed ZK server, etc.
It seems that the following addition to hbase-default.xml would be enough to
make it happen:
{noformat}
+
+hbase.zookeeper.property.clientPort
+4181
+
{noformat}
This will take care of the master/client for HBase and can be overridden in
hbase-site if needed.
Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it
[ https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HBASE-4209: Assignee: Roman Shaposhnik Status: Patch Available (was: Open) > The HBase hbase-daemon.sh SIGKILLs master when stopping it > -- > > Key: HBASE-4209 > URL: https://issues.apache.org/jira/browse/HBASE-4209 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: HBASE-4209.patch.txt > > > There's a bit of code in hbase-daemon.sh that makes HBase master being > SIGKILLed when stopping it rather than trying SIGTERM (like it does for other > daemons). When HBase is executed in a standalone mode (and the only daemon > you need to run is master) that causes newly created tables to go missing as > unflushed data is thrown out. If there was not a good reason to kill master > with SIGKILL perhaps we can take that special case out and rely on SIGTERM. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4243) HADOOP_HOME should be auto-detected
[
https://issues.apache.org/jira/browse/HBASE-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HBASE-4243:
Status: Patch Available (was: Reopened)
> HADOOP_HOME should be auto-detected
> ---
>
> Key: HBASE-4243
> URL: https://issues.apache.org/jira/browse/HBASE-4243
> Project: HBase
> Issue Type: Improvement
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
> Attachments: HBASE-4243-2.patch.txt, HBASE-4243.patch.txt
>
>
> Now that HBASE-3465 has been integrated, perhaps we should try to auto-detect
> the HADOOP_HOME setting if it is not given explicitly. Something along the
> lines of:
> {noformat}
> # check for hadoop in the path
> 141 HADOOP_IN_PATH=`which hadoop 2>/dev/null`
> 142 if [ -f ${HADOOP_IN_PATH} ]; then
> 143 HADOOP_DIR=`dirname "$HADOOP_IN_PATH"`/..
> 144 fi
> 145 # HADOOP_HOME env variable overrides hadoop in the path
> 146 HADOOP_HOME=${HADOOP_HOME:-$HADOOP_DIR}
> 147 if [ "$HADOOP_HOME" == "" ]; then
> 148 echo "Cannot find hadoop installation: \$HADOOP_HOME must be set or
> hadoop must be in the path";
> 149 exit 4;
> 150 fi
> {noformat}
> Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4243) HADOOP_HOME should be auto-detected
[
https://issues.apache.org/jira/browse/HBASE-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HBASE-4243:
Attachment: HBASE-4243-2.patch.txt
Hand-crafted implementation of readlink. Not pretty buts seems to work. Please
let me know if you want it to be polished some more.
> HADOOP_HOME should be auto-detected
> ---
>
> Key: HBASE-4243
> URL: https://issues.apache.org/jira/browse/HBASE-4243
> Project: HBase
> Issue Type: Improvement
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
> Attachments: HBASE-4243-2.patch.txt, HBASE-4243.patch.txt
>
>
> Now that HBASE-3465 has been integrated, perhaps we should try to auto-detect
> the HADOOP_HOME setting if it is not given explicitly. Something along the
> lines of:
> {noformat}
> # check for hadoop in the path
> 141 HADOOP_IN_PATH=`which hadoop 2>/dev/null`
> 142 if [ -f ${HADOOP_IN_PATH} ]; then
> 143 HADOOP_DIR=`dirname "$HADOOP_IN_PATH"`/..
> 144 fi
> 145 # HADOOP_HOME env variable overrides hadoop in the path
> 146 HADOOP_HOME=${HADOOP_HOME:-$HADOOP_DIR}
> 147 if [ "$HADOOP_HOME" == "" ]; then
> 148 echo "Cannot find hadoop installation: \$HADOOP_HOME must be set or
> hadoop must be in the path";
> 149 exit 4;
> 150 fi
> {noformat}
> Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it
[ https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HBASE-4209: Attachment: HBASE-4209.patch.txt I'm attaching a sample patch that seems to be working well enough. The idea behind it is to reference count the fsShutdownHooks that we have to run from shutdown sequences of multiple RegionServers and only *really* run each when the last registered regionserver shuts down. Please let me know what do you think. > The HBase hbase-daemon.sh SIGKILLs master when stopping it > -- > > Key: HBASE-4209 > URL: https://issues.apache.org/jira/browse/HBASE-4209 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Roman Shaposhnik > Attachments: HBASE-4209.patch.txt > > > There's a bit of code in hbase-daemon.sh that makes HBase master being > SIGKILLed when stopping it rather than trying SIGTERM (like it does for other > daemons). When HBase is executed in a standalone mode (and the only daemon > you need to run is master) that causes newly created tables to go missing as > unflushed data is thrown out. If there was not a good reason to kill master > with SIGKILL perhaps we can take that special case out and rely on SIGTERM. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it
[ https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101674#comment-13101674 ] Roman Shaposhnik commented on HBASE-4209: - stack, I'm sorry for putting this on a backburner, but at least I now have a better understanding of what's going on. Basically I got confused in a situation where suppressHdfsShutdownHook would be called multiple times on the same filesystem object. The first call would succeed, but all the other ones would fail. This is, obviously, just a problem with my patch, not the minihdfs cluster. I'll cook up an alternative and once I run the tests will attach an updated version. P.S. Thanks for the encouragement! > The HBase hbase-daemon.sh SIGKILLs master when stopping it > -- > > Key: HBASE-4209 > URL: https://issues.apache.org/jira/browse/HBASE-4209 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Roman Shaposhnik > > There's a bit of code in hbase-daemon.sh that makes HBase master being > SIGKILLed when stopping it rather than trying SIGTERM (like it does for other > daemons). When HBase is executed in a standalone mode (and the only daemon > you need to run is master) that causes newly created tables to go missing as > unflushed data is thrown out. If there was not a good reason to kill master > with SIGKILL perhaps we can take that special case out and rely on SIGTERM. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4243) HADOOP_HOME should be auto-detected
[
https://issues.apache.org/jira/browse/HBASE-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101540#comment-13101540
]
Roman Shaposhnik commented on HBASE-4243:
-
Sorry about that. I tend to assume Linux these days. What's the level of UNIX
API I can count on? POSIX? Or even less than that? Please let me know and I'll
update the patch.
> HADOOP_HOME should be auto-detected
> ---
>
> Key: HBASE-4243
> URL: https://issues.apache.org/jira/browse/HBASE-4243
> Project: HBase
> Issue Type: Improvement
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>Priority: Minor
> Attachments: HBASE-4243.patch.txt
>
>
> Now that HBASE-3465 has been integrated, perhaps we should try to auto-detect
> the HADOOP_HOME setting if it is not given explicitly. Something along the
> lines of:
> {noformat}
> # check for hadoop in the path
> 141 HADOOP_IN_PATH=`which hadoop 2>/dev/null`
> 142 if [ -f ${HADOOP_IN_PATH} ]; then
> 143 HADOOP_DIR=`dirname "$HADOOP_IN_PATH"`/..
> 144 fi
> 145 # HADOOP_HOME env variable overrides hadoop in the path
> 146 HADOOP_HOME=${HADOOP_HOME:-$HADOOP_DIR}
> 147 if [ "$HADOOP_HOME" == "" ]; then
> 148 echo "Cannot find hadoop installation: \$HADOOP_HOME must be set or
> hadoop must be in the path";
> 149 exit 4;
> 150 fi
> {noformat}
> Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4076) hbase should pick up HADOOP_CONF_DIR on its classpath
[
https://issues.apache.org/jira/browse/HBASE-4076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094207#comment-13094207
]
Roman Shaposhnik commented on HBASE-4076:
-
Isn't it a dup of HBASE-3465. At least from the implementation point of view:
{noformat}
#If avail, add Hadoop to the CLASSPATH and to the JAVA_LIBRARY_PATH
if [ ! -z $HADOOP_HOME ]; then
HADOOPCPPATH=""
if [ -z $HADOOP_CONF_DIR ]; then
HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" "${HADOOP_HOME}/conf")
else
HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" "${HADOOP_CONF_DIR}")
fi
...
{noformat}
> hbase should pick up HADOOP_CONF_DIR on its classpath
> -
>
> Key: HBASE-4076
> URL: https://issues.apache.org/jira/browse/HBASE-4076
> Project: HBase
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
> Fix For: 0.94.0
>
>
> Currently, hbase doesn't automatically include the hadoop config on its
> classpath. It usually works out OK since we specify hbase.rootdir with a
> fullly qualified hdfs://nnhost/path URL. But, in secure environments for
> example, there are some other Hadoop configs that need to be available to
> HBase (eg NN krb5 principal info).
> We should change the HBase scripts to automatically pick up HADOOP_CONF_DIR
> or HADOOP_HOME/conf on the classpath when available.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it
[
https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091246#comment-13091246
]
Roman Shaposhnik commented on HBASE-4209:
-
stack, I have tried it with a standalone HBase writing to a local filesystem (I
think there's no way making a standalone instance of HBase use HDFS). And that
works quite nicely. Tables get flushed when I SIGTERM the process, etc. Once it
came to testing I had to run it with miniDFS and that's where things got
interesting. suppressHdfsShutdownHook can't disable the HDFS shutdown hook of
the miniDFS, simply because there's nothing to disable.
So here's the question, should I work around this one special case by simply
testing for the
{noformat}
LocalHBaseCluster.isLocal(hrs.getConfiguration())
{noformat}
or would you propose something else?
I'd appreciate your advice, since I'm just learning my way around HBase.
> The HBase hbase-daemon.sh SIGKILLs master when stopping it
> --
>
> Key: HBASE-4209
> URL: https://issues.apache.org/jira/browse/HBASE-4209
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Roman Shaposhnik
>
> There's a bit of code in hbase-daemon.sh that makes HBase master being
> SIGKILLed when stopping it rather than trying SIGTERM (like it does for other
> daemons). When HBase is executed in a standalone mode (and the only daemon
> you need to run is master) that causes newly created tables to go missing as
> unflushed data is thrown out. If there was not a good reason to kill master
> with SIGKILL perhaps we can take that special case out and rely on SIGTERM.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it
[
https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090423#comment-13090423
]
Roman Shaposhnik commented on HBASE-4209:
-
Perhaps this will be a naive suggestion, but wouldn't it make sense to install
Shutdown hooks regardless of whether the Regioserver was started in a
standalone mode. Will a trivial change like this one make sense:
{noformat}
--- src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java
+++ src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java
@@ -179,7 +179,11 @@ public class JVMClusterUtil {
}
if (regionservers != null) {
for (JVMClusterUtil.RegionServerThread t: regionservers) {
-t.start();
+ try {
+ HRegionServer.startRegionServer(t.getRegionServer());
+ } catch (IOException e) {
+ // Nothing to do, really
+ }
}
}
if (masters == null || masters.isEmpty()) {
{noformat}
> The HBase hbase-daemon.sh SIGKILLs master when stopping it
> --
>
> Key: HBASE-4209
> URL: https://issues.apache.org/jira/browse/HBASE-4209
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Roman Shaposhnik
>
> There's a bit of code in hbase-daemon.sh that makes HBase master being
> SIGKILLed when stopping it rather than trying SIGTERM (like it does for other
> daemons). When HBase is executed in a standalone mode (and the only daemon
> you need to run is master) that causes newly created tables to go missing as
> unflushed data is thrown out. If there was not a good reason to kill master
> with SIGKILL perhaps we can take that special case out and rely on SIGTERM.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4243) HADOOP_HOME should be auto-detected
[
https://issues.apache.org/jira/browse/HBASE-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HBASE-4243:
Status: Patch Available (was: Open)
Here's the basic code. I can always add echoing the value that we
auto-discover, but wouldn't it make hbase CLI script too verbose? Please let me
know.
> HADOOP_HOME should be auto-detected
> ---
>
> Key: HBASE-4243
> URL: https://issues.apache.org/jira/browse/HBASE-4243
> Project: HBase
> Issue Type: Improvement
>Reporter: Roman Shaposhnik
>Priority: Minor
> Attachments: HBASE-4243.patch.txt
>
>
> Now that HBASE-3465 has been integrated, perhaps we should try to auto-detect
> the HADOOP_HOME setting if it is not given explicitly. Something along the
> lines of:
> {noformat}
> # check for hadoop in the path
> 141 HADOOP_IN_PATH=`which hadoop 2>/dev/null`
> 142 if [ -f ${HADOOP_IN_PATH} ]; then
> 143 HADOOP_DIR=`dirname "$HADOOP_IN_PATH"`/..
> 144 fi
> 145 # HADOOP_HOME env variable overrides hadoop in the path
> 146 HADOOP_HOME=${HADOOP_HOME:-$HADOOP_DIR}
> 147 if [ "$HADOOP_HOME" == "" ]; then
> 148 echo "Cannot find hadoop installation: \$HADOOP_HOME must be set or
> hadoop must be in the path";
> 149 exit 4;
> 150 fi
> {noformat}
> Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4243) HADOOP_HOME should be auto-detected
[
https://issues.apache.org/jira/browse/HBASE-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Shaposhnik updated HBASE-4243:
Attachment: HBASE-4243.patch.txt
> HADOOP_HOME should be auto-detected
> ---
>
> Key: HBASE-4243
> URL: https://issues.apache.org/jira/browse/HBASE-4243
> Project: HBase
> Issue Type: Improvement
>Reporter: Roman Shaposhnik
>Priority: Minor
> Attachments: HBASE-4243.patch.txt
>
>
> Now that HBASE-3465 has been integrated, perhaps we should try to auto-detect
> the HADOOP_HOME setting if it is not given explicitly. Something along the
> lines of:
> {noformat}
> # check for hadoop in the path
> 141 HADOOP_IN_PATH=`which hadoop 2>/dev/null`
> 142 if [ -f ${HADOOP_IN_PATH} ]; then
> 143 HADOOP_DIR=`dirname "$HADOOP_IN_PATH"`/..
> 144 fi
> 145 # HADOOP_HOME env variable overrides hadoop in the path
> 146 HADOOP_HOME=${HADOOP_HOME:-$HADOOP_DIR}
> 147 if [ "$HADOOP_HOME" == "" ]; then
> 148 echo "Cannot find hadoop installation: \$HADOOP_HOME must be set or
> hadoop must be in the path";
> 149 exit 4;
> 150 fi
> {noformat}
> Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4243) HADOOP_HOME should be auto-detected
HADOOP_HOME should be auto-detected
---
Key: HBASE-4243
URL: https://issues.apache.org/jira/browse/HBASE-4243
Project: HBase
Issue Type: Improvement
Reporter: Roman Shaposhnik
Priority: Minor
Now that HBASE-3465 has been integrated, perhaps we should try to auto-detect
the HADOOP_HOME setting if it is not given explicitly. Something along the
lines of:
{noformat}
# check for hadoop in the path
141 HADOOP_IN_PATH=`which hadoop 2>/dev/null`
142 if [ -f ${HADOOP_IN_PATH} ]; then
143 HADOOP_DIR=`dirname "$HADOOP_IN_PATH"`/..
144 fi
145 # HADOOP_HOME env variable overrides hadoop in the path
146 HADOOP_HOME=${HADOOP_HOME:-$HADOOP_DIR}
147 if [ "$HADOOP_HOME" == "" ]; then
148 echo "Cannot find hadoop installation: \$HADOOP_HOME must be set or
hadoop must be in the path";
149 exit 4;
150 fi
{noformat}
Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it
[
https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087966#comment-13087966
]
Roman Shaposhnik commented on HBASE-4209:
-
stack, thanks for the clarifications -- much appreciated! Now, as I said, I
only see addShutdownHook call in regionserver code:
{noformat}
$ git grep addShutdownHook
src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java:
Runtime.getRuntime().addShutdownHook(t);
{noformat}
and then
{noformat}
/**
* @param hrs
* @param name
* @return Thread the RegionServer is running in correctly named.
* @throws IOException
*/
public static Thread startRegionServer(final HRegionServer hrs,
final String name) throws IOException {
Thread t = new Thread(hrs);
t.setName(name);
t.start();
// Install shutdown hook that will catch signals and run an orderly shutdown
// of the hrs.
ShutdownHook.install(hrs.getConfiguration(), FileSystem.get(hrs
.getConfiguration()), hrs, t);
return t;
}
{noformat}
Now, the behavior that I observe is that when HBase is running in a distributed
fashion, flushing of the newly created tables happen when regionserve receives
a SIGTERM. When it is running in a standalone configuration tables don't get
flushed UNLESS I explicitly call stop-hbase.sh before sending a SIGTERM.
Once would assume that the same code would be called in both cases since it is
registering that hook unconditionally. What am I missing here?
> The HBase hbase-daemon.sh SIGKILLs master when stopping it
> --
>
> Key: HBASE-4209
> URL: https://issues.apache.org/jira/browse/HBASE-4209
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Roman Shaposhnik
>
> There's a bit of code in hbase-daemon.sh that makes HBase master being
> SIGKILLed when stopping it rather than trying SIGTERM (like it does for other
> daemons). When HBase is executed in a standalone mode (and the only daemon
> you need to run is master) that causes newly created tables to go missing as
> unflushed data is thrown out. If there was not a good reason to kill master
> with SIGKILL perhaps we can take that special case out and rely on SIGTERM.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it
[ https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086513#comment-13086513 ] Roman Shaposhnik commented on HBASE-4209: - stack, before I submit the patch, I would really appreciate if you could let me know the relationship between bin/stop-hbase.sh and bin/hbase-daemon.sh. I was under the impression that whatever stop-hbase.sh triggers in the master code would also be triggered by JVM shutdown hook upon receiving SIGTERM, but it doesn't seem to be that way. Do we have to call bin/stop-hbase.sh manually from within the hbase-daemon.sh before stopping daemons? > The HBase hbase-daemon.sh SIGKILLs master when stopping it > -- > > Key: HBASE-4209 > URL: https://issues.apache.org/jira/browse/HBASE-4209 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Roman Shaposhnik > > There's a bit of code in hbase-daemon.sh that makes HBase master being > SIGKILLed when stopping it rather than trying SIGTERM (like it does for other > daemons). When HBase is executed in a standalone mode (and the only daemon > you need to run is master) that causes newly created tables to go missing as > unflushed data is thrown out. If there was not a good reason to kill master > with SIGKILL perhaps we can take that special case out and rely on SIGTERM. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it
The HBase hbase-daemon.sh SIGKILLs master when stopping it -- Key: HBASE-4209 URL: https://issues.apache.org/jira/browse/HBASE-4209 Project: HBase Issue Type: Bug Components: master Reporter: Roman Shaposhnik There's a bit of code in hbase-daemon.sh that makes HBase master being SIGKILLed when stopping it rather than trying SIGTERM (like it does for other daemons). When HBase is executed in a standalone mode (and the only daemon you need to run is master) that causes newly created tables to go missing as unflushed data is thrown out. If there was not a good reason to kill master with SIGKILL perhaps we can take that special case out and rely on SIGTERM. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3882) hbase-config.sh needs to be updated so it can auto-detects the sun jre provided by RHEL6
[ https://issues.apache.org/jira/browse/HBASE-3882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HBASE-3882: Attachment: HBASE-3882.txt > hbase-config.sh needs to be updated so it can auto-detects the sun jre > provided by RHEL6 > > > Key: HBASE-3882 > URL: https://issues.apache.org/jira/browse/HBASE-3882 > Project: HBase > Issue Type: Bug >Affects Versions: 0.90.1 > Environment: RHEL6 >Reporter: Roman Shaposhnik >Assignee: Bruno Mahé > Fix For: 0.90.2 > > Attachments: HBASE-3882.txt > > > RHEL6 will install its sun jdk in > /usr/lib/jvm/java-1.6.0-sun-1.6.0... > So this ticket is about adding this path to the jdk autodetection mechanism > used in hbase-config.sh -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3882) hbase-config.sh needs to be updated so it can auto-detects the sun jre provided by RHEL6
[ https://issues.apache.org/jira/browse/HBASE-3882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Shaposhnik updated HBASE-3882: Status: Patch Available (was: Open) Trivial change to also account for JRE > hbase-config.sh needs to be updated so it can auto-detects the sun jre > provided by RHEL6 > > > Key: HBASE-3882 > URL: https://issues.apache.org/jira/browse/HBASE-3882 > Project: HBase > Issue Type: Bug >Affects Versions: 0.90.1 > Environment: RHEL6 >Reporter: Roman Shaposhnik >Assignee: Bruno Mahé > Fix For: 0.90.2 > > Attachments: HBASE-3882.txt > > > RHEL6 will install its sun jdk in > /usr/lib/jvm/java-1.6.0-sun-1.6.0... > So this ticket is about adding this path to the jdk autodetection mechanism > used in hbase-config.sh -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-3882) hbase-config.sh needs to be updated so it can auto-detects the sun jre provided by RHEL6
hbase-config.sh needs to be updated so it can auto-detects the sun jre provided by RHEL6 Key: HBASE-3882 URL: https://issues.apache.org/jira/browse/HBASE-3882 Project: HBase Issue Type: Bug Affects Versions: 0.90.1 Environment: RHEL6 Reporter: Roman Shaposhnik Assignee: Bruno Mahé Fix For: 0.90.2 RHEL6 will install its sun jdk in /usr/lib/jvm/java-1.6.0-sun-1.6.0... So this ticket is about adding this path to the jdk autodetection mechanism used in hbase-config.sh -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
