[jira] [Commented] (HBASE-6585) Audit log messages should contain info about the higher level operation being executed
[ https://issues.apache.org/jira/browse/HBASE-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535760#comment-13535760 ] stack commented on HBASE-6585: -- I applied to trunk. I tried backporting but a bunch of hunks failed after recalibrating file location. Let me leave this issue open to see if Matteo is up for a 0.94 patch > Audit log messages should contain info about the higher level operation being > executed > -- > > Key: HBASE-6585 > URL: https://issues.apache.org/jira/browse/HBASE-6585 > Project: HBase > Issue Type: Improvement > Components: security >Affects Versions: 0.96.0 >Reporter: Marcelo Vanzin >Assignee: Matteo Bertozzi >Priority: Minor > Labels: acl > Fix For: 0.96.0, 0.94.4 > > Attachments: HBASE-6585-v0.patch, HBASE-6585-v1.patch, > HBASE-6585-v2.patch > > > Currently, audit log messages contains the "action" for which access was > checked; this is one of READ, WRITE, CREATE or ADMIN. > These give very little information to the person digging into the logs about > what was done, though. You can't ask "who deleted rows from table x?", > because "delete" is translated to a "WRITE" action. > It would be nice if the audit logs contained the higher-level operation, > either replacing or in addition to the RWCA information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7372) Check in the generated website so can point apache infrastructure at what to publish as our hbase.apache.org
[
https://issues.apache.org/jira/browse/HBASE-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535755#comment-13535755
]
stack commented on HBASE-7372:
--
I added the checked in website BESIDE trunk and branches. See here:
http://svn.apache.org/viewvc/hbase/hbase.apache.org/trunk/ I updated the INFRA
ticket pointing at this new location. Leaving issue open till we close INFRA
ticket.
> Check in the generated website so can point apache infrastructure at what to
> publish as our hbase.apache.org
>
>
> Key: HBASE-7372
> URL: https://issues.apache.org/jira/browse/HBASE-7372
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Attachments: 7372v2.txt
>
>
> January 1st is deadline for changing how we publish our website. We may no
> longer rsync out to people.apache.org. Apache infrastructure supplies two
> options here: http://www.apache.org/dev/project-site.html We could redo our
> site in apache cms format. Or we could just use svnpubsub and keep on w/ how
> the site is currently generated and on checkin, have it autopublished. I'll
> go the latter route unless I hear otherwise.
> For svnpubsub, we need to point apache infrastructure at a directory that has
> our checkedin site in it. I was thinking ${hbasedir}/hbase.apache.org
> Let me raise this on the dev list too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7372) Check in the generated website so can point apache infrastructure at what to publish as our hbase.apache.org
[
https://issues.apache.org/jira/browse/HBASE-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535754#comment-13535754
]
Hudson commented on HBASE-7372:
---
Integrated in HBase-TRUNK #3637 (See
[https://builds.apache.org/job/HBase-TRUNK/3637/])
HBASE-7372 Check in the generated website so can point apache
infrastructure at what to publish as our hbase.apache.org (Revision 1423766)
Result = FAILURE
stack :
Files :
* /hbase/trunk/hbase.apache.org
* /hbase/trunk/pom.xml
* /hbase/trunk/src/site/site.xml
> Check in the generated website so can point apache infrastructure at what to
> publish as our hbase.apache.org
>
>
> Key: HBASE-7372
> URL: https://issues.apache.org/jira/browse/HBASE-7372
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Attachments: 7372v2.txt
>
>
> January 1st is deadline for changing how we publish our website. We may no
> longer rsync out to people.apache.org. Apache infrastructure supplies two
> options here: http://www.apache.org/dev/project-site.html We could redo our
> site in apache cms format. Or we could just use svnpubsub and keep on w/ how
> the site is currently generated and on checkin, have it autopublished. I'll
> go the latter route unless I hear otherwise.
> For svnpubsub, we need to point apache infrastructure at a directory that has
> our checkedin site in it. I was thinking ${hbasedir}/hbase.apache.org
> Let me raise this on the dev list too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7388) Snapshot branch 12/18 rebase broke TestSnapshotFromMaster#testSnapshotHFileArchiving
[
https://issues.apache.org/jira/browse/HBASE-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535753#comment-13535753
]
Hadoop QA commented on HBASE-7388:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561657/hbase-7388.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3606//console
This message is automatically generated.
> Snapshot branch 12/18 rebase broke
> TestSnapshotFromMaster#testSnapshotHFileArchiving
>
>
> Key: HBASE-7388
> URL: https://issues.apache.org/jira/browse/HBASE-7388
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots, test
>Affects Versions: hbase-6055
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: hbase-7388.patch
>
>
> {code}
> Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 30.232 sec
> <<< FAILURE!
> Results :
> Failed tests:
> testSnapshotHFileArchiving(org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster):
> Archived hfiles [] is missing snapshot
> file:hdfs://localhost:36177/user/jon/hbase/.snapshot/snapshot/8b7235fe93a521daee2a446915d87bfe/fam/01a951c716a445dea55698bdd79c294c
> {code}
> Something in trunk changed the number of hfiles generated such that only four
> are generated. The previous default was a min of 5 hfiles before compaction
> actually started so files were never compacted, and thus never archived.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7388) Snapshot branch 12/18 rebase broke TestSnapshotFromMaster#testSnapshotHFileArchiving
[
https://issues.apache.org/jira/browse/HBASE-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7388:
--
Component/s: (was: Zookeeper)
(was: regionserver)
(was: master)
(was: Client)
test
> Snapshot branch 12/18 rebase broke
> TestSnapshotFromMaster#testSnapshotHFileArchiving
>
>
> Key: HBASE-7388
> URL: https://issues.apache.org/jira/browse/HBASE-7388
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots, test
>Affects Versions: hbase-6055
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: hbase-7388.patch
>
>
> {code}
> Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 30.232 sec
> <<< FAILURE!
> Results :
> Failed tests:
> testSnapshotHFileArchiving(org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster):
> Archived hfiles [] is missing snapshot
> file:hdfs://localhost:36177/user/jon/hbase/.snapshot/snapshot/8b7235fe93a521daee2a446915d87bfe/fam/01a951c716a445dea55698bdd79c294c
> {code}
> Something in trunk changed the number of hfiles generated such that only four
> are generated. The previous default was a min of 5 hfiles before compaction
> actually started so files were never compacted, and thus never archived.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7283) Backport HBASE-6564 + HBASE-7202 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7283: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to 0.94. Thanks for the patch Matteo. > Backport HBASE-6564 + HBASE-7202 to 0.94 > > > Key: HBASE-7283 > URL: https://issues.apache.org/jira/browse/HBASE-7283 > Project: HBase > Issue Type: Task >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7283.094.txt, HBASE-6564-0.94.patch, > HBASE-7202-0.94.patch > > > * HBASE-6564: HDFS space is not reclaimed when a column family is deleted > * HBASE-7202: Family Store Files are not archived on admin.deleteColumn() > (the second one is a fix for the first, to use the archiver) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7388) Snapshot branch 12/18 rebase broke TestSnapshotFromMaster#testSnapshotHFileArchiving
[
https://issues.apache.org/jira/browse/HBASE-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535748#comment-13535748
]
Jonathan Hsieh commented on HBASE-7388:
---
With patch applied:
{code}
mvn clean test -PlocalTests -Dtest=TestSnapshotFromMaster
...
Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.196 sec
Results :
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
{code}
> Snapshot branch 12/18 rebase broke
> TestSnapshotFromMaster#testSnapshotHFileArchiving
>
>
> Key: HBASE-7388
> URL: https://issues.apache.org/jira/browse/HBASE-7388
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0
>
> Attachments: hbase-7388.patch
>
>
> {code}
> Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 30.232 sec
> <<< FAILURE!
> Results :
> Failed tests:
> testSnapshotHFileArchiving(org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster):
> Archived hfiles [] is missing snapshot
> file:hdfs://localhost:36177/user/jon/hbase/.snapshot/snapshot/8b7235fe93a521daee2a446915d87bfe/fam/01a951c716a445dea55698bdd79c294c
> {code}
> Something in trunk changed the number of hfiles generated such that only four
> are generated. The previous default was a min of 5 hfiles before compaction
> actually started so files were never compacted, and thus never archived.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7388) Snapshot branch 12/18 rebase broke TestSnapshotFromMaster#testSnapshotHFileArchiving
[
https://issues.apache.org/jira/browse/HBASE-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7388:
--
Attachment: hbase-7388.patch
> Snapshot branch 12/18 rebase broke
> TestSnapshotFromMaster#testSnapshotHFileArchiving
>
>
> Key: HBASE-7388
> URL: https://issues.apache.org/jira/browse/HBASE-7388
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0
>
> Attachments: hbase-7388.patch
>
>
> {code}
> Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 30.232 sec
> <<< FAILURE!
> Results :
> Failed tests:
> testSnapshotHFileArchiving(org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster):
> Archived hfiles [] is missing snapshot
> file:hdfs://localhost:36177/user/jon/hbase/.snapshot/snapshot/8b7235fe93a521daee2a446915d87bfe/fam/01a951c716a445dea55698bdd79c294c
> {code}
> Something in trunk changed the number of hfiles generated such that only four
> are generated. The previous default was a min of 5 hfiles before compaction
> actually started so files were never compacted, and thus never archived.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7388) Snapshot branch 12/18 rebase broke TestSnapshotFromMaster#testSnapshotHFileArchiving
[
https://issues.apache.org/jira/browse/HBASE-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7388:
--
Fix Version/s: (was: 0.96.0)
Affects Version/s: hbase-6055
Status: Patch Available (was: Open)
> Snapshot branch 12/18 rebase broke
> TestSnapshotFromMaster#testSnapshotHFileArchiving
>
>
> Key: HBASE-7388
> URL: https://issues.apache.org/jira/browse/HBASE-7388
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Affects Versions: hbase-6055
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: hbase-7388.patch
>
>
> {code}
> Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 30.232 sec
> <<< FAILURE!
> Results :
> Failed tests:
> testSnapshotHFileArchiving(org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster):
> Archived hfiles [] is missing snapshot
> file:hdfs://localhost:36177/user/jon/hbase/.snapshot/snapshot/8b7235fe93a521daee2a446915d87bfe/fam/01a951c716a445dea55698bdd79c294c
> {code}
> Something in trunk changed the number of hfiles generated such that only four
> are generated. The previous default was a min of 5 hfiles before compaction
> actually started so files were never compacted, and thus never archived.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6642) enable_all,disable_all,drop_all can call "list" command with regex directly.
[
https://issues.apache.org/jira/browse/HBASE-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-6642:
-
Fix Version/s: (was: 0.94.4)
0.94.5
Looking at this again. Are we changing the meaning?
^{regex}$ requires the table name needs to match in its entirety.
Just {regex} (depending on how it is called) could match when only a subset of
the table name matches.
> enable_all,disable_all,drop_all can call "list" command with regex directly.
>
>
> Key: HBASE-6642
> URL: https://issues.apache.org/jira/browse/HBASE-6642
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 0.92.2, 0.94.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: rajeshbabu
> Fix For: 0.92.3, 0.96.0, 0.94.5
>
> Attachments: HBASE-6642_trunk.patch
>
>
> created few tables. then performing disable_all operation in shell prompt.
> but it is not performing operation successfully.
> {noformat}
> hbase(main):043:0> disable_all '*'
> table12
> zk0113
> zk0114
> Disable the above 3 tables (y/n)?
> y/
> 3 tables successfully disabled
> just it is showing the message but operation is not success.
> but the following way only performing successfully
> hbase(main):043:0> disable_all '*.*'
> table12
> zk0113
> zk0114
> Disable the above 3 tables (y/n)?
> y
> 3 tables successfully disabled
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7388) Snapshot branch 12/18 rebase broke TestSnapshotFromMaster#testSnapshotHFileArchiving
[
https://issues.apache.org/jira/browse/HBASE-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh reassigned HBASE-7388:
-
Assignee: Jonathan Hsieh
> Snapshot branch 12/18 rebase broke
> TestSnapshotFromMaster#testSnapshotHFileArchiving
>
>
> Key: HBASE-7388
> URL: https://issues.apache.org/jira/browse/HBASE-7388
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0
>
>
> {code}
> Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 30.232 sec
> <<< FAILURE!
> Results :
> Failed tests:
> testSnapshotHFileArchiving(org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster):
> Archived hfiles [] is missing snapshot
> file:hdfs://localhost:36177/user/jon/hbase/.snapshot/snapshot/8b7235fe93a521daee2a446915d87bfe/fam/01a951c716a445dea55698bdd79c294c
> {code}
> Something in trunk changed the number of hfiles generated such that only four
> are generated. The previous default was a min of 5 hfiles before compaction
> actually started so files were never compacted, and thus never archived.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7388) Snapshot branch 12/18 rebase broke TestSnapshotFromMaster#testSnapshotHFileArchiving
Jonathan Hsieh created HBASE-7388:
-
Summary: Snapshot branch 12/18 rebase broke
TestSnapshotFromMaster#testSnapshotHFileArchiving
Key: HBASE-7388
URL: https://issues.apache.org/jira/browse/HBASE-7388
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Hsieh
{code}
Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 30.232 sec <<<
FAILURE!
Results :
Failed tests:
testSnapshotHFileArchiving(org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster):
Archived hfiles [] is missing snapshot
file:hdfs://localhost:36177/user/jon/hbase/.snapshot/snapshot/8b7235fe93a521daee2a446915d87bfe/fam/01a951c716a445dea55698bdd79c294c
{code}
Something in trunk changed the number of hfiles generated such that only four
are generated. The previous default was a min of 5 hfiles before compaction
actually started so files were never compacted, and thus never archived.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7388) Snapshot branch 12/18 rebase broke TestSnapshotFromMaster#testSnapshotHFileArchiving
[
https://issues.apache.org/jira/browse/HBASE-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535746#comment-13535746
]
Jonathan Hsieh commented on HBASE-7388:
---
>From this commit:
{code}
commit c24a19f7f40964d978747648ca93ab38010d0736
Author: Jonathan M Hsieh
Date: Tue Dec 18 14:51:57 2012 -0800
HBASE-7174 [snapshots] Refactor snapshot file cleaner cache to use Snapshot
FileVisitor (Matteo Bertozzi)
{code}
> Snapshot branch 12/18 rebase broke
> TestSnapshotFromMaster#testSnapshotHFileArchiving
>
>
> Key: HBASE-7388
> URL: https://issues.apache.org/jira/browse/HBASE-7388
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0
>
>
> {code}
> Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 30.232 sec
> <<< FAILURE!
> Results :
> Failed tests:
> testSnapshotHFileArchiving(org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster):
> Archived hfiles [] is missing snapshot
> file:hdfs://localhost:36177/user/jon/hbase/.snapshot/snapshot/8b7235fe93a521daee2a446915d87bfe/fam/01a951c716a445dea55698bdd79c294c
> {code}
> Something in trunk changed the number of hfiles generated such that only four
> are generated. The previous default was a min of 5 hfiles before compaction
> actually started so files were never compacted, and thus never archived.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535744#comment-13535744 ] stack commented on HBASE-7386: -- I'd say just repurpose 'autorestart', especially if now broke. What was there previous was mickey mouse. This is real deal. bq. ... and does not respect any other environment settings (e.g. HBASE_CONF_DIR). Would this be fixed if we "...through all the config files in hbase-daemon and do something appropriate."? On questions: 1. Yes this is valid direction. Perhaps we could extract the stuff you hacked out into a 'wrapper' script, a poor-mans' supervise such that it was there as an option... you could run it if you wanted poor-mans' supervise but otherwise, scripts ran as they used to. But this would likely be wasted effort... effort better spent getting it so optionally, if supervisord was installed, you could just run with it. 2. I agree with nkeywal that templates/samples inevitably rot. Unused software also rots so providing supervisord scripts unless they are used, they will go bad. How much work involved making it so could do ./bin/start-supervisord-hbase.sh? Would be coolio if you could do ./bin/start-hbase.sh and ./bin/start-supervisord-hbase.sh if supervisor available (likely on most systems I'd say) and then in doc. we encourage folks to do the latter. What to do for the case where a shop has chosen other than supervisord to monitor their processes? I suppose we could let them do the convertion from 'supervise' to 'god', etc.? This is great stuff G. > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.96.0 > > Attachments: HBASE-7386-v0.patch, supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6480) If callQueueSize exceed maxQueueSize, all call will be rejected, do not reject priorityCall
[
https://issues.apache.org/jira/browse/HBASE-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-6480:
-
Fix Version/s: (was: 0.94.5)
Actually, let's keep this to 0.96 only.
> If callQueueSize exceed maxQueueSize, all call will be rejected, do not
> reject priorityCall
>
>
> Key: HBASE-6480
> URL: https://issues.apache.org/jira/browse/HBASE-6480
> Project: HBase
> Issue Type: Bug
>Reporter: binlijin
> Fix For: 0.96.0
>
> Attachments: HBASE-6480-94.patch, HBASE-6480-trunk.patch
>
>
> Current if the callQueueSize exceed maxQueueSize, all call will be rejected,
> Should we let the priority Call pass through?
> Current:
> {code}
> if ((callSize + callQueueSize.get()) > maxQueueSize) {
> Call callTooBig = xxx
> return ;
> }
> if (priorityCallQueue != null && getQosLevel(param) > highPriorityLevel) {
> priorityCallQueue.put(call);
> updateCallQueueLenMetrics(priorityCallQueue);
> } else {
> callQueue.put(call); // queue the call; maybe blocked here
> updateCallQueueLenMetrics(callQueue);
> }
> {code}
> Should we change it to :
> {code}
> if (priorityCallQueue != null && getQosLevel(param) > highPriorityLevel) {
> priorityCallQueue.put(call);
> updateCallQueueLenMetrics(priorityCallQueue);
> } else {
> if ((callSize + callQueueSize.get()) > maxQueueSize) {
>Call callTooBig = xxx
>return ;
> }
> callQueue.put(call); // queue the call; maybe blocked here
> updateCallQueueLenMetrics(callQueue);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6416) hbck dies on NPE when a region folder exists but the table does not
[
https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-6416:
-
Fix Version/s: (was: 0.94.5)
> hbck dies on NPE when a region folder exists but the table does not
> ---
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0
>
> Attachments: hbase-6416.patch, hbase-6416-v1.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for
> region null
> java.io.FileNotFoundException: File does not exist:
> /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at
> org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at
> org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir:
> hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!",
> tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any
> case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6417) hbck merges .META. regions if there's an old leftover
[ https://issues.apache.org/jira/browse/HBASE-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6417: - Fix Version/s: (was: 0.94.5) Removing from 0.94 > hbck merges .META. regions if there's an old leftover > - > > Key: HBASE-6417 > URL: https://issues.apache.org/jira/browse/HBASE-6417 > Project: HBase > Issue Type: Bug >Reporter: Jean-Daniel Cryans > Fix For: 0.96.0 > > Attachments: hbck.log > > > Trying to see what caused HBASE-6310, one of the things I figured is that the > bad .META. row is actually one from the time that we were permitting meta > splitting and that folder had just been staying there for a while. > So I tried to recreate the issue with -repair and it merged my good .META. > region with the one that's 3 years old that also has the same start key. I > ended up with a brand new .META. region! > I'll be attaching the full log in a separate file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6155) [copytable] Unexpected behavior if --starttime is not specifed but --endtime is.
[ https://issues.apache.org/jira/browse/HBASE-6155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535740#comment-13535740 ] Prasanna commented on HBASE-6155: - Can someone help me find the find bugs warnings related to my patch. Also help me to find what portion of my patch caused the Release Audit problems. Thanks Prasanna > [copytable] Unexpected behavior if --starttime is not specifed but --endtime > is. > > > Key: HBASE-6155 > URL: https://issues.apache.org/jira/browse/HBASE-6155 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.90.6, 0.92.1, 0.94.0, 0.96.0 >Reporter: Jonathan Hsieh >Assignee: Prasanna > Labels: noob > Fix For: 0.96.0 > > Attachments: 6155.diff, 6155-patch-1.diff > > > If one uses copytable and specifies only an endtime, I'd expect to include > all rows from unix epoch time upto the specified endtime. Instead, it copies > all the rows. > The workaround for copies with this kind of range is to specify --startime=1 > (Note not --starttime=0), which is also unintuitive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7101) HBase stuck in Region SPLIT
[ https://issues.apache.org/jira/browse/HBASE-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7101: - Fix Version/s: (was: 0.94.4) 0.94.5 Moving to 0.94.5, since we do not have a patch. > HBase stuck in Region SPLIT > > > Key: HBASE-7101 > URL: https://issues.apache.org/jira/browse/HBASE-7101 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.1 >Reporter: Bing Jiang > Fix For: 0.96.0, 0.94.5 > > > I found this issue from a zknode which has existed for a long time in the > unassigned parent.And HMaster report warnning log increasingly.The loop log > is at below. > WARN org.apache.hadoop.hbase.master.AssignmentManager: Region > 1a1c950ad45812d7b4b9b90ebf268468 not found on server > sev0040,60020,1350378314041; failed processing > WARN org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for > region 1a1c950ad45812d7b4b9b90ebf268468 from server > sev0040,60020,1350378314041 but it doesn't exist anymore, probably already > processed its split > WARN org.apache.hadoop.hbase.master.AssignmentManager: Region > 1a1c950ad45812d7b4b9b90ebf268468 not found on server > gs-dpo-sev0040,60020,1350378314041; failed processing > WARN org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for > region 1a1c950ad45812d7b4b9b90ebf268468 from server > sev0040,60020,1350378314041 but it doesn't exist anymore, probably already > processed its split > we use Hbase-0.92.1, and I trace back to the source code. HMaster > AssignmentManager have already deleted the SPLIT_Region in its memory > structure,but HRegionServer SplitTransaction has found the > unassigned/parent-node existed in a transient state, precisely > SplitTransaction executes tickleNodeSplit to update a new version a little > later than AssignmentManager deleting unassigned/parent-znode. After > updating a version of the znode, it will intrigue the handleRegion operation > again, however, AssignmentManager assert that the RegionState in Memory has > been deleted, and transaction goes into a retry loop. > In the SplitTransaction, transitionZKNode will retry tickleNodeSplit after > sleeping 100ms. In my opinion, if the time is much longger than 100ms, all > the operation from AssignmentManagement will finish off completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6841) Meta prefetching is slower than doing multiple meta lookups
[ https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6841: - Fix Version/s: (was: 0.94.4) 0.94.5 > Meta prefetching is slower than doing multiple meta lookups > --- > > Key: HBASE-6841 > URL: https://issues.apache.org/jira/browse/HBASE-6841 > Project: HBase > Issue Type: Improvement >Reporter: Jean-Daniel Cryans >Assignee: Lars Hofhansl > Fix For: 0.94.5 > > Attachments: 6841-0.94.txt, 6841-0.96.txt > > > I got myself into a situation where I needed to truncate a massive table > while it was getting hits and surprisingly the clients were not recovering. > What I see in the logs is that every time we prefetch .META. we setup a new > HConnection because we close it on the way out. It's awfully slow. > We should just turn it off or make it useful. jstacks coming up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6417) hbck merges .META. regions if there's an old leftover
[ https://issues.apache.org/jira/browse/HBASE-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6417: - Fix Version/s: (was: 0.94.4) 0.94.5 > hbck merges .META. regions if there's an old leftover > - > > Key: HBASE-6417 > URL: https://issues.apache.org/jira/browse/HBASE-6417 > Project: HBase > Issue Type: Bug >Reporter: Jean-Daniel Cryans > Fix For: 0.96.0, 0.94.5 > > Attachments: hbck.log > > > Trying to see what caused HBASE-6310, one of the things I figured is that the > bad .META. row is actually one from the time that we were permitting meta > splitting and that folder had just been staying there for a while. > So I tried to recreate the issue with -repair and it merged my good .META. > region with the one that's 3 years old that also has the same start key. I > ended up with a brand new .META. region! > I'll be attaching the full log in a separate file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5498: - Fix Version/s: (was: 0.94.4) 0.94.5 Moving out to 0.94.5. We should finish this, though. Not sure why it got stuck. > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.96.0, 0.94.5 > > Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, > HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, > HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch, > HBASE-5498_trunk_4.patch, HBASE-5498_trunk.patch > > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6416) hbck dies on NPE when a region folder exists but the table does not
[
https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-6416:
-
Fix Version/s: (was: 0.94.4)
0.94.5
Shifting to 0.94.5
> hbck dies on NPE when a region folder exists but the table does not
> ---
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.5
>
> Attachments: hbase-6416.patch, hbase-6416-v1.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for
> region null
> java.io.FileNotFoundException: File does not exist:
> /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at
> org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at
> org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir:
> hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!",
> tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any
> case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7283) Backport HBASE-6564 + HBASE-7202 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535727#comment-13535727 ] Lars Hofhansl commented on HBASE-7283: -- I see. +1 on commit. > Backport HBASE-6564 + HBASE-7202 to 0.94 > > > Key: HBASE-7283 > URL: https://issues.apache.org/jira/browse/HBASE-7283 > Project: HBase > Issue Type: Task >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7283.094.txt, HBASE-6564-0.94.patch, > HBASE-7202-0.94.patch > > > * HBASE-6564: HDFS space is not reclaimed when a column family is deleted > * HBASE-7202: Family Store Files are not archived on admin.deleteColumn() > (the second one is a fix for the first, to use the archiver) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6480) If callQueueSize exceed maxQueueSize, all call will be rejected, do not reject priorityCall
[
https://issues.apache.org/jira/browse/HBASE-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-6480:
-
Fix Version/s: (was: 0.94.4)
0.94.5
Where are we on this? Moving out to 0.94.5 for now.
> If callQueueSize exceed maxQueueSize, all call will be rejected, do not
> reject priorityCall
>
>
> Key: HBASE-6480
> URL: https://issues.apache.org/jira/browse/HBASE-6480
> Project: HBase
> Issue Type: Bug
>Reporter: binlijin
> Fix For: 0.96.0, 0.94.5
>
> Attachments: HBASE-6480-94.patch, HBASE-6480-trunk.patch
>
>
> Current if the callQueueSize exceed maxQueueSize, all call will be rejected,
> Should we let the priority Call pass through?
> Current:
> {code}
> if ((callSize + callQueueSize.get()) > maxQueueSize) {
> Call callTooBig = xxx
> return ;
> }
> if (priorityCallQueue != null && getQosLevel(param) > highPriorityLevel) {
> priorityCallQueue.put(call);
> updateCallQueueLenMetrics(priorityCallQueue);
> } else {
> callQueue.put(call); // queue the call; maybe blocked here
> updateCallQueueLenMetrics(callQueue);
> }
> {code}
> Should we change it to :
> {code}
> if (priorityCallQueue != null && getQosLevel(param) > highPriorityLevel) {
> priorityCallQueue.put(call);
> updateCallQueueLenMetrics(priorityCallQueue);
> } else {
> if ((callSize + callQueueSize.get()) > maxQueueSize) {
>Call callTooBig = xxx
>return ;
> }
> callQueue.put(call); // queue the call; maybe blocked here
> updateCallQueueLenMetrics(callQueue);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7337) SingleColumnValueFilter seems to get unavailble data
[
https://issues.apache.org/jira/browse/HBASE-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-7337:
-
Fix Version/s: (was: 0.94.4)
0.94.5
> SingleColumnValueFilter seems to get unavailble data
>
>
> Key: HBASE-7337
> URL: https://issues.apache.org/jira/browse/HBASE-7337
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.94.3, 0.96.0
> Environment: 0.94
>Reporter: Zhou wenjian
>Assignee: Zhou wenjian
> Fix For: 0.96.0, 0.94.5
>
>
> put multi versions of a row.
> r1 cf:q version:1 value:1
> r1 cf:q version:2 value:3
> r1 cf:q version:3 value:2
> the filter in scan is set as below:
> SingleColumnValueFilter valueF = new SingleColumnValueFilter(
> family,qualifier,CompareOp.EQUAL,new BinaryComparator(Bytes
> .toBytes("2")));
> then i found all of the three versions will be emmitted, then i set
> latestVersionOnly to false, the result does no change.
> {code}
> public ReturnCode filterKeyValue(KeyValue keyValue) {
> // System.out.println("REMOVE KEY=" + keyValue.toString() + ", value=" +
> Bytes.toString(keyValue.getValue()));
> if (this.matchedColumn) {
> // We already found and matched the single column, all keys now pass
> return ReturnCode.INCLUDE;
> } else if (this.latestVersionOnly && this.foundColumn) {
> // We found but did not match the single column, skip to next row
> return ReturnCode.NEXT_ROW;
> }
> if (!keyValue.matchingColumn(this.columnFamily, this.columnQualifier)) {
> return ReturnCode.INCLUDE;
> }
> foundColumn = true;
> if (filterColumnValue(keyValue.getBuffer(),
> keyValue.getValueOffset(), keyValue.getValueLength())) {
> return this.latestVersionOnly? ReturnCode.NEXT_ROW: ReturnCode.INCLUDE;
> }
> this.matchedColumn = true;
> return ReturnCode.INCLUDE;
> }
> {code}
> From the code above, it seeems that version 3 will be first emmited, and set
> matchedColumn to true, which leads the following version 2 and 1 emmited too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7283) Backport HBASE-6564 + HBASE-7202 to 0.94
[
https://issues.apache.org/jira/browse/HBASE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535724#comment-13535724
]
stack commented on HBASE-7283:
--
[~lhofhansl] Tracing, the IOE becomes a LOG#ERROR in the super
TableEventHandler class:
} catch (IOException e) {
LOG.error("Error manipulating table " + Bytes.toString(tableName), e);
At least it doesn't kill the server.
Leaving trash in the filesystem is probably ok logged as an error I'd say.
> Backport HBASE-6564 + HBASE-7202 to 0.94
>
>
> Key: HBASE-7283
> URL: https://issues.apache.org/jira/browse/HBASE-7283
> Project: HBase
> Issue Type: Task
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7283.094.txt, HBASE-6564-0.94.patch,
> HBASE-7202-0.94.patch
>
>
> * HBASE-6564: HDFS space is not reclaimed when a column family is deleted
> * HBASE-7202: Family Store Files are not archived on admin.deleteColumn()
> (the second one is a fix for the first, to use the archiver)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535721#comment-13535721 ] nkeywal commented on HBASE-7386: bq. 1) Do we think this is a useful direction? Personally, I do. We just need to have a default way to start/stop and ideally monitor HBase. Scripts are the simplest way. Supervisor is better. bq. 2) If so, what level of supervisor scripts do we provide? Samples are doomed to be broken. So I would go for Actual scripts + ssh scripting to actually run with a "start-hbase.sh", i.e. "start-supervised-hbase.sh". Is there any reason not to use supervisor by default (licenses, supported platforms?) > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.96.0 > > Attachments: HBASE-7386-v0.patch, supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7381) Lightweight data transfer for Class Result
[ https://issues.apache.org/jira/browse/HBASE-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7381: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to 0.94. Thanks for the patch Cheng. > Lightweight data transfer for Class Result > -- > > Key: HBASE-7381 > URL: https://issues.apache.org/jira/browse/HBASE-7381 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Cheng Hao >Priority: Trivial > Fix For: 0.94.4 > > Attachments: result_lightweight_copy_v2.patch > > > Currently,the data transferring between 2 Result objects in the same process, > will cause additional/unnecessary data parsing & copying; as we have to do > that via "Writables.copyWritable(result1, result2)", which internally is > serialization, data copying, and de-serialization. > The use case are quite common when integrated with Hadoop job running; > The protocol org.apache.hadoop.mapred.RecordReader defined in Hadoop, > provides 3 interfaces: > 1) K createKey(); > 2) V createValue(); > 3) boolean next(K key, V value) throws IOException; > In the 3rd method implementation, most likely requires the value (should be > Result object) to be filled, with the Result object from HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7387) StoreScanner need to be able to be subclassed
[ https://issues.apache.org/jira/browse/HBASE-7387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535717#comment-13535717 ] stack commented on HBASE-7387: -- +1 from me. Seems reasonable. > StoreScanner need to be able to be subclassed > - > > Key: HBASE-7387 > URL: https://issues.apache.org/jira/browse/HBASE-7387 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.0 >Reporter: Raymond Liu >Priority: Minor > Fix For: 0.96.0 > > Attachments: StoreScanner.patch > > > StoreScanner can be replaced by preStoreScannerOpen hook with CP. In order to > reuse most of the logic in current StoreScanner, subclass it might be the > best approaching. Thus a lot of private member need to be changed from > private to protected. > At present, in order to to implement a custom storescanner for dot > (HBASE-6805), only a few of the private member need to be changed as in the > attached patch, while should we change all the reasonable field from private > to protected? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[ https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535712#comment-13535712 ] Lars Hofhansl commented on HBASE-6945: -- Some minor comments: * Do we need to create the JVM object every time, or can be cache it statically like we did with osStats? * After the new JVM() the null check is unnecessary. * in runUnixMXBeanMethod, do can we cache the outcome of classForName (even when that is null), so that we do not have to make this call each time? Maybe these are not problems, because this is not a hot code path? > Compilation errors when using non-Sun JDKs to build HBase-0.94 > -- > > Key: HBASE-6945 > URL: https://issues.apache.org/jira/browse/HBASE-6945 > Project: HBase > Issue Type: Sub-task > Components: build >Affects Versions: 0.94.1 > Environment: RHEL 6.3, IBM Java 7 >Reporter: Kumar Ravi >Assignee: Kumar Ravi > Labels: patch > Fix For: 0.96.0, 0.94.4 > > Attachments: 6945-v2.txt, HBASE_0.94.3.patch, HBASE-6945.patch > > > When using IBM Java 7 to build HBase-0.94.1, the following comilation error > is seen. > [INFO] - > [ERROR] COMPILATION ERROR : > [INFO] - > [ERROR] > /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25] > error: package com.sun.management does not exist > [ERROR] > /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25] > error: cannot find symbol > [ERROR] symbol: class UnixOperatingSystemMXBean > location: class ResourceAnalyzer > /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29] > error: cannot find symbol > [ERROR] symbol: class UnixOperatingSystemMXBean > location: class ResourceAnalyzer > /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23] > error: cannot find symbol > [INFO] 4 errors > [INFO] - > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > I have a patch available which should work for all JDKs including Sun. > I am in the process of testing this patch. Preliminary tests indicate the > build is working fine with this patch. I will post this patch when I am done > testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6155) [copytable] Unexpected behavior if --starttime is not specifed but --endtime is.
[
https://issues.apache.org/jira/browse/HBASE-6155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535709#comment-13535709
]
Hadoop QA commented on HBASE-6155:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561639/6155-patch-1.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 28 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 232
release audit warnings (more than the trunk's current 84 warnings).
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:red}-1 core zombie tests{color}. There are zombie tests. See build
logs for details.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3604//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3604//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3604//console
This message is automatically generated.
> [copytable] Unexpected behavior if --starttime is not specifed but --endtime
> is.
>
>
> Key: HBASE-6155
> URL: https://issues.apache.org/jira/browse/HBASE-6155
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.90.6, 0.92.1, 0.94.0, 0.96.0
>Reporter: Jonathan Hsieh
>Assignee: Prasanna
> Labels: noob
> Fix For: 0.96.0
>
> Attachments: 6155.diff, 6155-patch-1.diff
>
>
> If one uses copytable and specifies only an endtime, I'd expect to include
> all rows from unix epoch time upto the specified endtime. Instead, it copies
> all the rows.
> The workaround for copies with this kind of range is to specify --startime=1
> (Note not --starttime=0), which is also unintuitive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7210) Backport HBASE-6059 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535707#comment-13535707 ] Lars Hofhansl commented on HBASE-7210: -- Somehow this patch scares me a bit. Since I cannot really why, though, and since it fixes a real problem, I'm fine with committing it to 0.94. > Backport HBASE-6059 to 0.94 > --- > > Key: HBASE-7210 > URL: https://issues.apache.org/jira/browse/HBASE-7210 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: ramkrishna.s.vasudevan > Fix For: 0.94.4 > > Attachments: 6059-94.patch > > > HBASE-6059 seems to be an important issue. Chunhui has already given a patch > for 94. Need to rebase if it does not apply cleanly. > Raising a new one as the old issue is already closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7387) StoreScanner need to be able to be subclassed
[ https://issues.apache.org/jira/browse/HBASE-7387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Liu updated HBASE-7387: --- Attachment: StoreScanner.patch > StoreScanner need to be able to be subclassed > - > > Key: HBASE-7387 > URL: https://issues.apache.org/jira/browse/HBASE-7387 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.0 >Reporter: Raymond Liu >Priority: Minor > Fix For: 0.96.0 > > Attachments: StoreScanner.patch > > > StoreScanner can be replaced by preStoreScannerOpen hook with CP. In order to > reuse most of the logic in current StoreScanner, subclass it might be the > best approaching. Thus a lot of private member need to be changed from > private to protected. > At present, in order to to implement a custom storescanner for dot > (HBASE-6805), only a few of the private member need to be changed as in the > attached patch, while should we change all the reasonable field from private > to protected? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7387) StoreScanner need to be able to be subclassed
Raymond Liu created HBASE-7387: -- Summary: StoreScanner need to be able to be subclassed Key: HBASE-7387 URL: https://issues.apache.org/jira/browse/HBASE-7387 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.96.0 Reporter: Raymond Liu Priority: Minor Fix For: 0.96.0 StoreScanner can be replaced by preStoreScannerOpen hook with CP. In order to reuse most of the logic in current StoreScanner, subclass it might be the best approaching. Thus a lot of private member need to be changed from private to protected. At present, in order to to implement a custom storescanner for dot (HBASE-6805), only a few of the private member need to be changed as in the attached patch, while should we change all the reasonable field from private to protected? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6628) Add HBASE-6059 to 0.94 branch
[ https://issues.apache.org/jira/browse/HBASE-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-6628. -- Resolution: Duplicate Fix Version/s: (was: 0.94.5) Actually this is a dup of HBASE-7210 (which has a patch on it) > Add HBASE-6059 to 0.94 branch > - > > Key: HBASE-6628 > URL: https://issues.apache.org/jira/browse/HBASE-6628 > Project: HBase > Issue Type: Task >Reporter: stack > > Look at adding HBASE-6059 to 0.94. Its in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6862) Apply custom umask to HLog
[ https://issues.apache.org/jira/browse/HBASE-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-6862. -- Resolution: Later Fix Version/s: (was: 0.94.4) (was: 0.96.0) "Closing" > Apply custom umask to HLog > -- > > Key: HBASE-6862 > URL: https://issues.apache.org/jira/browse/HBASE-6862 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl > > See parent for details. > Gist is that it should be possible to restrict the web ui to browse the > directory but not allow access to the data files itself. > Parent does for HFiles, but forgot to do this for the HLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6628) Add HBASE-6059 to 0.94 branch
[ https://issues.apache.org/jira/browse/HBASE-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6628: - Fix Version/s: (was: 0.94.4) 0.94.5 Moving on to 0.94.5 :) > Add HBASE-6059 to 0.94 branch > - > > Key: HBASE-6628 > URL: https://issues.apache.org/jira/browse/HBASE-6628 > Project: HBase > Issue Type: Task >Reporter: stack > Fix For: 0.94.5 > > > Look at adding HBASE-6059 to 0.94. Its in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7091) support custom GC options in hbase-env.sh
[
https://issues.apache.org/jira/browse/HBASE-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535695#comment-13535695
]
Lars Hofhansl commented on HBASE-7091:
--
I think we should commit a version of this.
> support custom GC options in hbase-env.sh
> -
>
> Key: HBASE-7091
> URL: https://issues.apache.org/jira/browse/HBASE-7091
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 0.94.4
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Labels: newbie
> Fix For: 0.94.4
>
> Attachments: hbase-7091-v1.patch
>
>
> When running things like bin/start-hbase and bin/hbase-daemon.sh start
> [master|regionserver|etc] we end up setting HBASE_OPTS property a couple
> times via calling hbase-env.sh. This is generally not a problem for most
> cases, but when you want to set your own GC log properties, one would think
> you should set HBASE_GC_OPTS, which get added to HBASE_OPTS.
> NOPE! That would make too much sense.
> Running bin/hbase-daemons.sh will run bin/hbase-daemon.sh with the daemons it
> needs to start. Each time through hbase-daemon.sh we also call bin/hbase.
> This isn't a big deal except for each call to hbase-daemon.sh, we also source
> hbase-env.sh twice (once in the script and once in bin/hbase). This is
> important for my next point.
> Note that to turn on GC logging, you uncomment:
> {code}
> # export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps $HBASE_GC_OPTS"
> {code}
> and then to log to a gc file for each server, you then uncomment:
> {code}
> # export HBASE_USE_GC_LOGFILE=true
> {code}
> in hbase-env.sh
> On the first pass through hbase-daemon.sh, HBASE_GC_OPTS isn't set, so
> HBASE_OPTS doesn't get anything funky, but we set HBASE_USE_GC_LOGFILE, which
> then sets HBASE_GC_OPTS to the log file (-Xloggc:...). Then in bin/hbase we
> again run hbase-env.sh, which now hs HBASE_GC_OPTS set, adding the GC file.
> This isn't a general problem because HBASE_OPTS is set without prefixing the
> existing HBASE_OPTS (eg. HBASE_OPTS="$HBASE_OPTS ..."), allowing easy
> updating. However, GC OPTS don't work the same and this is really odd
> behavior when you want to set your own GC opts, which can include turning on
> GC log rolling (yes, yes, they really are jvm opts, but they ought to support
> their own param, to help minimize clutter).
> The simple version of this patch will just add an idempotent GC option to
> hbase-env.sh and some comments that uncommenting
> {code}
> # export HBASE_USE_GC_LOGFILE=true
> {code}
> will lead to a custom gc log file per server (along with an example name), so
> you don't need to set "-Xloggc".
> The more complex solution does the above and also solves the multiple calls
> to hbase-env.sh so we can be sane about how all this works. Note that to fix
> this, hbase-daemon.sh just needs to read in HBASE_USE_GC_LOGFILE after
> sourcing hbase-env.sh and then update HBASE_OPTS. Oh and also not source
> hbase-env.sh in bin/hbase.
> Even further, we might want to consider adding options just for cases where
> we don't need gc logging - i.e. the shell, the config reading tool, hcbk,
> etc. This is the hardest version to handle since the first couple will
> willy-nilly apply the gc options.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3852) ThriftServer leaks scanners
[ https://issues.apache.org/jira/browse/HBASE-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-3852: - Fix Version/s: (was: 0.94.4) 0.94.5 And on to 0.94.5 > ThriftServer leaks scanners > --- > > Key: HBASE-3852 > URL: https://issues.apache.org/jira/browse/HBASE-3852 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 0.90.2 >Reporter: Jean-Daniel Cryans > Fix For: 0.94.5 > > Attachments: 3852.txt, thrift2-scanner.patch > > > The scannerMap in ThriftServer relies on the user to clean it by closing the > scanner. If that doesn't happen, the ResultScanner will stay in the thrift > server's memory and if any pre-fetching was done, it will also start > accumulating Results (with all their data). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7313) ColumnPaginationFilter should reset count when moving to NEXT_ROW
[ https://issues.apache.org/jira/browse/HBASE-7313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7313: - Resolution: Invalid Fix Version/s: (was: 0.94.4) (was: 0.96.0) Status: Resolved (was: Patch Available) I am closing this for now. Please reopen if this is a real issue. (Note that it might be working as designed, but not work as expected) > ColumnPaginationFilter should reset count when moving to NEXT_ROW > - > > Key: HBASE-7313 > URL: https://issues.apache.org/jira/browse/HBASE-7313 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 0.94.3, 0.96.0 >Reporter: Varun Sharma >Assignee: Varun Sharma > Attachments: 7313-0.94.txt, 7313-trunk.txt > > > ColumnPaginationFilter does not reset count to zero on moving to next row. > Hence, if we have already gotten "limit" number of columns - the subsequent > rows will always return 0 columns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7375) Acquire readLock do not apply timeout in flushcache
[
https://issues.apache.org/jira/browse/HBASE-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl resolved HBASE-7375.
--
Resolution: Duplicate
Fix Version/s: (was: 0.94.4)
(was: 0.96.0)
Yes. It's a dup. closing.
> Acquire readLock do not apply timeout in flushcache
> ---
>
> Key: HBASE-7375
> URL: https://issues.apache.org/jira/browse/HBASE-7375
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.96.0, 0.94.4
>Reporter: binlijin
>
> {code}
> HRegion
> public boolean flushcache() throws IOException {
>lock(lock.readLock());
> }
> {code}
> The HRegion.flushcache is called by the normal flush cache, so if we use a
> timeout, the MemStoreFlusher may be get a RegionTooBusyException, it is safe
> to do not use a timeout.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6990) Pretty print TTL
[ https://issues.apache.org/jira/browse/HBASE-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6990: - Fix Version/s: (was: 0.94.4) 0.94.5 > Pretty print TTL > > > Key: HBASE-6990 > URL: https://issues.apache.org/jira/browse/HBASE-6990 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.2, 0.94.2 >Reporter: Jean-Daniel Cryans >Assignee: Kevin Odell >Priority: Minor > Fix For: 0.92.3, 0.96.0, 0.94.5 > > > I've seen a lot of users getting confused by the TTL configuration and I > think that if we just pretty printed it it would solve most of the issues. > For example, let's say a user wanted to set a TTL of 90 days. That would be > 7776000. But let's say that it was typo'd to 7776 instead, it gives you > 900 days! > So when we print the TTL we could do something like "x days, x hours, x > minutes, x seconds (real_ttl_value)". This would also help people when they > use ms instead of seconds as they would see really big values in there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6919) Remove unnecessary cast from Bytes.readVLong
[
https://issues.apache.org/jira/browse/HBASE-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-6919:
-
Fix Version/s: (was: 0.94.4)
> Remove unnecessary cast from Bytes.readVLong
>
>
> Key: HBASE-6919
> URL: https://issues.apache.org/jira/browse/HBASE-6919
> Project: HBase
> Issue Type: Bug
>Reporter: James Taylor
>Priority: Minor
> Fix For: 0.96.0
>
>
> Remove the throws IOException so that caller doesn't have to catch and ignore.
> {code}
> public static long readVLong(final byte [] buffer, final int offset)
> throws IOException
> {code}
> Also, add
> {code}
> public static int readVInt(final byte [] buffer, final int offset)
> throws IOException {
> return (int)readVLong(buffer,offset);
> }
> {code}
> and these are useful too:
> {code}
> /**
> * Put long as variable length encoded number at the offset in
> * the result byte array.
> * @param vint Integer to make a vint of.
> * @param result buffer to put vint into
> * @return Vint length in bytes of vint
> */
> public static int vintToBytes(byte[] result, int offset, final long vint)
> {
> long i = vint;
> if (i >= -112 && i <= 127) {
> result[offset] = (byte) i;
> return 1;
> }
> int len = -112;
> if (i < 0) {
> i ^= -1L; // take one's complement'
> len = -120;
> }
> long tmp = i;
> while (tmp != 0) {
> tmp = tmp >> 8;
> len--;
> }
> result[offset++] = (byte) len;
> len = (len < -120) ? -(len + 120) : -(len + 112);
> for (int idx = len; idx != 0; idx--) {
> int shiftbits = (idx - 1) * 8;
> long mask = 0xFFL << shiftbits;
> result[offset++] = (byte)((i & mask) >> shiftbits);
> }
> return len + 1;
> }
> /**
> * Decode a vint from the buffer pointed at to by ptr and
> * increment the offset of the ptr by the length of the
> * vint.
> * @param ptr a pointer to a byte array buffer
> * @return the decoded vint value as an int
> */
> public static int vintFromBytes(ImmutableBytesWritable ptr) {
> return (int) vlongFromBytes(ptr);
> }
>
> /**
> * Decode a vint from the buffer pointed at to by ptr and
> * increment the offset of the ptr by the length of the
> * vint.
> * @param ptr a pointer to a byte array buffer
> * @return the decoded vint value as a long
> */
> public static long vlongFromBytes(ImmutableBytesWritable ptr) {
> final byte [] buffer = ptr.get();
> final int offset = ptr.getOffset();
> byte firstByte = buffer[offset];
> int len = WritableUtils.decodeVIntSize(firstByte);
> if (len == 1) {
> ptr.set(buffer, offset+1, ptr.getLength());
> return firstByte;
> }
> long i = 0;
> for (int idx = 0; idx < len-1; idx++) {
> byte b = buffer[offset + 1 + idx];
> i = i << 8;
> i = i | (b & 0xFF);
> }
> ptr.set(buffer, offset+len, ptr.getLength());
> return (WritableUtils.isNegativeVInt(firstByte) ? ~i : i);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7283) Backport HBASE-6564 + HBASE-7202 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535689#comment-13535689 ] Lars Hofhansl commented on HBASE-7283: -- Patch looks good. +1 One question: Should we just log a warning when the column family's directory cannot be deleted (instead of throwing IOException)? > Backport HBASE-6564 + HBASE-7202 to 0.94 > > > Key: HBASE-7283 > URL: https://issues.apache.org/jira/browse/HBASE-7283 > Project: HBase > Issue Type: Task >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7283.094.txt, HBASE-6564-0.94.patch, > HBASE-7202-0.94.patch > > > * HBASE-6564: HDFS space is not reclaimed when a column family is deleted > * HBASE-7202: Family Store Files are not archived on admin.deleteColumn() > (the second one is a fix for the first, to use the archiver) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7226) HRegion.checkAndMutate uses incorrect comparison result for <, <=, > and >=
[
https://issues.apache.org/jira/browse/HBASE-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535683#comment-13535683
]
Lars Hofhansl edited comment on HBASE-7226 at 12/19/12 5:59 AM:
Moving to next point release for now. We can pull it back if we like.
was (Author: lhofhansl):
Moving to next point release for now. We can pull it back and we like.
> HRegion.checkAndMutate uses incorrect comparison result for <, <=, > and >=
> ---
>
> Key: HBASE-7226
> URL: https://issues.apache.org/jira/browse/HBASE-7226
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.2
> Environment: 0.94.2
>Reporter: Feng Honghua
>Priority: Minor
> Fix For: 0.94.5
>
> Attachments: HRegion_HBASE_7226_0.94.2.patch
>
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> in HRegion.checkAndMutate, incorrect comparison results are used for <, <=, >
> and >=, as below:
> switch (compareOp) {
> case LESS:
> matches = compareResult <= 0; // should be '<' here
> break;
> case LESS_OR_EQUAL:
> matches = compareResult < 0; // should be '<=' here
> break;
> case EQUAL:
> matches = compareResult == 0;
> break;
> case NOT_EQUAL:
> matches = compareResult != 0;
> break;
> case GREATER_OR_EQUAL:
> matches = compareResult > 0; // should be '>=' here
> break;
> case GREATER:
> matches = compareResult >= 0; // should be '>' here
> break;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7226) HRegion.checkAndMutate uses incorrect comparison result for <, <=, > and >=
[
https://issues.apache.org/jira/browse/HBASE-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-7226:
-
Fix Version/s: (was: 0.94.4)
0.94.5
Moving to next point release for now. We can pull it back and we like.
> HRegion.checkAndMutate uses incorrect comparison result for <, <=, > and >=
> ---
>
> Key: HBASE-7226
> URL: https://issues.apache.org/jira/browse/HBASE-7226
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.2
> Environment: 0.94.2
>Reporter: Feng Honghua
>Priority: Minor
> Fix For: 0.94.5
>
> Attachments: HRegion_HBASE_7226_0.94.2.patch
>
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> in HRegion.checkAndMutate, incorrect comparison results are used for <, <=, >
> and >=, as below:
> switch (compareOp) {
> case LESS:
> matches = compareResult <= 0; // should be '<' here
> break;
> case LESS_OR_EQUAL:
> matches = compareResult < 0; // should be '<=' here
> break;
> case EQUAL:
> matches = compareResult == 0;
> break;
> case NOT_EQUAL:
> matches = compareResult != 0;
> break;
> case GREATER_OR_EQUAL:
> matches = compareResult > 0; // should be '>=' here
> break;
> case GREATER:
> matches = compareResult >= 0; // should be '>' here
> break;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7226) HRegion.checkAndMutate uses incorrect comparison result for <, <=, > and >=
[
https://issues.apache.org/jira/browse/HBASE-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535682#comment-13535682
]
Lars Hofhansl commented on HBASE-7226:
--
Although it does look like the code is indeed incorrect.
> HRegion.checkAndMutate uses incorrect comparison result for <, <=, > and >=
> ---
>
> Key: HBASE-7226
> URL: https://issues.apache.org/jira/browse/HBASE-7226
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.2
> Environment: 0.94.2
>Reporter: Feng Honghua
>Priority: Minor
> Fix For: 0.94.4
>
> Attachments: HRegion_HBASE_7226_0.94.2.patch
>
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> in HRegion.checkAndMutate, incorrect comparison results are used for <, <=, >
> and >=, as below:
> switch (compareOp) {
> case LESS:
> matches = compareResult <= 0; // should be '<' here
> break;
> case LESS_OR_EQUAL:
> matches = compareResult < 0; // should be '<=' here
> break;
> case EQUAL:
> matches = compareResult == 0;
> break;
> case NOT_EQUAL:
> matches = compareResult != 0;
> break;
> case GREATER_OR_EQUAL:
> matches = compareResult > 0; // should be '>=' here
> break;
> case GREATER:
> matches = compareResult >= 0; // should be '>' here
> break;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7381) Lightweight data transfer for Class Result
[ https://issues.apache.org/jira/browse/HBASE-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535676#comment-13535676 ] Cheng Hao commented on HBASE-7381: -- oh, sorry, I will take care of that next time. Thanks Lars. > Lightweight data transfer for Class Result > -- > > Key: HBASE-7381 > URL: https://issues.apache.org/jira/browse/HBASE-7381 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Cheng Hao >Priority: Trivial > Fix For: 0.94.4 > > Attachments: result_lightweight_copy_v2.patch > > > Currently,the data transferring between 2 Result objects in the same process, > will cause additional/unnecessary data parsing & copying; as we have to do > that via "Writables.copyWritable(result1, result2)", which internally is > serialization, data copying, and de-serialization. > The use case are quite common when integrated with Hadoop job running; > The protocol org.apache.hadoop.mapred.RecordReader defined in Hadoop, > provides 3 interfaces: > 1) K createKey(); > 2) V createValue(); > 3) boolean next(K key, V value) throws IOException; > In the 3rd method implementation, most likely requires the value (should be > Result object) to be filled, with the Result object from HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7226) HRegion.checkAndMutate uses incorrect comparison result for <, <=, > and >=
[
https://issues.apache.org/jira/browse/HBASE-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535674#comment-13535674
]
Lars Hofhansl commented on HBASE-7226:
--
Yeah, I doubt this too. This would not have gone unnoticed for so long.
> HRegion.checkAndMutate uses incorrect comparison result for <, <=, > and >=
> ---
>
> Key: HBASE-7226
> URL: https://issues.apache.org/jira/browse/HBASE-7226
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.2
> Environment: 0.94.2
>Reporter: Feng Honghua
>Priority: Minor
> Fix For: 0.94.4
>
> Attachments: HRegion_HBASE_7226_0.94.2.patch
>
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> in HRegion.checkAndMutate, incorrect comparison results are used for <, <=, >
> and >=, as below:
> switch (compareOp) {
> case LESS:
> matches = compareResult <= 0; // should be '<' here
> break;
> case LESS_OR_EQUAL:
> matches = compareResult < 0; // should be '<=' here
> break;
> case EQUAL:
> matches = compareResult == 0;
> break;
> case NOT_EQUAL:
> matches = compareResult != 0;
> break;
> case GREATER_OR_EQUAL:
> matches = compareResult > 0; // should be '>=' here
> break;
> case GREATER:
> matches = compareResult >= 0; // should be '>' here
> break;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7381) Lightweight data transfer for Class Result
[ https://issues.apache.org/jira/browse/HBASE-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535671#comment-13535671 ] Lars Hofhansl commented on HBASE-7381: -- I'll commit v2 tonight unless I hear objections. (BTW... It it is typically better to keep the old attachments around so that comments referring to those still make sense when somebody sees this later) > Lightweight data transfer for Class Result > -- > > Key: HBASE-7381 > URL: https://issues.apache.org/jira/browse/HBASE-7381 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Cheng Hao >Priority: Trivial > Fix For: 0.94.4 > > Attachments: result_lightweight_copy_v2.patch > > > Currently,the data transferring between 2 Result objects in the same process, > will cause additional/unnecessary data parsing & copying; as we have to do > that via "Writables.copyWritable(result1, result2)", which internally is > serialization, data copying, and de-serialization. > The use case are quite common when integrated with Hadoop job running; > The protocol org.apache.hadoop.mapred.RecordReader defined in Hadoop, > provides 3 interfaces: > 1) K createKey(); > 2) V createValue(); > 3) boolean next(K key, V value) throws IOException; > In the 3rd method implementation, most likely requires the value (should be > Result object) to be filled, with the Result object from HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7381) Lightweight data transfer for Class Result
[ https://issues.apache.org/jira/browse/HBASE-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535669#comment-13535669 ] Lars Hofhansl commented on HBASE-7381: -- Well yeah, it's different. bytes is gone from trunk. But the API should be the same. > Lightweight data transfer for Class Result > -- > > Key: HBASE-7381 > URL: https://issues.apache.org/jira/browse/HBASE-7381 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Cheng Hao >Priority: Trivial > Fix For: 0.94.4 > > Attachments: result_lightweight_copy_v2.patch > > > Currently,the data transferring between 2 Result objects in the same process, > will cause additional/unnecessary data parsing & copying; as we have to do > that via "Writables.copyWritable(result1, result2)", which internally is > serialization, data copying, and de-serialization. > The use case are quite common when integrated with Hadoop job running; > The protocol org.apache.hadoop.mapred.RecordReader defined in Hadoop, > provides 3 interfaces: > 1) K createKey(); > 2) V createValue(); > 3) boolean next(K key, V value) throws IOException; > In the 3rd method implementation, most likely requires the value (should be > Result object) to be filled, with the Result object from HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7381) Lightweight data transfer for Class Result
[
https://issues.apache.org/jira/browse/HBASE-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535667#comment-13535667
]
Ted Yu commented on HBASE-7381:
---
Here is Result.copyFrom() in trunk:
{code}
public void copyFrom(Result other) {
this.row = null;
this.familyMap = null;
this.kvs = other.kvs;
}
{code}
It is actually different from the method in patch.
> Lightweight data transfer for Class Result
> --
>
> Key: HBASE-7381
> URL: https://issues.apache.org/jira/browse/HBASE-7381
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Cheng Hao
>Priority: Trivial
> Fix For: 0.94.4
>
> Attachments: result_lightweight_copy_v2.patch
>
>
> Currently,the data transferring between 2 Result objects in the same process,
> will cause additional/unnecessary data parsing & copying; as we have to do
> that via "Writables.copyWritable(result1, result2)", which internally is
> serialization, data copying, and de-serialization.
> The use case are quite common when integrated with Hadoop job running;
> The protocol org.apache.hadoop.mapred.RecordReader defined in Hadoop,
> provides 3 interfaces:
> 1) K createKey();
> 2) V createValue();
> 3) boolean next(K key, V value) throws IOException;
> In the 3rd method implementation, most likely requires the value (should be
> Result object) to be filled, with the Result object from HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5778) Turn on WAL compression by default
[ https://issues.apache.org/jira/browse/HBASE-5778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535665#comment-13535665 ] stack commented on HBASE-5778: -- I'm ok w/ committing it but I think it should be off in 0.96. It looks too flakey to be on by default (thanks for the OOME reminder Ram). > Turn on WAL compression by default > -- > > Key: HBASE-5778 > URL: https://issues.apache.org/jira/browse/HBASE-5778 > Project: HBase > Issue Type: Improvement >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5778.addendum, 5778-addendum.txt, HBASE-5778-0.94.patch, > HBASE-5778-0.94-v2.patch, HBASE-5778-0.94-v3.patch, HBASE-5778-0.94-v4.patch, > HBASE-5778-0.94-v5.patch, HBASE-5778-0.94-v6.patch, HBASE-5778-0.94-v7.patch, > HBASE-5778.patch, HBASE-5778-trunk-v6.patch, HBASE-5778-trunk-v7.patch > > > I ran some tests to verify if WAL compression should be turned on by default. > For a use case where it's not very useful (values two order of magnitude > bigger than the keys), the insert time wasn't different and the CPU usage 15% > higher (150% CPU usage VS 130% when not compressing the WAL). > When values are smaller than the keys, I saw a 38% improvement for the insert > run time and CPU usage was 33% higher (600% CPU usage VS 450%). I'm not sure > WAL compression accounts for all the additional CPU usage, it might just be > that we're able to insert faster and we spend more time in the MemStore per > second (because our MemStores are bad when they contain tens of thousands of > values). > Those are two extremes, but it shows that for the price of some CPU we can > save a lot. My machines have 2 quads with HT, so I still had a lot of idle > CPUs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7381) Lightweight data transfer for Class Result
[
https://issues.apache.org/jira/browse/HBASE-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535661#comment-13535661
]
Hadoop QA commented on HBASE-7381:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561644/result_lightweight_copy_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3605//console
This message is automatically generated.
> Lightweight data transfer for Class Result
> --
>
> Key: HBASE-7381
> URL: https://issues.apache.org/jira/browse/HBASE-7381
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Cheng Hao
>Priority: Trivial
> Fix For: 0.94.4
>
> Attachments: result_lightweight_copy_v2.patch
>
>
> Currently,the data transferring between 2 Result objects in the same process,
> will cause additional/unnecessary data parsing & copying; as we have to do
> that via "Writables.copyWritable(result1, result2)", which internally is
> serialization, data copying, and de-serialization.
> The use case are quite common when integrated with Hadoop job running;
> The protocol org.apache.hadoop.mapred.RecordReader defined in Hadoop,
> provides 3 interfaces:
> 1) K createKey();
> 2) V createValue();
> 3) boolean next(K key, V value) throws IOException;
> In the 3rd method implementation, most likely requires the value (should be
> Result object) to be filled, with the Result object from HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7372) Check in the generated website so can point apache infrastructure at what to publish as our hbase.apache.org
[
https://issues.apache.org/jira/browse/HBASE-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535658#comment-13535658
]
stack commented on HBASE-7372:
--
Changed my mind on checking site in under trunk. Instead, will check it in
under a directory adjacent to trunk named hbase.apache.org.
Reverting all of the above commits.
> Check in the generated website so can point apache infrastructure at what to
> publish as our hbase.apache.org
>
>
> Key: HBASE-7372
> URL: https://issues.apache.org/jira/browse/HBASE-7372
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Attachments: 7372v2.txt
>
>
> January 1st is deadline for changing how we publish our website. We may no
> longer rsync out to people.apache.org. Apache infrastructure supplies two
> options here: http://www.apache.org/dev/project-site.html We could redo our
> site in apache cms format. Or we could just use svnpubsub and keep on w/ how
> the site is currently generated and on checkin, have it autopublished. I'll
> go the latter route unless I hear otherwise.
> For svnpubsub, we need to point apache infrastructure at a directory that has
> our checkedin site in it. I was thinking ${hbasedir}/hbase.apache.org
> Let me raise this on the dev list too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7381) Lightweight data transfer for Class Result
[ https://issues.apache.org/jira/browse/HBASE-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Hao updated HBASE-7381: - Attachment: (was: result_lightweight_copy.patch) > Lightweight data transfer for Class Result > -- > > Key: HBASE-7381 > URL: https://issues.apache.org/jira/browse/HBASE-7381 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Cheng Hao >Priority: Trivial > Fix For: 0.94.4 > > Attachments: result_lightweight_copy_v2.patch > > > Currently,the data transferring between 2 Result objects in the same process, > will cause additional/unnecessary data parsing & copying; as we have to do > that via "Writables.copyWritable(result1, result2)", which internally is > serialization, data copying, and de-serialization. > The use case are quite common when integrated with Hadoop job running; > The protocol org.apache.hadoop.mapred.RecordReader defined in Hadoop, > provides 3 interfaces: > 1) K createKey(); > 2) V createValue(); > 3) boolean next(K key, V value) throws IOException; > In the 3rd method implementation, most likely requires the value (should be > Result object) to be filled, with the Result object from HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7381) Lightweight data transfer for Class Result
[ https://issues.apache.org/jira/browse/HBASE-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Hao updated HBASE-7381: - Attachment: result_lightweight_copy_v2.patch Thanks, Lars. I didn't notice that in trunk. It will be helpful if the changes can apply to 0.94, too. I created a new patch with slightly different changes compared to the trunk, but keep the same method name. > Lightweight data transfer for Class Result > -- > > Key: HBASE-7381 > URL: https://issues.apache.org/jira/browse/HBASE-7381 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Cheng Hao >Priority: Trivial > Fix For: 0.94.4 > > Attachments: result_lightweight_copy_v2.patch > > > Currently,the data transferring between 2 Result objects in the same process, > will cause additional/unnecessary data parsing & copying; as we have to do > that via "Writables.copyWritable(result1, result2)", which internally is > serialization, data copying, and de-serialization. > The use case are quite common when integrated with Hadoop job running; > The protocol org.apache.hadoop.mapred.RecordReader defined in Hadoop, > provides 3 interfaces: > 1) K createKey(); > 2) V createValue(); > 3) boolean next(K key, V value) throws IOException; > In the 3rd method implementation, most likely requires the value (should be > Result object) to be filled, with the Result object from HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6585) Audit log messages should contain info about the higher level operation being executed
[ https://issues.apache.org/jira/browse/HBASE-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535649#comment-13535649 ] Lars Hofhansl commented on HBASE-6585: -- +1 for 0.94 as well. Wanna make a 0.94 patch too, [~mbertozzi]? If not, I'll make one (maybe trunk can just be applied with -p1 anyway). > Audit log messages should contain info about the higher level operation being > executed > -- > > Key: HBASE-6585 > URL: https://issues.apache.org/jira/browse/HBASE-6585 > Project: HBase > Issue Type: Improvement > Components: security >Affects Versions: 0.96.0 >Reporter: Marcelo Vanzin >Assignee: Matteo Bertozzi >Priority: Minor > Labels: acl > Fix For: 0.96.0, 0.94.4 > > Attachments: HBASE-6585-v0.patch, HBASE-6585-v1.patch, > HBASE-6585-v2.patch > > > Currently, audit log messages contains the "action" for which access was > checked; this is one of READ, WRITE, CREATE or ADMIN. > These give very little information to the person digging into the logs about > what was done, though. You can't ask "who deleted rows from table x?", > because "delete" is translated to a "WRITE" action. > It would be nice if the audit logs contained the higher-level operation, > either replacing or in addition to the RWCA information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5778) Turn on WAL compression by default
[ https://issues.apache.org/jira/browse/HBASE-5778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535650#comment-13535650 ] ramkrishna.s.vasudevan commented on HBASE-5778: --- @JD http://mail-archives.apache.org/mod_mbox/hbase-dev/201205.mbox/%3C00bc01cd31e6$7caf1320$760d3960$%25vasude...@huawei.com%3E. Do we still get the OOME with WAL compression? > Turn on WAL compression by default > -- > > Key: HBASE-5778 > URL: https://issues.apache.org/jira/browse/HBASE-5778 > Project: HBase > Issue Type: Improvement >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5778.addendum, 5778-addendum.txt, HBASE-5778-0.94.patch, > HBASE-5778-0.94-v2.patch, HBASE-5778-0.94-v3.patch, HBASE-5778-0.94-v4.patch, > HBASE-5778-0.94-v5.patch, HBASE-5778-0.94-v6.patch, HBASE-5778-0.94-v7.patch, > HBASE-5778.patch, HBASE-5778-trunk-v6.patch, HBASE-5778-trunk-v7.patch > > > I ran some tests to verify if WAL compression should be turned on by default. > For a use case where it's not very useful (values two order of magnitude > bigger than the keys), the insert time wasn't different and the CPU usage 15% > higher (150% CPU usage VS 130% when not compressing the WAL). > When values are smaller than the keys, I saw a 38% improvement for the insert > run time and CPU usage was 33% higher (600% CPU usage VS 450%). I'm not sure > WAL compression accounts for all the additional CPU usage, it might just be > that we're able to insert faster and we spend more time in the MemStore per > second (because our MemStores are bad when they contain tens of thousands of > values). > Those are two extremes, but it shows that for the price of some CPU we can > save a lot. My machines have 2 quads with HT, so I still had a lot of idle > CPUs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6294) Detect leftover data in ZK after a user delete all its HBase data
[
https://issues.apache.org/jira/browse/HBASE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-6294:
-
Priority: Major (was: Critical)
Fix Version/s: (was: 0.94.4)
0.94.5
No fix... Does not seem to be critical to anyone :)
> Detect leftover data in ZK after a user delete all its HBase data
> -
>
> Key: HBASE-6294
> URL: https://issues.apache.org/jira/browse/HBASE-6294
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.94.0
>Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.5
>
>
> It seems we have a new failure mode when a user deletes the hbase root.dir
> but doesn't delete the ZK data. For example a user on IRC came with this log:
> {noformat}
> 2012-06-30 09:07:48,017 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open
> region: kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02.
> 2012-06-30 09:07:48,017 WARN org.apache.hadoop.hbase.util.FSTableDescriptors:
> The following folder is in HBase's root directory and doesn't contain a table
> descriptor, do consider deleting it: kw
> 2012-06-30 09:07:48,018 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:34193-0x1383bfe01b70001 Attempting to transition node
> 2e8a318837602c9c9961e9d690b7fd02 from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 2012-06-30 09:07:48,018 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=M_ZK_REGION_OFFLINE, server=localhost,50890,1341036299694,
> region=2e8a318837602c9c9961e9d690b7fd02
> 2012-06-30 09:07:48,020 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_FAILED_OPEN, server=localhost,34193,1341036300138,
> region=b254af24c9127b8bb22cb6d24e523dad
> 2012-06-30 09:07:48,020 DEBUG
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
> event for b254af24c9127b8bb22cb6d24e523dad
> 2012-06-30 09:07:48,020 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=kw_r,,1340981822374.b254af24c9127b8bb22cb6d24e523dad. state=CLOSED,
> ts=1341036467998, server=localhost,34193,1341036300138
> 2012-06-30 09:07:48,020 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50890-0x1383bfe01b7 Creating (or updating) unassigned node for
> b254af24c9127b8bb22cb6d24e523dad with OFFLINE state
> 2012-06-30 09:07:48,028 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:34193-0x1383bfe01b70001 Successfully transitioned node
> 2e8a318837602c9c9961e9d690b7fd02 from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 2012-06-30 09:07:48,028 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> Opening region: {NAME =>
> 'kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02.', STARTKEY => '', ENDKEY
> => '', ENCODED => 2e8a318837602c9c9961e9d690b7fd02,}
> 2012-06-30 09:07:48,029 ERROR
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
> of region=kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02., starting to
> roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3490)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3628)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3487)
> ... 7 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:133)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:125)
> at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:411)
> ... 11 more
> 2012-06-30 09:07:48,031 INFO
> org.apache.hadoop.hbase.regionserver.
[jira] [Updated] (HBASE-6748) Endless recursive of deleteNode happened in SplitLogManager#DeleteAsyncCallback
[
https://issues.apache.org/jira/browse/HBASE-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-6748:
-
Priority: Major (was: Critical)
Fix Version/s: (was: 0.94.4)
0.94.5
No patch. Moving out.
> Endless recursive of deleteNode happened in
> SplitLogManager#DeleteAsyncCallback
> ---
>
> Key: HBASE-6748
> URL: https://issues.apache.org/jira/browse/HBASE-6748
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.94.1, 0.96.0
>Reporter: Jieshan Bean
> Fix For: 0.96.0, 0.94.5
>
>
> You can ealily understand the problem from the below logs:
> {code}
> [2012-09-01 11:41:02,062] [WARN ]
> [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
> [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978]
> create rc =SESSIONEXPIRED for
> /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
> remaining retries=3
> [2012-09-01 11:41:02,062] [WARN ]
> [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
> [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978]
> create rc =SESSIONEXPIRED for
> /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
> remaining retries=2
> [2012-09-01 11:41:02,063] [WARN ]
> [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
> [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978]
> create rc =SESSIONEXPIRED for
> /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
> remaining retries=1
> [2012-09-01 11:41:02,063] [WARN ]
> [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
> [org.apache.hadoop.hbase.master.SplitLogManager$CreateAsyncCallback 978]
> create rc =SESSIONEXPIRED for
> /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
> remaining retries=0
> [2012-09-01 11:41:02,063] [WARN ]
> [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
> [org.apache.hadoop.hbase.master.SplitLogManager 393] failed to create task
> node/hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
> [2012-09-01 11:41:02,063] [WARN ]
> [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
> [org.apache.hadoop.hbase.master.SplitLogManager 353] Error splitting
> /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
> [2012-09-01 11:41:02,063] [WARN ]
> [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
> [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052]
> delete rc=SESSIONEXPIRED for
> /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
> remaining retries=9223372036854775807
> [2012-09-01 11:41:02,064] [WARN ]
> [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
> [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052]
> delete rc=SESSIONEXPIRED for
> /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
> remaining retries=9223372036854775806
> [2012-09-01 11:41:02,064] [WARN ]
> [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
> [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052]
> delete rc=SESSIONEXPIRED for
> /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
> remaining retries=9223372036854775805
> [2012-09-01 11:41:02,064] [WARN ]
> [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
> [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052]
> delete rc=SESSIONEXPIRED for
> /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
> remaining retries=9223372036854775804
> [2012-09-01 11:41:02,065] [WARN ]
> [MASTER_SERVER_OPERATIONS-xh03,2,1339549619270-1]
> [org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback 1052]
> delete rc=SESSIONEXPIRED for
> /hbase/splitlog/hdfs%3A%2F%2Fxh01%3A9000%2Fhbase%2F.logs%2Fxh01%2C20020%2C1339552105088-splitting%2Fxh01%252C20020%252C1339552105088.1339557014846
> remaining retries=9223372036854775803
> ...
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535646#comment-13535646
]
Lars Hofhansl commented on HBASE-7259:
--
Can we wrap that logic into resetZookeeperTrackers?
Otherwise I'm fine to commit this.
> Deadlock in HBaseClient when KeeperException occured
>
>
> Key: HBASE-7259
> URL: https://issues.apache.org/jira/browse/HBASE-7259
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Affects Versions: 0.94.0, 0.94.1, 0.94.2
>Reporter: liwei
>Priority: Critical
> Fix For: 0.94.4
>
> Attachments: 7259-0.94-branch.txt, HBASE-7259-0.94.2.txt
>
>
> HBaseClient was running after a period of time, all of get operation became
> too slow.
> From the client logs I could see the following:
> 1. Unable to get data of znode /hbase/root-region-server
> {code}
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
> at
> org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> at org.apache.hadoop.hbase.client.HTable.(HTable.java:174)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
> at
> org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
> at
> org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
> at
> org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
> at
> org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
> at
> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
> at
> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
> {code}
> 2. Catalina.out found one Java-level deadlock:
> {code}
> =
> "catalina-exec-800":
> waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a
> java.lang.Object),
> which is held by "catalina-exec-710"
> "catalina-exec-710":
> waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
> which is held by "catalina-exec-29-EventThread"
> "catalina-exec-29-EventThread":
> waiting to lock monitor 0x5f9f0af0 (object 0x
[jira] [Commented] (HBASE-7158) Allow CopyTable to identify the source cluster (for replication scenarios)
[
https://issues.apache.org/jira/browse/HBASE-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535644#comment-13535644
]
Lars Hofhansl commented on HBASE-7158:
--
Let's commit this?
> Allow CopyTable to identify the source cluster (for replication scenarios)
> --
>
> Key: HBASE-7158
> URL: https://issues.apache.org/jira/browse/HBASE-7158
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
> Fix For: 0.96.0, 0.94.4
>
> Attachments: HBASE-7158-0.94-v1.patch
>
>
> When I worked on HBASE-2195 I added a mechanism for an edit to identify its
> source cluster, so that replication would not bounce it back to the source.
> See: {{this.clusterId =
> zkHelper.getUUIDForCluster(zkHelper.getZookeeperWatcher());}} in
> ReplicationSource, and {{put.setClusterId(entry.getKey().getClusterId());}}
> in ReplicationSink.
> In master-master replication scenarios, it would very useful if CopyTable
> would identify the source cluster (by tagging each Put/Delete with the source
> clusterId before applying it).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7385) Do not abort regionserver if StoreFlusher.flushCache() fails
[
https://issues.apache.org/jira/browse/HBASE-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535643#comment-13535643
]
Lars Hofhansl commented on HBASE-7385:
--
How will Memstore.snapshot() become idempotent?
> Do not abort regionserver if StoreFlusher.flushCache() fails
>
>
> Key: HBASE-7385
> URL: https://issues.apache.org/jira/browse/HBASE-7385
> Project: HBase
> Issue Type: Improvement
> Components: io, regionserver
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> A rare NN failover may cause RS abort, in the following sequence of events:
> - RS tries to flush the memstore
> - Create a file, start block, and acquire a lease
> - Block is complete, lease removed, but before we send the RPC response back
> to the client, NN is killed.
> - New NN comes up, client retries the block complete again, the new NN
> throws lease expired since the block was already complete.
> - RS receives the exception, and aborts.
> This is actually a NN+DFSClient issue that, the dfs client from RS does not
> receive the rpc response about the block close, and upon retry on the new NN,
> it gets the exception, since the file was already closed. However, although
> this is DFS client specific, we can also make RS more resilient by not
> aborting the RS upon exception from the flushCache(). We can change
> StoreFlusher so that:
> StoreFlusher.prepare() will become idempotent (so will Memstore.snapshot())
> StoreFlusher.flushCache() will throw with IOException upon DFS exception, but
> we catch IOException, and just abort the flush request (not RS).
> StoreFlusher.commit() still cause RS abort on exception. This is also
> debatable. If dfs is alive, and we can undo the flush changes, than we should
> not abort.
> logs:
> {code}
> org.apache.hadoop.hbase.DroppedSnapshotException: region:
> loadtest_ha,e658,1355820729877.298bcbd550b80507a379fe67eefbe5ea.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1485)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1364)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:896)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:845)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /apps/hbase/data/loadtest_ha/298bcbd550b80507a379fe67eefbe5ea/.tmp/5cf8951ee12449ce8e4e6dd0bf1645c2
> File is not open for writing. [Lease. Holder:
> DFSClient_hb_rs_XXX,60020,1355813552066_203591774_25, pendingcreates: 1]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1724)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1707)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1762)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1750)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:779)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at $Proxy10.complete(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.r
[jira] [Updated] (HBASE-5778) Turn on WAL compression by default
[ https://issues.apache.org/jira/browse/HBASE-5778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Daniel Cryans updated HBASE-5778: -- Attachment: HBASE-5778-0.94-v7.patch HBASE-5778-trunk-v7.patch New patches for 0.94 and trunk that pass TestHLogSplit and I also added TestHLogSplitCompressed for 0.94. I also incorporated Ted's comments. bq. Sounds like WAL compression could do w/ some more testing especially if it becomes default in 0.96. Agreed although the same can be said of many other features in trunk. May I suggest we commit this then open new jiras if other issues are found during further testing? > Turn on WAL compression by default > -- > > Key: HBASE-5778 > URL: https://issues.apache.org/jira/browse/HBASE-5778 > Project: HBase > Issue Type: Improvement >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 5778.addendum, 5778-addendum.txt, HBASE-5778-0.94.patch, > HBASE-5778-0.94-v2.patch, HBASE-5778-0.94-v3.patch, HBASE-5778-0.94-v4.patch, > HBASE-5778-0.94-v5.patch, HBASE-5778-0.94-v6.patch, HBASE-5778-0.94-v7.patch, > HBASE-5778.patch, HBASE-5778-trunk-v6.patch, HBASE-5778-trunk-v7.patch > > > I ran some tests to verify if WAL compression should be turned on by default. > For a use case where it's not very useful (values two order of magnitude > bigger than the keys), the insert time wasn't different and the CPU usage 15% > higher (150% CPU usage VS 130% when not compressing the WAL). > When values are smaller than the keys, I saw a 38% improvement for the insert > run time and CPU usage was 33% higher (600% CPU usage VS 450%). I'm not sure > WAL compression accounts for all the additional CPU usage, it might just be > that we're able to insert faster and we spend more time in the MemStore per > second (because our MemStores are bad when they contain tens of thousands of > values). > Those are two extremes, but it shows that for the price of some CPU we can > save a lot. My machines have 2 quads with HT, so I still had a lot of idle > CPUs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6155) [copytable] Unexpected behavior if --starttime is not specifed but --endtime is.
[ https://issues.apache.org/jira/browse/HBASE-6155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6155: -- Fix Version/s: 0.96.0 Status: Patch Available (was: Open) > [copytable] Unexpected behavior if --starttime is not specifed but --endtime > is. > > > Key: HBASE-6155 > URL: https://issues.apache.org/jira/browse/HBASE-6155 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.0, 0.92.1, 0.90.6, 0.96.0 >Reporter: Jonathan Hsieh >Assignee: Prasanna > Labels: noob > Fix For: 0.96.0 > > Attachments: 6155.diff, 6155-patch-1.diff > > > If one uses copytable and specifies only an endtime, I'd expect to include > all rows from unix epoch time upto the specified endtime. Instead, it copies > all the rows. > The workaround for copies with this kind of range is to specify --startime=1 > (Note not --starttime=0), which is also unintuitive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6155) [copytable] Unexpected behavior if --starttime is not specifed but --endtime is.
[ https://issues.apache.org/jira/browse/HBASE-6155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanna updated HBASE-6155: Attachment: 6155-patch-1.diff Thanks for your comments. Updated the Javadoc and marked the test as LargeTest. > [copytable] Unexpected behavior if --starttime is not specifed but --endtime > is. > > > Key: HBASE-6155 > URL: https://issues.apache.org/jira/browse/HBASE-6155 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.90.6, 0.92.1, 0.94.0, 0.96.0 >Reporter: Jonathan Hsieh >Assignee: Prasanna > Labels: noob > Attachments: 6155.diff, 6155-patch-1.diff > > > If one uses copytable and specifies only an endtime, I'd expect to include > all rows from unix epoch time upto the specified endtime. Instead, it copies > all the rows. > The workaround for copies with this kind of range is to specify --startime=1 > (Note not --starttime=0), which is also unintuitive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7385) Do not abort regionserver if StoreFlusher.flushCache() fails
[
https://issues.apache.org/jira/browse/HBASE-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535633#comment-13535633
]
ramkrishna.s.vasudevan commented on HBASE-7385:
---
Nice one. :)
> Do not abort regionserver if StoreFlusher.flushCache() fails
>
>
> Key: HBASE-7385
> URL: https://issues.apache.org/jira/browse/HBASE-7385
> Project: HBase
> Issue Type: Improvement
> Components: io, regionserver
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> A rare NN failover may cause RS abort, in the following sequence of events:
> - RS tries to flush the memstore
> - Create a file, start block, and acquire a lease
> - Block is complete, lease removed, but before we send the RPC response back
> to the client, NN is killed.
> - New NN comes up, client retries the block complete again, the new NN
> throws lease expired since the block was already complete.
> - RS receives the exception, and aborts.
> This is actually a NN+DFSClient issue that, the dfs client from RS does not
> receive the rpc response about the block close, and upon retry on the new NN,
> it gets the exception, since the file was already closed. However, although
> this is DFS client specific, we can also make RS more resilient by not
> aborting the RS upon exception from the flushCache(). We can change
> StoreFlusher so that:
> StoreFlusher.prepare() will become idempotent (so will Memstore.snapshot())
> StoreFlusher.flushCache() will throw with IOException upon DFS exception, but
> we catch IOException, and just abort the flush request (not RS).
> StoreFlusher.commit() still cause RS abort on exception. This is also
> debatable. If dfs is alive, and we can undo the flush changes, than we should
> not abort.
> logs:
> {code}
> org.apache.hadoop.hbase.DroppedSnapshotException: region:
> loadtest_ha,e658,1355820729877.298bcbd550b80507a379fe67eefbe5ea.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1485)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1364)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:896)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:845)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:119)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /apps/hbase/data/loadtest_ha/298bcbd550b80507a379fe67eefbe5ea/.tmp/5cf8951ee12449ce8e4e6dd0bf1645c2
> File is not open for writing. [Lease. Holder:
> DFSClient_hb_rs_XXX,60020,1355813552066_203591774_25, pendingcreates: 1]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1724)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1707)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1762)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1750)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:779)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at $Proxy10.complete(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.Delegating
[jira] [Commented] (HBASE-6788) Convert AuthenticationProtocol to protocol buffer service
[
https://issues.apache.org/jira/browse/HBASE-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535622#comment-13535622
]
Hadoop QA commented on HBASE-6788:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561623/HBASE-6788_5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 28 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 232
release audit warnings (more than the trunk's current 84 warnings).
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:red}-1 core zombie tests{color}. There are zombie tests. See build
logs for details.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3603//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3603//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3603//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3603//console
This message is automatically generated.
> Convert AuthenticationProtocol to protocol buffer service
> -
>
> Key: HBASE-6788
> URL: https://issues.apache.org/jira/browse/HBASE-6788
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 0.96.0
>
> Attachments: HBASE-6788_2.patch, HBASE-6788_3.patch,
> HBASE-6788_4.patch, HBASE-6788_5.patch, HBASE-6788.patch
>
>
> With coprocessor endpoints now exposed as protobuf defined services, we
> should convert over all of our built-in endpoints to PB services.
> AccessControllerProtocol was converted as part of HBASE-5448, but the
> authentication token provider still needs to be changed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7383) create integration test for HBASE-5416 (improving scan performance for certain filters)
[ https://issues.apache.org/jira/browse/HBASE-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535611#comment-13535611 ] Sergey Shelukhin commented on HBASE-7383: - DefaultDataGenerator - can be used for any tests/tools so should be public. > create integration test for HBASE-5416 (improving scan performance for > certain filters) > --- > > Key: HBASE-7383 > URL: https://issues.apache.org/jira/browse/HBASE-7383 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7383-v0.patch > > > HBASE-5416 is risky and needs an integration test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6788) Convert AuthenticationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-6788: - Status: Patch Available (was: Open) > Convert AuthenticationProtocol to protocol buffer service > - > > Key: HBASE-6788 > URL: https://issues.apache.org/jira/browse/HBASE-6788 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6788_2.patch, HBASE-6788_3.patch, > HBASE-6788_4.patch, HBASE-6788_5.patch, HBASE-6788.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. > AccessControllerProtocol was converted as part of HBASE-5448, but the > authentication token provider still needs to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7384) Introducing waitForCondition function into test cases
[
https://issues.apache.org/jira/browse/HBASE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HBASE-7384:
--
Attachment: Waiter.java
In Oozie we've been using this pattern quite successfully. In case you are
interested, the attached file is a cannibalized version of that code removing
all Oozie deps.
> Introducing waitForCondition function into test cases
> -
>
> Key: HBASE-7384
> URL: https://issues.apache.org/jira/browse/HBASE-7384
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Attachments: Waiter.java
>
>
> Recently I'm working on flaky test cases and found we have many places using
> while loop and sleep to wait for a condition to be true. There are several
> issues in existing ways:
> 1) Many similar code doing the same thing
> 2) When time out happens, different errors are reported without explicitly
> indicating a time out situation
> 3) When we want to increase the max timeout value to verify if a test case
> fails due to a not-enough time out value, we have to recompile & redeploy code
> I propose to create a waitForCondition function as a test utility function
> like the following:
> {code}
> public interface WaitCheck {
> public boolean Check() ;
> }
> public boolean waitForCondition(int timeOutInMilliSeconds, int
> checkIntervalInMilliSeconds, WaitCheck s)
> throws InterruptedException {
> int multiplier = 1;
> String multiplierProp = System.getProperty("extremeWaitMultiplier");
> if(multiplierProp != null) {
> multiplier = Integer.parseInt(multiplierProp);
> if(multiplier < 1) {
> LOG.warn(String.format("Invalid extremeWaitMultiplier
> property value:%s. is ignored.", multiplierProp));
> multiplier = 1;
> }
> }
> int timeElapsed = 0;
> while(timeElapsed < timeOutInMilliSeconds * multiplier) {
> if(s.Check()) {
> return true;
> }
> Thread.sleep(checkIntervalInMilliSeconds);
> timeElapsed += checkIntervalInMilliSeconds;
> }
> assertTrue("WaitForCondition failed due to time out(" +
> timeOutInMilliSeconds + " milliseconds expired)",
> false);
> return false;
> }
> {code}
> By doing the above way, there are several advantages:
> 1) Clearly report time out error when such situation happens
> 2) Use System property extremeWaitMultiplier to increase max time out
> dynamically for a quick verification
> 3) Standardize current wait situations
> Pleas let me know what your thoughts on this.
> Thanks,
> -Jeffrey
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7384) Introducing waitForCondition function into test cases
[
https://issues.apache.org/jira/browse/HBASE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535594#comment-13535594
]
Jeffrey Zhong commented on HBASE-7384:
--
Thanks Todd for pointing the GenericTestUtils#waitFor out. It almost does the
thing I proposed in the ticket. Surprisingly we don't use it. While the only
thing it misses is the capability to dynamically increase max time out value to
quickly verify a not-enough time out situation without recompile & deploy. If
you think the "dynamically increasing time out" capability is needed, I can
create a HBase waitFor version supporting the capability and internally calling
the hadoop common GenericTestUtils#waitFor.
Thanks,
-Jeffrey
> Introducing waitForCondition function into test cases
> -
>
> Key: HBASE-7384
> URL: https://issues.apache.org/jira/browse/HBASE-7384
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>
> Recently I'm working on flaky test cases and found we have many places using
> while loop and sleep to wait for a condition to be true. There are several
> issues in existing ways:
> 1) Many similar code doing the same thing
> 2) When time out happens, different errors are reported without explicitly
> indicating a time out situation
> 3) When we want to increase the max timeout value to verify if a test case
> fails due to a not-enough time out value, we have to recompile & redeploy code
> I propose to create a waitForCondition function as a test utility function
> like the following:
> {code}
> public interface WaitCheck {
> public boolean Check() ;
> }
> public boolean waitForCondition(int timeOutInMilliSeconds, int
> checkIntervalInMilliSeconds, WaitCheck s)
> throws InterruptedException {
> int multiplier = 1;
> String multiplierProp = System.getProperty("extremeWaitMultiplier");
> if(multiplierProp != null) {
> multiplier = Integer.parseInt(multiplierProp);
> if(multiplier < 1) {
> LOG.warn(String.format("Invalid extremeWaitMultiplier
> property value:%s. is ignored.", multiplierProp));
> multiplier = 1;
> }
> }
> int timeElapsed = 0;
> while(timeElapsed < timeOutInMilliSeconds * multiplier) {
> if(s.Check()) {
> return true;
> }
> Thread.sleep(checkIntervalInMilliSeconds);
> timeElapsed += checkIntervalInMilliSeconds;
> }
> assertTrue("WaitForCondition failed due to time out(" +
> timeOutInMilliSeconds + " milliseconds expired)",
> false);
> return false;
> }
> {code}
> By doing the above way, there are several advantages:
> 1) Clearly report time out error when such situation happens
> 2) Use System property extremeWaitMultiplier to increase max time out
> dynamically for a quick verification
> 3) Standardize current wait situations
> Pleas let me know what your thoughts on this.
> Thanks,
> -Jeffrey
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535590#comment-13535590 ] Gregory Chanan commented on HBASE-7386: --- Open Questions: 1) Do we think this is a useful direction? 2) If so, what level of supervisor scripts do we provide? None? Samples? Actual scripts that can be run [i.e. mimicking HBASE-daemon.sh -- probably want to move all the config setup to a common file]? Actual scripts + ssh scripting to actually run with a "start-hbase.sh", i.e. "start-supervised-hbase.sh" > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.96.0 > > Attachments: HBASE-7386-v0.patch, supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[
https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535587#comment-13535587
]
Gregory Chanan commented on HBASE-7386:
---
Here is how I verified this all works (apply both patches and set HBASE_HOME,
HBASE_LOG_DIR in your environment but nothing else). Also make sure
supervisord is installed.
{noformat}
1. $HBASE_HOME/bin/hbase-daemons.sh --config "" start zookeeper
2. cd $HBASE_HOME/conf/supervisord
3. supervisord -c supervisord.conf (call "jps" and record the pid of the
HMaster)
4. $HBASE_HOME/bin/hbase-daemons.sh --config "" --hosts localhost.txt start
regionserver (localhost.txt is a text file with the line "localhost")
5. $HBASE_HOME/bin/./local-master-backup.sh start 1
6. kill -9 [PID_OF_HMASTER_RECORDED_AT_STEP_3]
{noformat}
and notice the backup master takes over right away. Also use start-hbase.sh
and notice it has the same behavior as before (does not swallow output).
> Investigate providing some supervisor support for znode deletion
>
>
> Key: HBASE-7386
> URL: https://issues.apache.org/jira/browse/HBASE-7386
> Project: HBase
> Issue Type: Task
> Components: master, regionserver, scripts
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Fix For: 0.96.0
>
> Attachments: HBASE-7386-v0.patch, supervisordconfigs-v0.patch
>
>
> There a couple of JIRAs for deleting the znode on a process failure:
> HBASE-5844 (RS)
> HBASE-5926 (Master)
> which are pretty neat; on process failure, they delete the znode of the
> underlying process so HBase can recover faster.
> These JIRAs were implemented via the startup scripts; i.e. the script hangs
> around and waits for the process to exit, then deletes the znode.
> There are a few problems associated with this approach, as listed in the
> below JIRAs:
> 1) Hides startup output in script
> https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401
> 2) two hbase processes listed per launched daemon
> https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409
> 3) Not run by a real supervisor
> https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409
> 4) Weird output after kill -9 actual process in standalone mode
> https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801
> 5) Can kill existing RS if called again
> https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401
> 6) Hides stdout/stderr[6]
> https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832
> I suspect running in via something like supervisor.d can solve these issues
> if we provide the right support.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-7386: -- Attachment: supervisordconfigs-v0.patch Here are some sample supervisord scripts for restarting the master. They are not fully fledged yet; a better attempt would go through all the config files in hbase-daemon and do something appropriate. This requires that HBASE_HOME and HBASE_LOG_DIR are set and does not respect any other environment settings (e.g. HBASE_CONF_DIR). > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.96.0 > > Attachments: HBASE-7386-v0.patch, supervisordconfigs-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-7386: -- Attachment: HBASE-7386-v0.patch * Attached HBASE-7386-v0* Does the following: -Rips out the script changes made by HBASE-5844 and HBASE-5926 - Adds a new command to the master "supervise" which does clean + start. It should probably be called "autorestart" but that is taken by the script. Todos: - Need to include supervisor configs. I'll do that separately as I'm not sure whether we want them to be samples or full-fledged configs people can use. - I didn't pull out "autorestart" from the scripts, but it's probably broken by this. > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.96.0 > > Attachments: HBASE-7386-v0.patch > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7283) Backport HBASE-6564 + HBASE-7202 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535576#comment-13535576 ] Lars Hofhansl commented on HBASE-7283: -- Yes please, both are bad bugs. > Backport HBASE-6564 + HBASE-7202 to 0.94 > > > Key: HBASE-7283 > URL: https://issues.apache.org/jira/browse/HBASE-7283 > Project: HBase > Issue Type: Task >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7283.094.txt, HBASE-6564-0.94.patch, > HBASE-7202-0.94.patch > > > * HBASE-6564: HDFS space is not reclaimed when a column family is deleted > * HBASE-7202: Family Store Files are not archived on admin.deleteColumn() > (the second one is a fix for the first, to use the archiver) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7357) HBaseClient and HBaseServer should use hbase.security.authentication when negotiating authentication
[ https://issues.apache.org/jira/browse/HBASE-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535574#comment-13535574 ] Lars Hofhansl commented on HBASE-7357: -- +1 ... Does this need a documentation change somewhere aswell? Can do as separate jira if needed. > HBaseClient and HBaseServer should use hbase.security.authentication when > negotiating authentication > > > Key: HBASE-7357 > URL: https://issues.apache.org/jira/browse/HBASE-7357 > Project: HBase > Issue Type: Bug > Components: security >Reporter: Gary Helmling >Assignee: Gary Helmling > Attachments: HBASE-7357_0.94_2.patch, HBASE-7357_0.94.patch, > HBASE-7357.patch > > > This came up in the context of testing HBASE-6788. Currently HBaseClient and > HBaseServer call UserGroupInformation.isSecurityEnabled() when determining > whether or not to use SASL to negotiate connections. This means they are > using the hadoop.security.authentication configuration value. Since this is > in the context of HBase RPC connections, it seems more correct to use the > hbase.security.authentication configuration value by calling > User.isHBaseSecurityEnabled(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7374) Expose master table operations for coprocessors by way of MasterServices
[ https://issues.apache.org/jira/browse/HBASE-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535563#comment-13535563 ] Lars Hofhansl commented on HBASE-7374: -- +1 on 0.94. > Expose master table operations for coprocessors by way of MasterServices > > > Key: HBASE-7374 > URL: https://issues.apache.org/jira/browse/HBASE-7374 > Project: HBase > Issue Type: Improvement > Components: Coprocessors, master >Affects Versions: 0.96.0, 0.94.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Attachments: HBASE-7374-0.94.patch, HBASE-7374.patch > > > I have something I'm working on that, as a coprocessor, when it initializes > would like to add a column to a table should that column be missing. Exposing > master table operations for coprocessors by way of MasterServices was how I > solved this problem, and is generally useful for all master coprocessors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535562#comment-13535562 ] Gregory Chanan commented on HBASE-7386: --- Thanks for pointing that out, Enis. What I'm doing here is compatible with that, I think. Rather than deleting the znode we just expire the session. Should be a small change. > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.96.0 > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7351) Periodic health check chore
[ https://issues.apache.org/jira/browse/HBASE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vandana Ayyalasomayajula updated HBASE-7351: Attachment: HBASE-7331_94_1.patch New version with review comments addressed. > Periodic health check chore > --- > > Key: HBASE-7351 > URL: https://issues.apache.org/jira/browse/HBASE-7351 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Attachments: HBASE-7331_94_1.patch, HBASE-7331_94.patch, > HBASE-7331_trunk.patch > > > Similar to MAPREDUCE-211, region server should also have a mechanism to check > the health of the node. It should run the health check script periodically > and if there is any errors, it should stop itself gracefully. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6788) Convert AuthenticationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-6788: - Attachment: HBASE-6788_5.patch Updated patch rebased against current trunk. This pulls in the HBASE-7357 changes and makes TestTokenAuthentication work around the ZK SASL setup from HBASE-4791. > Convert AuthenticationProtocol to protocol buffer service > - > > Key: HBASE-6788 > URL: https://issues.apache.org/jira/browse/HBASE-6788 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6788_2.patch, HBASE-6788_3.patch, > HBASE-6788_4.patch, HBASE-6788_5.patch, HBASE-6788.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. > AccessControllerProtocol was converted as part of HBASE-5448, but the > authentication token provider still needs to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6788) Convert AuthenticationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-6788: - Status: Open (was: Patch Available) > Convert AuthenticationProtocol to protocol buffer service > - > > Key: HBASE-6788 > URL: https://issues.apache.org/jira/browse/HBASE-6788 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6788_2.patch, HBASE-6788_3.patch, > HBASE-6788_4.patch, HBASE-6788_5.patch, HBASE-6788.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. > AccessControllerProtocol was converted as part of HBASE-5448, but the > authentication token provider still needs to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7384) Introducing waitForCondition function into test cases
[
https://issues.apache.org/jira/browse/HBASE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535552#comment-13535552
]
Todd Lipcon commented on HBASE-7384:
Hadoop Common has GenericTestUtils#waitFor which basically does this (though
not with a backoff).
> Introducing waitForCondition function into test cases
> -
>
> Key: HBASE-7384
> URL: https://issues.apache.org/jira/browse/HBASE-7384
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>
> Recently I'm working on flaky test cases and found we have many places using
> while loop and sleep to wait for a condition to be true. There are several
> issues in existing ways:
> 1) Many similar code doing the same thing
> 2) When time out happens, different errors are reported without explicitly
> indicating a time out situation
> 3) When we want to increase the max timeout value to verify if a test case
> fails due to a not-enough time out value, we have to recompile & redeploy code
> I propose to create a waitForCondition function as a test utility function
> like the following:
> {code}
> public interface WaitCheck {
> public boolean Check() ;
> }
> public boolean waitForCondition(int timeOutInMilliSeconds, int
> checkIntervalInMilliSeconds, WaitCheck s)
> throws InterruptedException {
> int multiplier = 1;
> String multiplierProp = System.getProperty("extremeWaitMultiplier");
> if(multiplierProp != null) {
> multiplier = Integer.parseInt(multiplierProp);
> if(multiplier < 1) {
> LOG.warn(String.format("Invalid extremeWaitMultiplier
> property value:%s. is ignored.", multiplierProp));
> multiplier = 1;
> }
> }
> int timeElapsed = 0;
> while(timeElapsed < timeOutInMilliSeconds * multiplier) {
> if(s.Check()) {
> return true;
> }
> Thread.sleep(checkIntervalInMilliSeconds);
> timeElapsed += checkIntervalInMilliSeconds;
> }
> assertTrue("WaitForCondition failed due to time out(" +
> timeOutInMilliSeconds + " milliseconds expired)",
> false);
> return false;
> }
> {code}
> By doing the above way, there are several advantages:
> 1) Clearly report time out error when such situation happens
> 2) Use System property extremeWaitMultiplier to increase max time out
> dynamically for a quick verification
> 3) Standardize current wait situations
> Pleas let me know what your thoughts on this.
> Thanks,
> -Jeffrey
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7351) Periodic health check chore
[ https://issues.apache.org/jira/browse/HBASE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535549#comment-13535549 ] Vandana Ayyalasomayajula commented on HBASE-7351: - [~apurtell] I agree that the health checker can be common to both master and region server. I tried to do something similar to Hadoop, the datanode kills itself if its unhealthy. In our use case we wanted to the region server also to shutdown gracefully if something was wrong with the node. In case of the master, it does not do much of heavy lifting like the region server and also has a back up master in case it dies. So, I had the health checked chore only for the region server. I can move the classes HealthChecker, HealthReport, RegionServerHealthCheckChore and TestServerHealthChecker one level up and make them more generic. > Periodic health check chore > --- > > Key: HBASE-7351 > URL: https://issues.apache.org/jira/browse/HBASE-7351 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Attachments: HBASE-7331_94.patch, HBASE-7331_trunk.patch > > > Similar to MAPREDUCE-211, region server should also have a mechanism to check > the health of the node. It should run the health check script periodically > and if there is any errors, it should stop itself gracefully. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[
https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535543#comment-13535543
]
Hadoop QA commented on HBASE-5416:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561614/HBASE-5416-v10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 28 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 232
release audit warnings (more than the trunk's current 84 warnings).
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
org.apache.hadoop.hbase.TestDrainingServer
{color:red}-1 core zombie tests{color}. There are zombie tests. See build
logs for details.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3601//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3601//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3601//console
This message is automatically generated.
> Improve performance of scans with some kind of filters.
> ---
>
> Key: HBASE-5416
> URL: https://issues.apache.org/jira/browse/HBASE-5416
> Project: HBase
> Issue Type: Improvement
> Components: Filters, Performance, regionserver
>Affects Versions: 0.90.4
>Reporter: Max Lapan
>Assignee: Max Lapan
> Fix For: 0.96.0
>
> Attachments: 5416-Filtered_scans_v6.patch, 5416-v5.txt, 5416-v6.txt,
> Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch,
> Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch,
> Filtered_scans_v7.patch, HBASE-5416-v10.patch, HBASE-5416-v7-rebased.patch,
> HBASE-5416-v8.patch, HBASE-5416-v9.patch
>
>
> When the scan is performed, whole row is loaded into result list, after that
> filter (if exists) is applied to detect that row is needed.
> But when scan is performed on several CFs and filter checks only data from
> the subset of these CFs, data from CFs, not checked by a filter is not needed
> on a filter stage. Only when we decided to include current row. And in such
> case we can significantly reduce amount of IO performed by a scan, by loading
> only values, actually checked by a filter.
> For example, we have two CFs: flags and snap. Flags is quite small (bunch of
> megabytes) and is used to filter large entries from snap. Snap is very large
> (10s of GB) and it is quite costly to scan it. If we needed only rows with
> some flag specified, we use SingleColumnValueFilter to limit result to only
> small subset of region. But current implementation is loading both CFs to
> perform scan, when only small subset is needed.
> Attached patch adds one routine to Filter interface to allow filter to
> specify which CF is needed to it's operation. In HRegion, we separate all
> scanners into two groups: needed for filter and the rest (joined). When new
> row is considered, only needed data is loa
[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding
[
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535541#comment-13535541
]
Ted Yu commented on HBASE-4676:
---
{code}
+++
b/hbase-prefix-tree/src/test/java/org/apache/hbase/codec/prefixtree/timestamp/data/TestTimestamps1.java
+++
b/hbase-prefix-tree/src/test/java/org/apache/hbase/codec/prefixtree/timestamp/data/TestTimestamps2.java
+++
b/hbase-prefix-tree/src/test/java/org/apache/hbase/codec/prefixtree/timestamp/data/TestTimestamps3.java
+++
b/hbase-prefix-tree/src/test/java/org/apache/hbase/codec/prefixtree/builder/data/TestSortedByteArrays1.java
{code}
Can the class names be more informative about what they test ?
For TimestampCompressorTests.java, KeyValueToolTests.java,
PrefixTreeBlockMetaTests.java, etc, normally their class names should start
with Test.
> Prefix Compression - Trie data block encoding
> -
>
> Key: HBASE-4676
> URL: https://issues.apache.org/jira/browse/HBASE-4676
> Project: HBase
> Issue Type: New Feature
> Components: io, Performance, regionserver
>Affects Versions: 0.96.0
>Reporter: Matt Corgan
>Assignee: Matt Corgan
> Attachments: HBASE-4676-0.94-v1.patch,
> HBASE-4676-common-and-server-v8.patch, HBASE-4676-prefix-tree-trunk-v1.patch,
> HBASE-4676-prefix-tree-trunk-v2.patch, HBASE-4676-prefix-tree-trunk-v3.patch,
> HBASE-4676-prefix-tree-trunk-v4.patch, HBASE-4676-prefix-tree-trunk-v5.patch,
> HBASE-4676-prefix-tree-trunk-v6.patch, HBASE-4676-prefix-tree-trunk-v7.patch,
> HBASE-4676-prefix-tree-trunk-v8.patch, hbase-prefix-trie-0.1.jar,
> PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by
> blockSize.png
>
>
> The HBase data block format has room for 2 significant improvements for
> applications that have high block cache hit ratios.
> First, there is no prefix compression, and the current KeyValue format is
> somewhat metadata heavy, so there can be tremendous memory bloat for many
> common data layouts, specifically those with long keys and short values.
> Second, there is no random access to KeyValues inside data blocks. This
> means that every time you double the datablock size, average seek time (or
> average cpu consumption) goes up by a factor of 2. The standard 64KB block
> size is ~10x slower for random seeks than a 4KB block size, but block sizes
> as small as 4KB cause problems elsewhere. Using block sizes of 256KB or 1MB
> or more may be more efficient from a disk access and block-cache perspective
> in many big-data applications, but doing so is infeasible from a random seek
> perspective.
> The PrefixTrie block encoding format attempts to solve both of these
> problems. Some features:
> * trie format for row key encoding completely eliminates duplicate row keys
> and encodes similar row keys into a standard trie structure which also saves
> a lot of space
> * the column family is currently stored once at the beginning of each block.
> this could easily be modified to allow multiple family names per block
> * all qualifiers in the block are stored in their own trie format which
> caters nicely to wide rows. duplicate qualifers between rows are eliminated.
> the size of this trie determines the width of the block's qualifier
> fixed-width-int
> * the minimum timestamp is stored at the beginning of the block, and deltas
> are calculated from that. the maximum delta determines the width of the
> block's timestamp fixed-width-int
> The block is structured with metadata at the beginning, then a section for
> the row trie, then the column trie, then the timestamp deltas, and then then
> all the values. Most work is done in the row trie, where every leaf node
> (corresponding to a row) contains a list of offsets/references corresponding
> to the cells in that row. Each cell is fixed-width to enable binary
> searching and is represented by [1 byte operationType, X bytes qualifier
> offset, X bytes timestamp delta offset].
> If all operation types are the same for a block, there will be zero per-cell
> overhead. Same for timestamps. Same for qualifiers when i get a chance.
> So, the compression aspect is very strong, but makes a few small sacrifices
> on VarInt size to enable faster binary searches in trie fan-out nodes.
> A more compressed but slower version might build on this by also applying
> further (suffix, etc) compression on the trie nodes at the cost of slower
> write speed. Even further compression could be obtained by using all VInts
> instead of FInts with a sacrifice on random seek speed (though not huge).
> One current drawback is the current write speed. While programmed with good
> constructs like TreeMaps, ByteBuffers, binary searches, etc, it's not
> programmed with the same level of optimizat
[jira] [Updated] (HBASE-7372) Check in the generated website so can point apache infrastructure at what to publish as our hbase.apache.org
[
https://issues.apache.org/jira/browse/HBASE-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-7372:
-
Attachment: 7372v2.txt
This patch moves src/site to content and adds some properties so we might not
have to check in our website (I was reading here
http://www.apache.org/dev/cmsadoption#maven)
> Check in the generated website so can point apache infrastructure at what to
> publish as our hbase.apache.org
>
>
> Key: HBASE-7372
> URL: https://issues.apache.org/jira/browse/HBASE-7372
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Attachments: 7372v2.txt
>
>
> January 1st is deadline for changing how we publish our website. We may no
> longer rsync out to people.apache.org. Apache infrastructure supplies two
> options here: http://www.apache.org/dev/project-site.html We could redo our
> site in apache cms format. Or we could just use svnpubsub and keep on w/ how
> the site is currently generated and on checkin, have it autopublished. I'll
> go the latter route unless I hear otherwise.
> For svnpubsub, we need to point apache infrastructure at a directory that has
> our checkedin site in it. I was thinking ${hbasedir}/hbase.apache.org
> Let me raise this on the dev list too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-7372) Check in the generated website so can point apache infrastructure at what to publish as our hbase.apache.org
[
https://issues.apache.org/jira/browse/HBASE-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reopened HBASE-7372:
--
Reopening until we get over apache infrastructure. Will do all the work under
this issue.
> Check in the generated website so can point apache infrastructure at what to
> publish as our hbase.apache.org
>
>
> Key: HBASE-7372
> URL: https://issues.apache.org/jira/browse/HBASE-7372
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>
> January 1st is deadline for changing how we publish our website. We may no
> longer rsync out to people.apache.org. Apache infrastructure supplies two
> options here: http://www.apache.org/dev/project-site.html We could redo our
> site in apache cms format. Or we could just use svnpubsub and keep on w/ how
> the site is currently generated and on checkin, have it autopublished. I'll
> go the latter route unless I hear otherwise.
> For svnpubsub, we need to point apache infrastructure at a directory that has
> our checkedin site in it. I was thinking ${hbasedir}/hbase.apache.org
> Let me raise this on the dev list too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7379) Port '[89-fb] prevent OOM possibility due to per connection responseQueue being unbounded' to trunk
[ https://issues.apache.org/jira/browse/HBASE-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7379: -- Issue Type: Sub-task (was: Bug) Parent: HBASE-2506 > Port '[89-fb] prevent OOM possibility due to per connection responseQueue > being unbounded' to trunk > --- > > Key: HBASE-7379 > URL: https://issues.apache.org/jira/browse/HBASE-7379 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 7379-trunk.txt, 7379-trunk-v2.txt > > > HBASE-6728 ported '[89-fb] prevent OOM possibility due to per connection > responseQueue being unbounded' to 0.94 branch > This issue is to port it to trunk -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7383) create integration test for HBASE-5416 (improving scan performance for certain filters)
[
https://issues.apache.org/jira/browse/HBASE-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535521#comment-13535521
]
Ted Yu commented on HBASE-7383:
---
Looked at the patch briefly.
{code}
+ * {@link
HBaseTestingUtility#createPreSplitLoadTestTable(org.apache.hadoop.hbase.Configuration,
byte[], byte[], Algorithm, DataBlockEncoding)}
{code}
Wrap long line.
{code}
+ public static class DefaultDataGenerator extends LoadTestDataGenerator {
{code}
The above class can be package-private, right ?
> create integration test for HBASE-5416 (improving scan performance for
> certain filters)
> ---
>
> Key: HBASE-7383
> URL: https://issues.apache.org/jira/browse/HBASE-7383
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7383-v0.patch
>
>
> HBASE-5416 is risky and needs an integration test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[ https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535520#comment-13535520 ] Sergey Shelukhin commented on HBASE-7268: - No, in case of a server it's server timestamp (e.g. LATEST_TIMESTAMP), as there's nothing from master. Given that split happens locally I wonder of original region TS can be used, this will be later than any other change from master for that region but earlier than any change for daughters. Overall though I wonder what you think about this patch (the way it works is supplying master timestamp thru open and close region requests to the server and saving them into Meta after that)... it seems like sort of an overkill for me and I don't feel good about mucking with meta timestamps. > correct local region location cache information can be overwritten w/stale > information from an old server > - > > Key: HBASE-7268 > URL: https://issues.apache.org/jira/browse/HBASE-7268 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7268-v0.patch, HBASE-7268-v0.patch, > HBASE-7268-v1.patch, HBASE-7268-v2.patch, HBASE-7268-v2-plus-masterTs.patch, > HBASE-7268-v2-plus-masterTs.patch > > > Discovered via HBASE-7250; related to HBASE-5877. > Test is writing from multiple threads. > Server A has region R; client knows that. > R gets moved from A to server B. > B gets killed. > R gets moved by master to server C. > ~15 seconds later, client tries to write to it (on A?). > Multiple client threads report from RegionMoved exception processing logic "R > moved from C to B", even though such transition never happened (neither in > nor before the sequence described below). Not quite sure how the client > learned of the transition to C, I assume it's from meta from some other > thread... > Then, put fails (it may fail due to accumulated errors that are not logged, > which I am investigating... but the bogus cache update is there > nonwithstanding). > I have a patch but not sure if it works, test still fails locally for yet > unknown reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7386) Investigate providing some supervisor support for znode deletion
[ https://issues.apache.org/jira/browse/HBASE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535517#comment-13535517 ] Gregory Chanan commented on HBASE-7386: --- I'm going to first tackle the Master case, HBASE-5926. The proposal is to rip out the script modification of HBASE-5926 and replace with supervisor.d support. The java code from HBASE-5926 would of course stay, only the restart/cleanup code in the scripts would go. The goal is to solve the above listed issues: #2 is solved by running via supervisor (jps reports only one HMaster process when HMaster started via supervisor.d) #3 is self-evidently solved by running on a real supervisor #5 is clearly not relevant, as it is only related to the RS. So, we are left with #1, #4, and #6. #1 and #6 are solved because the previous script behavior returns when not run in supervisor and supervisor can redirect stdout/stderr when run. #4 is solved because the old script behavior returns when not run in supervisor and I don't think it makes any sense to run a standalone HBase via supervisor. Example patch coming soon. > Investigate providing some supervisor support for znode deletion > > > Key: HBASE-7386 > URL: https://issues.apache.org/jira/browse/HBASE-7386 > Project: HBase > Issue Type: Task > Components: master, regionserver, scripts >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.96.0 > > > There a couple of JIRAs for deleting the znode on a process failure: > HBASE-5844 (RS) > HBASE-5926 (Master) > which are pretty neat; on process failure, they delete the znode of the > underlying process so HBase can recover faster. > These JIRAs were implemented via the startup scripts; i.e. the script hangs > around and waits for the process to exit, then deletes the znode. > There are a few problems associated with this approach, as listed in the > below JIRAs: > 1) Hides startup output in script > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 2) two hbase processes listed per launched daemon > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 3) Not run by a real supervisor > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 > 4) Weird output after kill -9 actual process in standalone mode > https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 > 5) Can kill existing RS if called again > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 > 6) Hides stdout/stderr[6] > https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 > I suspect running in via something like supervisor.d can solve these issues > if we provide the right support. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7386) Investigate providing some supervisor support for znode deletion
Gregory Chanan created HBASE-7386: - Summary: Investigate providing some supervisor support for znode deletion Key: HBASE-7386 URL: https://issues.apache.org/jira/browse/HBASE-7386 Project: HBase Issue Type: Task Components: master, regionserver, scripts Reporter: Gregory Chanan Assignee: Gregory Chanan Fix For: 0.96.0 There a couple of JIRAs for deleting the znode on a process failure: HBASE-5844 (RS) HBASE-5926 (Master) which are pretty neat; on process failure, they delete the znode of the underlying process so HBase can recover faster. These JIRAs were implemented via the startup scripts; i.e. the script hangs around and waits for the process to exit, then deletes the znode. There are a few problems associated with this approach, as listed in the below JIRAs: 1) Hides startup output in script https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 2) two hbase processes listed per launched daemon https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 3) Not run by a real supervisor https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463409 4) Weird output after kill -9 actual process in standalone mode https://issues.apache.org/jira/browse/HBASE-5926?focusedCommentId=13506801&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506801 5) Can kill existing RS if called again https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13463401&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13463401 6) Hides stdout/stderr[6] https://issues.apache.org/jira/browse/HBASE-5844?focusedCommentId=13506832&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13506832 I suspect running in via something like supervisor.d can solve these issues if we provide the right support. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7318) Add verbose logging option to HConnectionManager
[ https://issues.apache.org/jira/browse/HBASE-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7318: Attachment: HBASE-7318-v1.patch Makes sense, updating the patch. > Add verbose logging option to HConnectionManager > > > Key: HBASE-7318 > URL: https://issues.apache.org/jira/browse/HBASE-7318 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7318-v0.patch, HBASE-7318-v1.patch > > > In the course of HBASE-7250 I found that client-side errors (as well as > server-side errors, but that's another question) are hard to debug. > I have some local commits with useful, not-that-hacky HConnectionManager > logging added. > Need to "productionize" it to be off by default but easy-to-enable for > debugging. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7372) Check in the generated website so can point apache infrastructure at what to publish as our hbase.apache.org
[
https://issues.apache.org/jira/browse/HBASE-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535511#comment-13535511
]
stack commented on HBASE-7372:
--
I filed INFRA-5680 about getting our site publishing migrated.
> Check in the generated website so can point apache infrastructure at what to
> publish as our hbase.apache.org
>
>
> Key: HBASE-7372
> URL: https://issues.apache.org/jira/browse/HBASE-7372
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>
> January 1st is deadline for changing how we publish our website. We may no
> longer rsync out to people.apache.org. Apache infrastructure supplies two
> options here: http://www.apache.org/dev/project-site.html We could redo our
> site in apache cms format. Or we could just use svnpubsub and keep on w/ how
> the site is currently generated and on checkin, have it autopublished. I'll
> go the latter route unless I hear otherwise.
> For svnpubsub, we need to point apache infrastructure at a directory that has
> our checkedin site in it. I was thinking ${hbasedir}/hbase.apache.org
> Let me raise this on the dev list too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7384) Introducing waitForCondition function into test cases
[
https://issues.apache.org/jira/browse/HBASE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535512#comment-13535512
]
Jeffrey Zhong commented on HBASE-7384:
--
Thanks!
> Introducing waitForCondition function into test cases
> -
>
> Key: HBASE-7384
> URL: https://issues.apache.org/jira/browse/HBASE-7384
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>
> Recently I'm working on flaky test cases and found we have many places using
> while loop and sleep to wait for a condition to be true. There are several
> issues in existing ways:
> 1) Many similar code doing the same thing
> 2) When time out happens, different errors are reported without explicitly
> indicating a time out situation
> 3) When we want to increase the max timeout value to verify if a test case
> fails due to a not-enough time out value, we have to recompile & redeploy code
> I propose to create a waitForCondition function as a test utility function
> like the following:
> {code}
> public interface WaitCheck {
> public boolean Check() ;
> }
> public boolean waitForCondition(int timeOutInMilliSeconds, int
> checkIntervalInMilliSeconds, WaitCheck s)
> throws InterruptedException {
> int multiplier = 1;
> String multiplierProp = System.getProperty("extremeWaitMultiplier");
> if(multiplierProp != null) {
> multiplier = Integer.parseInt(multiplierProp);
> if(multiplier < 1) {
> LOG.warn(String.format("Invalid extremeWaitMultiplier
> property value:%s. is ignored.", multiplierProp));
> multiplier = 1;
> }
> }
> int timeElapsed = 0;
> while(timeElapsed < timeOutInMilliSeconds * multiplier) {
> if(s.Check()) {
> return true;
> }
> Thread.sleep(checkIntervalInMilliSeconds);
> timeElapsed += checkIntervalInMilliSeconds;
> }
> assertTrue("WaitForCondition failed due to time out(" +
> timeOutInMilliSeconds + " milliseconds expired)",
> false);
> return false;
> }
> {code}
> By doing the above way, there are several advantages:
> 1) Clearly report time out error when such situation happens
> 2) Use System property extremeWaitMultiplier to increase max time out
> dynamically for a quick verification
> 3) Standardize current wait situations
> Pleas let me know what your thoughts on this.
> Thanks,
> -Jeffrey
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7351) Periodic health check chore
[ https://issues.apache.org/jira/browse/HBASE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13535510#comment-13535510 ] Andrew Purtell commented on HBASE-7351: --- See comments on RB. Renamed the issue because this could/should be common to the regionserver and master, right? > Periodic health check chore > --- > > Key: HBASE-7351 > URL: https://issues.apache.org/jira/browse/HBASE-7351 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Attachments: HBASE-7331_94.patch, HBASE-7331_trunk.patch > > > Similar to MAPREDUCE-211, region server should also have a mechanism to check > the health of the node. It should run the health check script periodically > and if there is any errors, it should stop itself gracefully. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
