[jira] [Updated] (HBASE-7340) Master coprocessor notification for assignmentManager.balance() is inconsistent
[
https://issues.apache.org/jira/browse/HBASE-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-7340:
--
Resolution: Fixed
Fix Version/s: 0.96.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Integrated to trunk.
Thanks for the review, Andy.
> Master coprocessor notification for assignmentManager.balance() is
> inconsistent
> ---
>
> Key: HBASE-7340
> URL: https://issues.apache.org/jira/browse/HBASE-7340
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Trivial
> Fix For: 0.96.0
>
> Attachments: 7340-v1.txt, 7340-v2.txt
>
>
> I found this issue when reading user discussion which is quoted below.
> In HMaster.moveRegion(), we have:
> {code}
> this.assignmentManager.balance(rp);
> if (this.cpHost != null) {
> this.cpHost.postMove(hri, rp.getSource(), rp.getDestination());
> }
> {code}
> Meaning, user can register master coprocessor which would receive region
> movement notification.
> The assignmentManager.balance(plan) call in HMaster.balance() doesn't send
> out such notification.
> I think we should enhance the following hook (at line 1335) with list of
> regions moved so that notification from master is consistent:
> {code}
> this.cpHost.postBalance();
> {code}
> Here is excerpt for user discussion:
> Sometimes user performs compaction after a region is moved (by balancer). We
> should provide 'hook' which lets user specify what follow-on actions to take
> after region movement.
> See discussion on user mailing list under the thread 'How to know it's time
> for a major compaction?' for background information:
> http://search-hadoop.com/m/BDx4S1jMjF92&subj=How+to+know+it+s+time+for+a+major+compaction+
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531533#comment-13531533 ] Shrijeet Paliwal commented on HBASE-5898: - After putting UseMembar in GC opts we have not seen the previously reported deadlock. Just wanted to update. > Consider double-checked locking for block cache lock > > > Key: HBASE-5898 > URL: https://issues.apache.org/jira/browse/HBASE-5898 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: 5898-0.94.txt, 5898-TestBlocksRead.txt, 5898-v2.txt, > 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, > HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt > > > Running a workload with a high query rate against a dataset that fits in > cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by > HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a > lot of CPU doing lock management here. I wrote a quick patch to switch to a > double-checked locking and it improved throughput substantially for this > workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7299) TestMultiParallel fails intermittently in trunk builds
[
https://issues.apache.org/jira/browse/HBASE-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531543#comment-13531543
]
Ted Yu commented on HBASE-7299:
---
I looped TestMultiParallel 9 times in 0.94 branch and didn't see test failure.
Looks like the failure is unique to trunk.
> TestMultiParallel fails intermittently in trunk builds
> --
>
> Key: HBASE-7299
> URL: https://issues.apache.org/jira/browse/HBASE-7299
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 0.96.0
>
>
> From trunk build #3598:
> {code}
> testFlushCommitsNoAbort(org.apache.hadoop.hbase.client.TestMultiParallel):
> Count of regions=8
> {code}
> It failed in 3595 as well:
> {code}
> java.lang.AssertionError: Server count=2, abort=true expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.failNotEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:128)
> at org.junit.Assert.assertEquals(Assert.java:472)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.doTestFlushCommits(TestMultiParallel.java:267)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsWithAbort(TestMultiParallel.java:226)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531555#comment-13531555 ] Lars Hofhansl commented on HBASE-5898: -- Thanks Shrijeet. How reliably have you seen this before (Once a day, once a mongth, etc)? If this really causes issues we should: # ship with -XX:+UseMembar by default in hbase-env.sh # document that this must be set Are we confident enough this to do that? > Consider double-checked locking for block cache lock > > > Key: HBASE-5898 > URL: https://issues.apache.org/jira/browse/HBASE-5898 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: 5898-0.94.txt, 5898-TestBlocksRead.txt, 5898-v2.txt, > 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, > HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt > > > Running a workload with a high query rate against a dataset that fits in > cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by > HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a > lot of CPU doing lock management here. I wrote a quick patch to switch to a > double-checked locking and it improved throughput substantially for this > workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically)
[
https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531558#comment-13531558
]
Hadoop QA commented on HBASE-7055:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12560839/HBASE-7055-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
104 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 24 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3534//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3534//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3534//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3534//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3534//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3534//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3534//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3534//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3534//console
This message is automatically generated.
> port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice
> (not configurable by cf or dynamically)
> -
>
> Key: HBASE-7055
> URL: https://issues.apache.org/jira/browse/HBASE-7055
> Project: HBase
> Issue Type: Task
> Components: Compaction
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.96.0
>
> Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch,
> HBASE-6371-v3-refactor-only-squashed.patch,
> HBASE-6371-v4-refactor-only-squashed.patch,
> HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch,
> HBASE-7055-v1.patch, HBASE-7055-v2.patch, HBASE-7055-v3.patch,
> HBASE-7055-v4.patch
>
>
> There's divergence in the code :(
> See HBASE-6371 for details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7315) Remove support for client-side RowLocks
[ https://issues.apache.org/jira/browse/HBASE-7315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-7315: -- Attachment: HBASE-7315-v3.patch * Attached HBASE-7315-v3.patch * Here is a version with the NullTypes removed. > Remove support for client-side RowLocks > --- > > Key: HBASE-7315 > URL: https://issues.apache.org/jira/browse/HBASE-7315 > Project: HBase > Issue Type: Sub-task > Components: Transactions/MVCC >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.96.0 > > Attachments: HBASE-7315.patch, HBASE-7315-v2.patch, > HBASE-7315-v3.patch > > > See comments in HBASE-7263. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7299) TestMultiParallel fails intermittently in trunk builds
[
https://issues.apache.org/jira/browse/HBASE-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531574#comment-13531574
]
Ted Yu commented on HBASE-7299:
---
Looking at Jenkins builds, this was the first build where the test started to
fail:
https://builds.apache.org/job/HBase-TRUNK/3582/
Before build #3582, there were long string of build failures due to:
{code}
java.lang.AssertionError: There are 14 test classes without category: [class
org.apache.hadoop.hbase.thrift.TestMetricsThriftServerSourceFactory, class
org.apache.hadoop.hbase.regionserver.TestMetricsRegionServerSourceFactory,
class org.apache.hadoop.hbase.master.TestMetricsMasterSourceImpl, class
org.apache.hadoop.hbase.metrics.TestBaseSourceImpl, class
org.apache.hadoop.hbase.replication.regionserver.TestMetricsReplicationSourceFactory,
class org.apache.hadoop.hbase.regionserver.wal.TestMetricsHLogSource, class
org.apache.hadoop.hbase.rest.TestRESTMetricsSourceImpl, class
org.apache.hadoop.hbase.regionserver.TestMetricsRegionSourceImpl, class
org.apache.hadoop.hbase.regionserver.wal.TestMetricsWALSourceImpl, class
org.apache.hadoop.hbase.thrift.TestThriftServerMetricsSourceFactoryImpl, class
org.apache.hadoop.hbase.regionserver.TestMetricsRegionServerSourceImpl, class
org.apache.hadoop.hbase.replication.regionserver.TestReplicationMetricsSourceImpl,
class org.apache.hadoop.hbase.master.TestMetricsMasterSourceFactory, class
org.apache.hadoop.hbase.rest.TestMetricsRESTSource]
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.hbase.TestCheckTestClasses.checkClasses(TestCheckTestClasses.java:58)
{code}
So it is hard to determine when TestMultiParallel started to fail.
> TestMultiParallel fails intermittently in trunk builds
> --
>
> Key: HBASE-7299
> URL: https://issues.apache.org/jira/browse/HBASE-7299
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 0.96.0
>
>
> From trunk build #3598:
> {code}
> testFlushCommitsNoAbort(org.apache.hadoop.hbase.client.TestMultiParallel):
> Count of regions=8
> {code}
> It failed in 3595 as well:
> {code}
> java.lang.AssertionError: Server count=2, abort=true expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.failNotEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:128)
> at org.junit.Assert.assertEquals(Assert.java:472)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.doTestFlushCommits(TestMultiParallel.java:267)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsWithAbort(TestMultiParallel.java:226)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531575#comment-13531575 ] Shrijeet Paliwal commented on HBASE-5898: - Lars, we saw this twice in two days (once each day, on different region servers). It was been 36 hours since we put -XX:+UseMembar back & restarted region servers, have not seen again yet. Stack documented this as part of HBASE-3622, http://hbase.apache.org/book/trouble.rs.html > Consider double-checked locking for block cache lock > > > Key: HBASE-5898 > URL: https://issues.apache.org/jira/browse/HBASE-5898 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: 5898-0.94.txt, 5898-TestBlocksRead.txt, 5898-v2.txt, > 5898-v3.txt, 5898-v4.txt, 5898-v4.txt, HBASE-5898-0.patch, > HBASE-5898-1.patch, HBASE-5898-1.patch, hbase-5898.txt > > > Running a workload with a high query rate against a dataset that fits in > cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by > HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a > lot of CPU doing lock management here. I wrote a quick patch to switch to a > double-checked locking and it improved throughput substantially for this > workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3171) Drop ROOT and instead store META location(s) directly in ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Daniel Cryans updated HBASE-3171: -- Attachment: HBASE-3171-v2.patch This v2 has some code cleanup done, HBCK's test is passing, and I tested it on a cluster where I moved .META. and killed the RS that was then holding it while importing data. Next up is trying to figure out what to do with MetaNodeTracker and metaAvailable in CatalogTracker, it seems we could do without them? > Drop ROOT and instead store META location(s) directly in ZooKeeper > -- > > Key: HBASE-3171 > URL: https://issues.apache.org/jira/browse/HBASE-3171 > Project: HBase > Issue Type: Improvement > Components: Client, master, regionserver, Zookeeper >Reporter: Jonathan Gray > Attachments: HBASE-3171.patch, HBASE-3171-v2.patch > > > Rather than storing the ROOT region location in ZooKeeper, going to ROOT, and > reading the META location, we should just store the META location directly in > ZooKeeper. > The purpose of the root region from the bigtable paper was to support > multiple meta regions. Currently, we explicitly only support a single meta > region, so the translation from our current code of a single root location to > a single meta location will be very simple. Long-term, it seems reasonable > that we could store several meta region locations in ZK. There's been some > discussion in HBASE-1755 about actually moving META into ZK, but I think this > jira is a good step towards taking some of the complexity out of how we have > to deal with catalog tables everywhere. > As-is, a new client already requires ZK to get the root location, so this > would not change those requirements in any way. > The primary motivation for this is to simplify things like CatalogTracker. > The way we can handle root in that class is really simple but the tracking of > meta is difficulty and a bit hacky. This hack on tracking of the meta > location is what caused one of the bugs over in HBASE-3159. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7349) Jenkins build should compare trunk vs patch for Javadoc warnings
Nick Dimiduk created HBASE-7349: --- Summary: Jenkins build should compare trunk vs patch for Javadoc warnings Key: HBASE-7349 URL: https://issues.apache.org/jira/browse/HBASE-7349 Project: HBase Issue Type: Improvement Components: build Reporter: Nick Dimiduk Priority: Minor The javadoc check should look for an increase in the number of warnings. It can do so by running javadoc against trunk before running it for the patch. This will increase build times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7341) Deprecate RowLocks in 0.94
[
https://issues.apache.org/jira/browse/HBASE-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531579#comment-13531579
]
stack commented on HBASE-7341:
--
Just for the record, we've had below text in refguide for a good while:
{code}
9.3.4. RowLocks
RowLocks are still in the client API however they are discouraged because if
not managed properly these can lock up the RegionServers.
There is an oustanding ticket HBASE-2332 to remove this feature from the client.
{code}
and
{code}
12.9.2. Runtime Errors
12.9.2.1. RegionServer Hanging
Are you running an old JVM (< 1.6.0_u21?)? When you look at a thread dump, does
it look like threads are BLOCKED but no one holds the lock all are blocked on?
See HBASE 3622 Deadlock in HBaseServer (JVM bug?). Adding -XX:+UseMembar to the
HBase HBASE_OPTS in conf/hbase-env.sh may fix it.
Also, are you using Section 9.3.4, “RowLocks”? These are discouraged because
they can lock up the RegionServers if not managed properly.
{code}
> Deprecate RowLocks in 0.94
> --
>
> Key: HBASE-7341
> URL: https://issues.apache.org/jira/browse/HBASE-7341
> Project: HBase
> Issue Type: Task
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.94.4
>
> Attachments: HBASE-7341.patch
>
>
> Since we are removing support in 0.96 (see HBASE-7315), we should deprecate
> in 0.94.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531586#comment-13531586 ] Sergey Shelukhin commented on HBASE-5416: - With default setting, I got tired of waiting for it after an ~hour and canceled it. I think it was still inserting rows, although my laptop drive was extreley sluggish so maybe logs haven't flushed so I didn't see "Data inserted" log message. However, after changing the number of writes to 5k, I get very impressive performance improvement: 2012-12-13 14:25:48,699 INFO [main] regionserver.TestJoinedScanners(155): Slow scanner finished in 5.15852 seconds, got 50 rows ... 2012-12-13 14:25:48,931 INFO [main] regionserver.TestJoinedScanners(175): Joined scanner finished in 0.231946 seconds, got 50 rows It looks like it would be a good idea from perf standpoint to include this in trunk. [~shmuma] do you want to continue on the patch? Otherwise I can continue on it. I think we'd want to make it optional (in config and/or per request) to address some of the stability concerns. I want to look at patch in more detail too to see what more testing can be done. It seems like maybe two LoadTestTool-s writing and verifying against the same table in parallel could be used, or something, in an integration test. I get "Filter with filterRow(List) incompatible with scan with limit!" in testScanner_JoinedScannersAndLimits after rebase, I'll look into it. > Improve performance of scans with some kind of filters. > --- > > Key: HBASE-5416 > URL: https://issues.apache.org/jira/browse/HBASE-5416 > Project: HBase > Issue Type: Improvement > Components: Filters, Performance, regionserver >Affects Versions: 0.90.4 >Reporter: Max Lapan >Assignee: Max Lapan > Fix For: 0.96.0 > > Attachments: 5416-Filtered_scans_v6.patch, 5416-v5.txt, 5416-v6.txt, > Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch, > Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch, > Filtered_scans_v7.patch > > > When the scan is performed, whole row is loaded into result list, after that > filter (if exists) is applied to detect that row is needed. > But when scan is performed on several CFs and filter checks only data from > the subset of these CFs, data from CFs, not checked by a filter is not needed > on a filter stage. Only when we decided to include current row. And in such > case we can significantly reduce amount of IO performed by a scan, by loading > only values, actually checked by a filter. > For example, we have two CFs: flags and snap. Flags is quite small (bunch of > megabytes) and is used to filter large entries from snap. Snap is very large > (10s of GB) and it is quite costly to scan it. If we needed only rows with > some flag specified, we use SingleColumnValueFilter to limit result to only > small subset of region. But current implementation is loading both CFs to > perform scan, when only small subset is needed. > Attached patch adds one routine to Filter interface to allow filter to > specify which CF is needed to it's operation. In HRegion, we separate all > scanners into two groups: needed for filter and the rest (joined). When new > row is considered, only needed data is loaded, filter applied, and only if > filter accepts the row, rest of data is loaded. At our data, this speeds up > such kind of scans 30-50 times. Also, this gives us the way to better > normalize the data into separate columns by optimizing the scans performed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7350) Flakey tests make CI unreliable
Nick Dimiduk created HBASE-7350: --- Summary: Flakey tests make CI unreliable Key: HBASE-7350 URL: https://issues.apache.org/jira/browse/HBASE-7350 Project: HBase Issue Type: Bug Components: build Reporter: Nick Dimiduk Most PreCommit and PostCommit builds are failing these days. Keeping an eye on usual suspects reveals them to be mostly longer-running tests. Either the tests need to me made more rigorous or the erroneous Jenkins configuration needs addressed. The usual suspects: - org.apache.hadoop.hbase.client.TestMultiParallel - org.apache.hadoop.hbase.TestDrainingServer - org.apache.hadoop.hbase.regionserver.TestSplitTransaction - org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsNoAbort - org.apache.hadoop.hbase.replication.TestReplication - org.apache.hadoop.hbase.util.TestHBaseFsck -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7340) Master coprocessor notification for assignmentManager.balance() is inconsistent
[
https://issues.apache.org/jira/browse/HBASE-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531602#comment-13531602
]
Hudson commented on HBASE-7340:
---
Integrated in HBase-TRUNK #3619 (See
[https://builds.apache.org/job/HBase-TRUNK/3619/])
HBASE-7340 Master coprocessor notification for assignmentManager.balance()
is inconsistent (Ted Yu) (Revision 1421537)
Result = FAILURE
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
> Master coprocessor notification for assignmentManager.balance() is
> inconsistent
> ---
>
> Key: HBASE-7340
> URL: https://issues.apache.org/jira/browse/HBASE-7340
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Trivial
> Fix For: 0.96.0
>
> Attachments: 7340-v1.txt, 7340-v2.txt
>
>
> I found this issue when reading user discussion which is quoted below.
> In HMaster.moveRegion(), we have:
> {code}
> this.assignmentManager.balance(rp);
> if (this.cpHost != null) {
> this.cpHost.postMove(hri, rp.getSource(), rp.getDestination());
> }
> {code}
> Meaning, user can register master coprocessor which would receive region
> movement notification.
> The assignmentManager.balance(plan) call in HMaster.balance() doesn't send
> out such notification.
> I think we should enhance the following hook (at line 1335) with list of
> regions moved so that notification from master is consistent:
> {code}
> this.cpHost.postBalance();
> {code}
> Here is excerpt for user discussion:
> Sometimes user performs compaction after a region is moved (by balancer). We
> should provide 'hook' which lets user specify what follow-on actions to take
> after region movement.
> See discussion on user mailing list under the thread 'How to know it's time
> for a major compaction?' for background information:
> http://search-hadoop.com/m/BDx4S1jMjF92&subj=How+to+know+it+s+time+for+a+major+compaction+
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7349) Jenkins build should compare trunk vs patch for Javadoc warnings
[ https://issues.apache.org/jira/browse/HBASE-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7349: Attachment: 7349-build-improve-javadoc-warnings.0.diff Compare the number of warnings produced on trunk to those produced by the patch and report as appropriate. Remove the orphaned OK_JAVADOC_WARNINGS property. > Jenkins build should compare trunk vs patch for Javadoc warnings > > > Key: HBASE-7349 > URL: https://issues.apache.org/jira/browse/HBASE-7349 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Nick Dimiduk >Priority: Minor > Attachments: 7349-build-improve-javadoc-warnings.0.diff > > > The javadoc check should look for an increase in the number of warnings. It > can do so by running javadoc against trunk before running it for the patch. > This will increase build times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7349) Jenkins build should compare trunk vs patch for Javadoc warnings
[ https://issues.apache.org/jira/browse/HBASE-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7349: Assignee: Nick Dimiduk Status: Patch Available (was: Open) > Jenkins build should compare trunk vs patch for Javadoc warnings > > > Key: HBASE-7349 > URL: https://issues.apache.org/jira/browse/HBASE-7349 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Attachments: 7349-build-improve-javadoc-warnings.0.diff > > > The javadoc check should look for an increase in the number of warnings. It > can do so by running javadoc against trunk before running it for the patch. > This will increase build times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues
[
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531619#comment-13531619
]
Matt Corgan commented on HBASE-7233:
{quote}Would we still have the trouble of teaching KeyValue about where in the
bytebuffers coming out of the encoder the tag data resides?{quote}Tags are a
little different than our current fields because there are multiple per Cell.
For performance sake, we may want to keep the Cell interface to these low level
methods: getTagsArray(), getNumTags(), getTagsOffset(), getTagsLength(), and
then have methods for parsing them like {code}Iterable tags =
CellTool.getTagsIterator(cell){code}. So the requirement for the
encoder/decoder would be to line them up in a single array: .
Behind the scenes, tags could be encoded similarly to qualifiers (speaking for
prefix-tree)
{quote}Any thoughts on how we might distinguish a KV with tags from one
without? {quote}Could just have Cell.getNumTags() return 0
> Serializing KeyValues
> -
>
> Key: HBASE-7233
> URL: https://issues.apache.org/jira/browse/HBASE-7233
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 0.96.0
>
> Attachments: 7233sketch.txt, 7233.txt, 7233-v2.txt,
> 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt,
> 7233v6_encoder.txt
>
>
> Undo KeyValue being a Writable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7315) Remove support for client-side RowLocks
[
https://issues.apache.org/jira/browse/HBASE-7315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531620#comment-13531620
]
Hadoop QA commented on HBASE-7315:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12560857/HBASE-7315-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 88 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
104 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 21 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestShell
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3535//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3535//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3535//console
This message is automatically generated.
> Remove support for client-side RowLocks
> ---
>
> Key: HBASE-7315
> URL: https://issues.apache.org/jira/browse/HBASE-7315
> Project: HBase
> Issue Type: Sub-task
> Components: Transactions/MVCC
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Fix For: 0.96.0
>
> Attachments: HBASE-7315.patch, HBASE-7315-v2.patch,
> HBASE-7315-v3.patch
>
>
> See comments in HBASE-7263.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7315) Remove support for client-side RowLocks
[ https://issues.apache.org/jira/browse/HBASE-7315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531622#comment-13531622 ] Gregory Chanan commented on HBASE-7315: --- TestShell failure is reproducible, I'm taking a look. > Remove support for client-side RowLocks > --- > > Key: HBASE-7315 > URL: https://issues.apache.org/jira/browse/HBASE-7315 > Project: HBase > Issue Type: Sub-task > Components: Transactions/MVCC >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.96.0 > > Attachments: HBASE-7315.patch, HBASE-7315-v2.patch, > HBASE-7315-v3.patch > > > See comments in HBASE-7263. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7350) Flakey tests make CI unreliable
[ https://issues.apache.org/jira/browse/HBASE-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7350: Attachment: 7350-build-reduce-parallelism.0.diff Seeking test stability: reduce the secondPartThreadCount to 1. > Flakey tests make CI unreliable > --- > > Key: HBASE-7350 > URL: https://issues.apache.org/jira/browse/HBASE-7350 > Project: HBase > Issue Type: Bug > Components: build >Reporter: Nick Dimiduk > Attachments: 7350-build-reduce-parallelism.0.diff, > 7350-build-warn-on-terminate.0.diff > > > Most PreCommit and PostCommit builds are failing these days. Keeping an eye > on usual suspects reveals them to be mostly longer-running tests. Either the > tests need to me made more rigorous or the erroneous Jenkins configuration > needs addressed. > The usual suspects: > - org.apache.hadoop.hbase.client.TestMultiParallel > - org.apache.hadoop.hbase.TestDrainingServer > - org.apache.hadoop.hbase.regionserver.TestSplitTransaction > - org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsNoAbort > - org.apache.hadoop.hbase.replication.TestReplication > - org.apache.hadoop.hbase.util.TestHBaseFsck -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7350) Flakey tests make CI unreliable
[ https://issues.apache.org/jira/browse/HBASE-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7350: Attachment: 7350-build-warn-on-terminate.0.diff warn before terminating any 'rogue' build processes. The More You Know! > Flakey tests make CI unreliable > --- > > Key: HBASE-7350 > URL: https://issues.apache.org/jira/browse/HBASE-7350 > Project: HBase > Issue Type: Bug > Components: build >Reporter: Nick Dimiduk > Attachments: 7350-build-reduce-parallelism.0.diff, > 7350-build-warn-on-terminate.0.diff > > > Most PreCommit and PostCommit builds are failing these days. Keeping an eye > on usual suspects reveals them to be mostly longer-running tests. Either the > tests need to me made more rigorous or the erroneous Jenkins configuration > needs addressed. > The usual suspects: > - org.apache.hadoop.hbase.client.TestMultiParallel > - org.apache.hadoop.hbase.TestDrainingServer > - org.apache.hadoop.hbase.regionserver.TestSplitTransaction > - org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsNoAbort > - org.apache.hadoop.hbase.replication.TestReplication > - org.apache.hadoop.hbase.util.TestHBaseFsck -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7350) Flakey tests make CI unreliable
[ https://issues.apache.org/jira/browse/HBASE-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7350: Status: Patch Available (was: Open) > Flakey tests make CI unreliable > --- > > Key: HBASE-7350 > URL: https://issues.apache.org/jira/browse/HBASE-7350 > Project: HBase > Issue Type: Bug > Components: build >Reporter: Nick Dimiduk > Attachments: 7350-build-reduce-parallelism.0.diff, > 7350-build-warn-on-terminate.0.diff > > > Most PreCommit and PostCommit builds are failing these days. Keeping an eye > on usual suspects reveals them to be mostly longer-running tests. Either the > tests need to me made more rigorous or the erroneous Jenkins configuration > needs addressed. > The usual suspects: > - org.apache.hadoop.hbase.client.TestMultiParallel > - org.apache.hadoop.hbase.TestDrainingServer > - org.apache.hadoop.hbase.regionserver.TestSplitTransaction > - org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsNoAbort > - org.apache.hadoop.hbase.replication.TestReplication > - org.apache.hadoop.hbase.util.TestHBaseFsck -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7350) Flakey tests make CI unreliable
[ https://issues.apache.org/jira/browse/HBASE-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531654#comment-13531654 ] stack commented on HBASE-7350: -- Is this an experiment Nick? Not permanent right (lowering thread count)? > Flakey tests make CI unreliable > --- > > Key: HBASE-7350 > URL: https://issues.apache.org/jira/browse/HBASE-7350 > Project: HBase > Issue Type: Bug > Components: build >Reporter: Nick Dimiduk > Attachments: 7350-build-reduce-parallelism.0.diff, > 7350-build-warn-on-terminate.0.diff > > > Most PreCommit and PostCommit builds are failing these days. Keeping an eye > on usual suspects reveals them to be mostly longer-running tests. Either the > tests need to me made more rigorous or the erroneous Jenkins configuration > needs addressed. > The usual suspects: > - org.apache.hadoop.hbase.client.TestMultiParallel > - org.apache.hadoop.hbase.TestDrainingServer > - org.apache.hadoop.hbase.regionserver.TestSplitTransaction > - org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsNoAbort > - org.apache.hadoop.hbase.replication.TestReplication > - org.apache.hadoop.hbase.util.TestHBaseFsck -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[ https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7268: Attachment: HBASE-7268-v2-plus-masterTs.patch Adding patch that uses master timestamp for open and close. This adds a lot of plumbing and messes with META timestamps... I wonder if it's worth it. > correct local region location cache information can be overwritten w/stale > information from an old server > - > > Key: HBASE-7268 > URL: https://issues.apache.org/jira/browse/HBASE-7268 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7268-v0.patch, HBASE-7268-v0.patch, > HBASE-7268-v1.patch, HBASE-7268-v2.patch, HBASE-7268-v2-plus-masterTs.patch > > > Discovered via HBASE-7250; related to HBASE-5877. > Test is writing from multiple threads. > Server A has region R; client knows that. > R gets moved from A to server B. > B gets killed. > R gets moved by master to server C. > ~15 seconds later, client tries to write to it (on A?). > Multiple client threads report from RegionMoved exception processing logic "R > moved from C to B", even though such transition never happened (neither in > nor before the sequence described below). Not quite sure how the client > learned of the transition to C, I assume it's from meta from some other > thread... > Then, put fails (it may fail due to accumulated errors that are not logged, > which I am investigating... but the bogus cache update is there > nonwithstanding). > I have a patch but not sure if it works, test still fails locally for yet > unknown reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7350) Flakey tests make CI unreliable
[ https://issues.apache.org/jira/browse/HBASE-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531663#comment-13531663 ] Nick Dimiduk commented on HBASE-7350: - The warning is more for information gathering -- are we frequently abandoning and killing processes? I see no reason not to commit it permanently. Reducing the thread count is an experiment to see if our more strenuous integration tests pass more consistently when run with less concurrency. I don't actually know what tests are included in the "secondPart," hence the request for review from others more knowledgeable. > Flakey tests make CI unreliable > --- > > Key: HBASE-7350 > URL: https://issues.apache.org/jira/browse/HBASE-7350 > Project: HBase > Issue Type: Bug > Components: build >Reporter: Nick Dimiduk > Attachments: 7350-build-reduce-parallelism.0.diff, > 7350-build-warn-on-terminate.0.diff > > > Most PreCommit and PostCommit builds are failing these days. Keeping an eye > on usual suspects reveals them to be mostly longer-running tests. Either the > tests need to me made more rigorous or the erroneous Jenkins configuration > needs addressed. > The usual suspects: > - org.apache.hadoop.hbase.client.TestMultiParallel > - org.apache.hadoop.hbase.TestDrainingServer > - org.apache.hadoop.hbase.regionserver.TestSplitTransaction > - org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsNoAbort > - org.apache.hadoop.hbase.replication.TestReplication > - org.apache.hadoop.hbase.util.TestHBaseFsck -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7336) HFileBlock.readAtOffset does not work well with multiple threads
[ https://issues.apache.org/jira/browse/HBASE-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531664#comment-13531664 ] Hudson commented on HBASE-7336: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #295 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/295/]) HBASE-7336 HFileBlock.readAtOffset does not work well with multiple threads (Revision 1421440) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java > HFileBlock.readAtOffset does not work well with multiple threads > > > Key: HBASE-7336 > URL: https://issues.apache.org/jira/browse/HBASE-7336 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 0.96.0, 0.94.4 > > Attachments: 7336-0.94.txt, 7336-0.96.txt > > > HBase grinds to a halt when many threads scan along the same set of blocks > and neither read short circuit is nor block caching is enabled for the dfs > client ... disabling the block cache makes sense on very large scans. > It turns out that synchronizing in istream in HFileBlock.readAtOffset is the > culprit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7340) Master coprocessor notification for assignmentManager.balance() is inconsistent
[
https://issues.apache.org/jira/browse/HBASE-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531665#comment-13531665
]
Hudson commented on HBASE-7340:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #295 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/295/])
HBASE-7340 Master coprocessor notification for assignmentManager.balance()
is inconsistent (Ted Yu) (Revision 1421537)
Result = FAILURE
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
> Master coprocessor notification for assignmentManager.balance() is
> inconsistent
> ---
>
> Key: HBASE-7340
> URL: https://issues.apache.org/jira/browse/HBASE-7340
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Trivial
> Fix For: 0.96.0
>
> Attachments: 7340-v1.txt, 7340-v2.txt
>
>
> I found this issue when reading user discussion which is quoted below.
> In HMaster.moveRegion(), we have:
> {code}
> this.assignmentManager.balance(rp);
> if (this.cpHost != null) {
> this.cpHost.postMove(hri, rp.getSource(), rp.getDestination());
> }
> {code}
> Meaning, user can register master coprocessor which would receive region
> movement notification.
> The assignmentManager.balance(plan) call in HMaster.balance() doesn't send
> out such notification.
> I think we should enhance the following hook (at line 1335) with list of
> regions moved so that notification from master is consistent:
> {code}
> this.cpHost.postBalance();
> {code}
> Here is excerpt for user discussion:
> Sometimes user performs compaction after a region is moved (by balancer). We
> should provide 'hook' which lets user specify what follow-on actions to take
> after region movement.
> See discussion on user mailing list under the thread 'How to know it's time
> for a major compaction?' for background information:
> http://search-hadoop.com/m/BDx4S1jMjF92&subj=How+to+know+it+s+time+for+a+major+compaction+
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7211) Improve hbase ref guide for the testing part.
[ https://issues.apache.org/jira/browse/HBASE-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531666#comment-13531666 ] Hudson commented on HBASE-7211: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #295 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/295/]) HBASE-7211 Improve hbase ref guide for the testing part. - 1st part (Jeffrey Zhong) (Revision 1421295) Result = FAILURE nkeywal : Files : * /hbase/trunk/pom.xml * /hbase/trunk/src/docbkx/developer.xml * /hbase/trunk/src/docbkx/zookeeper.xml > Improve hbase ref guide for the testing part. > - > > Key: HBASE-7211 > URL: https://issues.apache.org/jira/browse/HBASE-7211 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal >Priority: Minor > Attachments: hbase-7211-partial.patch > > > Here is some stuff I saw. I will propose a fix in a week or so, please add > the comment or issues you have in mind. > ??15.6.1. Apache HBase Modules?? > => We should be able to use categories in all modules. The default should be > small; but any test manipulating the time needs to be in a specific jvm > (hence medium), so it's not always related to minicluster. > ??15.6.3.6. hbasetests.sh?? > => We can remove this chapter, and the script > The script is not totally useless, but I think nobody actually uses it. > => Add a chapter on flakiness. > Some tests are, unfortunately, flaky. While there number decreases, we still > have some. Rules are: > - don't write flaky tests! :-) > - small tests cannot be flaky, as it blocks other test execution. Corollary: > if you have an issue with a small test, it's either your environment either a > severe issue. > - rerun the test a few time to validate, check the ports and file descriptors > used. > ??mvn test -P localTests -Dtest=MyTest?? > => We could actually activate the localTests profile whenever -Dtest is used. > If we do that, we can remove the reference from localTests in the doc. > ??mvn test -P runSmallTests?? ??mvn test -P runMediumTests?? > => I'm not sure it's actually used. We could remove them from the pom.xml > (and the doc). > ??The HBase build uses a patched version of the maven surefire plugin?? > => Hopefully, we will be able to remove this soon :-) > ??Integration tests are described TODO: POINTER_TO_INTEGRATION_TEST_SECTION?? > => Should be documented -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7338) Fix flaky condition for org.apache.hadoop.hbase.TestRegionRebalancing.testRebalanceOnRegionServerNumberChange
[
https://issues.apache.org/jira/browse/HBASE-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531667#comment-13531667
]
Hudson commented on HBASE-7338:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #295 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/295/])
HBASE-7338 Fix flaky condition for
org.apache.hadoop.hbase.TestRegionRebalancing.testRebalanceOnRegionServerNumberChange
(Himanshu) (Revision 1421444)
Result = FAILURE
jxiang :
Files :
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
> Fix flaky condition for
> org.apache.hadoop.hbase.TestRegionRebalancing.testRebalanceOnRegionServerNumberChange
> -
>
> Key: HBASE-7338
> URL: https://issues.apache.org/jira/browse/HBASE-7338
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.94.3, 0.96.0
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-7338.patch
>
>
> The balancer doesn't run in case a region is in-transition. The check to
> confirm whether there all regions are assigned looks for region count > 22,
> where the total regions are 27. This may result in a failure:
> {code}
> java.lang.AssertionError: After 5 attempts, region assignments were not
> balanced.
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.hadoop.hbase.TestRegionRebalancing.assertRegionsAreBalanced(TestRegionRebalancing.java:203)
> at
> org.apache.hadoop.hbase.TestRegionRebalancing.testRebalanceOnRegionServerNumberChange(TestRegionRebalancing.java:123)
> .
> 2012-12-11 13:47:02,231 INFO [pool-1-thread-1]
> hbase.TestRegionRebalancing(120): Added fourth
> server=p0118.mtv.cloudera.com,44414,1355262422083
> 2012-12-11 13:47:02,231 INFO
> [RegionServer:3;p0118.mtv.cloudera.com,44414,1355262422083]
> regionserver.HRegionServer(3769): Registered RegionServer MXBean
> 2012-12-11 13:47:02,231 DEBUG [pool-1-thread-1] master.HMaster(987): Not
> running balancer because 1 region(s) in transition:
> {c786446fb2542f190e937057cdc79d9d=test,kkk,1355262401365.c786446fb2542f190e937057cdc79d9d.
> state=OPENING, ts=1355262421037,
> server=p0118.mtv.cloudera.com,54281,1355262419765}
> 2012-12-11 13:47:02,232 DEBUG [pool-1-thread-1]
> hbase.TestRegionRebalancing(165): There are 4 servers and 26 regions. Load
> Average: 13.0 low border: 9, up border: 16; attempt: 0
> 2012-12-11 13:47:02,232 DEBUG [pool-1-thread-1]
> hbase.TestRegionRebalancing(171): p0118.mtv.cloudera.com,51590,1355262395329
> Avg: 13.0 actual: 11
> 2012-12-11 13:47:02,232 DEBUG [pool-1-thread-1]
> hbase.TestRegionRebalancing(171): p0118.mtv.cloudera.com,52987,1355262407916
> Avg: 13.0 actual: 15
> 2012-12-11 13:47:02,233 DEBUG [pool-1-thread-1]
> hbase.TestRegionRebalancing(171): p0118.mtv.cloudera.com,48044,1355262421787
> Avg: 13.0 actual: 0
> 2012-12-11 13:47:02,233 DEBUG [pool-1-thread-1]
> hbase.TestRegionRebalancing(179): p0118.mtv.cloudera.com,48044,1355262421787
> Isn't balanced!!! Avg: 13.0 actual: 0 slop: 0.2
> 2012-12-11 13:47:12,233 DEBUG [pool-1-thread-1] master.HMaster(987): Not
> running balancer because 1 region(s) in transition:
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531668#comment-13531668 ] Andrew Purtell commented on HBASE-7233: --- bq. For performance sake, we may want to keep the Cell interface to these low level methods: getTagsArray(), getNumTags(), getTagsOffset(), getTagsLength(), and then have methods for parsing them [an iterator] Agreed. My hack-patch does approximately this. I need to study up on how Cells would be materialized into KeyValues. In the AccessController we wrap InternalScanners with a filter that looks at each KeyValue on the way out to the client and evaluates their visibility to the user. Somehow from the KeyValue API we'd need to get to the cell tags iterator to extract the ACL (or visibility tag). A type byte or even making tags name-value pairs would avoid accumulation of ad-hoc means for distinguishing between them. Tags stored on disk shouldn't necessarily be sent to clients, though for the sake of performance we can concede this, where/if streaming on disk encoding directly to the client. > Serializing KeyValues > - > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7233sketch.txt, 7233.txt, 7233-v2.txt, > 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, > 7233v6_encoder.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7349) Jenkins build should compare trunk vs patch for Javadoc warnings
[
https://issues.apache.org/jira/browse/HBASE-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531670#comment-13531670
]
Hadoop QA commented on HBASE-7349:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12560863/7349-build-improve-javadoc-warnings.0.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
104 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 23 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3536//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3536//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3536//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3536//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3536//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3536//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3536//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3536//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3536//console
This message is automatically generated.
> Jenkins build should compare trunk vs patch for Javadoc warnings
>
>
> Key: HBASE-7349
> URL: https://issues.apache.org/jira/browse/HBASE-7349
> Project: HBase
> Issue Type: Improvement
> Components: build
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Attachments: 7349-build-improve-javadoc-warnings.0.diff
>
>
> The javadoc check should look for an increase in the number of warnings. It
> can do so by running javadoc against trunk before running it for the patch.
> This will increase build times.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7349) Jenkins build should compare trunk vs patch for Javadoc warnings
[ https://issues.apache.org/jira/browse/HBASE-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531673#comment-13531673 ] stack commented on HBASE-7349: -- [~ndimiduk] Did your patch do the right thing? > Jenkins build should compare trunk vs patch for Javadoc warnings > > > Key: HBASE-7349 > URL: https://issues.apache.org/jira/browse/HBASE-7349 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Attachments: 7349-build-improve-javadoc-warnings.0.diff > > > The javadoc check should look for an increase in the number of warnings. It > can do so by running javadoc against trunk before running it for the patch. > This will increase build times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7350) Flakey tests make CI unreliable
[ https://issues.apache.org/jira/browse/HBASE-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531676#comment-13531676 ] stack commented on HBASE-7350: -- Second part are all of our medium and above tests. The first part runs all tests in the one jvm. If we run w/ single thread, the build will take forever. I don't mind trying it if it will help you debug. Let me commit first info output change. > Flakey tests make CI unreliable > --- > > Key: HBASE-7350 > URL: https://issues.apache.org/jira/browse/HBASE-7350 > Project: HBase > Issue Type: Bug > Components: build >Reporter: Nick Dimiduk > Attachments: 7350-build-reduce-parallelism.0.diff, > 7350-build-warn-on-terminate.0.diff > > > Most PreCommit and PostCommit builds are failing these days. Keeping an eye > on usual suspects reveals them to be mostly longer-running tests. Either the > tests need to me made more rigorous or the erroneous Jenkins configuration > needs addressed. > The usual suspects: > - org.apache.hadoop.hbase.client.TestMultiParallel > - org.apache.hadoop.hbase.TestDrainingServer > - org.apache.hadoop.hbase.regionserver.TestSplitTransaction > - org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsNoAbort > - org.apache.hadoop.hbase.replication.TestReplication > - org.apache.hadoop.hbase.util.TestHBaseFsck -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7350) Flakey tests make CI unreliable
[ https://issues.apache.org/jira/browse/HBASE-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531677#comment-13531677 ] stack commented on HBASE-7350: -- I committed 7350-build-warn-on-terminate.0.diff > Flakey tests make CI unreliable > --- > > Key: HBASE-7350 > URL: https://issues.apache.org/jira/browse/HBASE-7350 > Project: HBase > Issue Type: Bug > Components: build >Reporter: Nick Dimiduk > Attachments: 7350-build-reduce-parallelism.0.diff, > 7350-build-warn-on-terminate.0.diff > > > Most PreCommit and PostCommit builds are failing these days. Keeping an eye > on usual suspects reveals them to be mostly longer-running tests. Either the > tests need to me made more rigorous or the erroneous Jenkins configuration > needs addressed. > The usual suspects: > - org.apache.hadoop.hbase.client.TestMultiParallel > - org.apache.hadoop.hbase.TestDrainingServer > - org.apache.hadoop.hbase.regionserver.TestSplitTransaction > - org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsNoAbort > - org.apache.hadoop.hbase.replication.TestReplication > - org.apache.hadoop.hbase.util.TestHBaseFsck -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7351) Region server should have health check chore which would run periodically.
Vandana Ayyalasomayajula created HBASE-7351: --- Summary: Region server should have health check chore which would run periodically. Key: HBASE-7351 URL: https://issues.apache.org/jira/browse/HBASE-7351 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Priority: Minor Similar to MAPREDUCE-211, region server should also have a mechanism to check the health of the node. It should run the health check script periodically and if there is any errors, it should stop itself gracefully. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues
[
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531685#comment-13531685
]
Ted Yu commented on HBASE-7233:
---
bq. A type byte or
Cell interface already provides type byte:
{code}
/**
* see {@link #KeyValue.TYPE}
* @return The byte representation of the KeyValue.TYPE of this cell: one of
Put, Delete, etc
*/
byte getTypeByte();
{code}
> Serializing KeyValues
> -
>
> Key: HBASE-7233
> URL: https://issues.apache.org/jira/browse/HBASE-7233
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 0.96.0
>
> Attachments: 7233sketch.txt, 7233.txt, 7233-v2.txt,
> 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt,
> 7233v6_encoder.txt
>
>
> Undo KeyValue being a Writable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531686#comment-13531686 ] Andrew Purtell commented on HBASE-7233: --- [~ted_yu] A type byte for the tag, not the KeyValue > Serializing KeyValues > - > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7233sketch.txt, 7233.txt, 7233-v2.txt, > 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, > 7233v6_encoder.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5979) Non-pread DFSInputStreams should be associated with scanners, not HFile.Readers
[ https://issues.apache.org/jira/browse/HBASE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531690#comment-13531690 ] Lars Hofhansl commented on HBASE-5979: -- Can we use a scan's cacheBlocks flag as an indicator for whether it would benefit from its own input stream? Presumably only large scans would have caching disabled. HBASE-7336 has a simple adhoc fix for the truly terrible read performance when two long running scanners scan the same store file. > Non-pread DFSInputStreams should be associated with scanners, not > HFile.Readers > --- > > Key: HBASE-5979 > URL: https://issues.apache.org/jira/browse/HBASE-5979 > Project: HBase > Issue Type: Improvement > Components: Performance, regionserver >Reporter: Todd Lipcon > > Currently, every HFile.Reader has a single DFSInputStream, which it uses to > service all gets and scans. For gets, we use the positional read API (aka > "pread") and for scans we use a synchronized block to seek, then read. The > advantage of pread is that it doesn't hold any locks, so multiple gets can > proceed at the same time. The advantage of seek+read for scans is that the > datanode starts to send the entire rest of the HDFS block, rather than just > the single hfile block necessary. So, in a single thread, pread is faster for > gets, and seek+read is faster for scans since you get a strong pipelining > effect. > However, in a multi-threaded case where there are multiple scans (including > scans which are actually part of compactions), the seek+read strategy falls > apart, since only one scanner may be reading at a time. Additionally, a large > amount of wasted IO is generated on the datanode side, and we get none of the > earlier-mentioned advantages. > In one test, I switched scans to always use pread, and saw a 5x improvement > in throughput of the YCSB scan-only workload, since it previously was > completely blocked by contention on the DFSIS lock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
Jonathan Hsieh created HBASE-7352:
-
Summary: clone operation from HBaseAdmin can hang forever.
Key: HBASE-7352
URL: https://issues.apache.org/jira/browse/HBASE-7352
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Hsieh
Sometimes the clone operation from the hbase shell can hang. The table has
been created (it shows up in the web ui), but does not have any entries in META.
There don't seem to be any clone, snapshot, enable or disable found in the
master's jstack.
Here's a trace from the HBaseAdmin:
{code}
"main" prio=10 tid=0x7f782800d000 nid=0x25c waiting on condition
[0x7f782f9bf000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
at
org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
at
org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
at
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
... (more jruby stack) ...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7349) Jenkins build should compare trunk vs patch for Javadoc warnings
[ https://issues.apache.org/jira/browse/HBASE-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531694#comment-13531694 ] Nick Dimiduk commented on HBASE-7349: - Unknown. The script is already running when it applies itself. I think the only way to test it is to create a branch, apply it, and then force a build of the branch. > Jenkins build should compare trunk vs patch for Javadoc warnings > > > Key: HBASE-7349 > URL: https://issues.apache.org/jira/browse/HBASE-7349 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Attachments: 7349-build-improve-javadoc-warnings.0.diff > > > The javadoc check should look for an increase in the number of warnings. It > can do so by running javadoc against trunk before running it for the patch. > This will increase build times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7349) Jenkins build should compare trunk vs patch for Javadoc warnings
[ https://issues.apache.org/jira/browse/HBASE-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531855#comment-13531855 ] stack commented on HBASE-7349: -- [~ndimiduk] Or commit it and keep an eye on it? I can commit if you keep an eye on it (smile) > Jenkins build should compare trunk vs patch for Javadoc warnings > > > Key: HBASE-7349 > URL: https://issues.apache.org/jira/browse/HBASE-7349 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Attachments: 7349-build-improve-javadoc-warnings.0.diff > > > The javadoc check should look for an increase in the number of warnings. It > can do so by running javadoc against trunk before running it for the patch. > This will increase build times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531857#comment-13531857
]
Jonathan Hsieh commented on HBASE-7352:
---
The snapshot can be cloned from another shell instance but must write to a
different name.
If you attempt to clone the snapshot (pe-11) to the same target table
(pe-11-table), you get:
{code}
hbase(main):006:0> clone_snapshot 'pe-11', 'pe-11-table'
ERROR: org.apache.hadoop.hbase.snapshot.exception.RestoreSnapshotException:
org.apache.hadoop.hbase.snapshot.exception.RestoreSnapshotExcept
ion: Couldn't clone the snapshot=name: "pe-11"
table: "TestTable"
creationTime: 1355441918484
type: FLUSH
version: 0
on table=pe-11-table
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:558)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.restoreSnapshot(SnapshotManager.java:597)
at
org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2528)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Server.call(ProtobufRpcEngine.java:356)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1816)
Caused by: org.apache.hadoop.hbase.TableExistsException: pe-11-table
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.(CreateTableHandler.java:96)
at
org.apache.hadoop.hbase.master.snapshot.CloneSnapshotHandler.(CloneSnapshotHandler.java:65)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:551)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:558)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.restoreSnapshot(SnapshotManager.java:597)
at
org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2528)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Server.call(ProtobufRpcEngine.java:356)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1816)
Caused by: org.apache.hadoop.hbase.TableExistsException: pe-11-table
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.(CreateTableHandler.java:96)
at
org.apache.hadoop.hbase.master.snapshot.CloneSnapshotHandler.(CloneSnapshotHandler.java:65)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:551)
... 7 more
{code}
The dir for seems to be present for table pe-11, but there seems to be a large
number of missing files.
In this particular case, the snapshot has 16 regions, while the failed attempt
to restore has 12 regions moved into real table position. This suggest that
something failed internally but was allowed to continue to do the dir rename at
some point.
If one removing the data bad data from the hdfs we still cannot clone to the
same pe-11-table name because there is some in memory state that blocks this.
> clone operation from HBaseAdmin can hang forever.
> -
>
> Key: HBASE-7352
> URL: https://issues.apache.org/jira/browse/HBASE-7352
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0
>
>
> Sometimes the clone operation from the hbase shell can hang. The table has
> been created (it shows up in the web ui), but does not have any entries in
> META.
> There don't seem to be any clone, snapshot, enable or disable found in the
> master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x7f782800d000 nid=0x25c waiting on condition
> [0x7f782f9bf000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
[jira] [Commented] (HBASE-7349) Jenkins build should compare trunk vs patch for Javadoc warnings
[ https://issues.apache.org/jira/browse/HBASE-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531861#comment-13531861 ] Nick Dimiduk commented on HBASE-7349: - [~stack] I sure as hell don't trust my AWK but if you're comfortable with it, help yourself. I'll continue to watch the PreCommit builds regardless. > Jenkins build should compare trunk vs patch for Javadoc warnings > > > Key: HBASE-7349 > URL: https://issues.apache.org/jira/browse/HBASE-7349 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Attachments: 7349-build-improve-javadoc-warnings.0.diff > > > The javadoc check should look for an increase in the number of warnings. It > can do so by running javadoc against trunk before running it for the patch. > This will increase build times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7349) Jenkins build should compare trunk vs patch for Javadoc warnings
[ https://issues.apache.org/jira/browse/HBASE-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531864#comment-13531864 ] stack commented on HBASE-7349: -- Committed to trunk. Will leave issue open in case amendment needed. > Jenkins build should compare trunk vs patch for Javadoc warnings > > > Key: HBASE-7349 > URL: https://issues.apache.org/jira/browse/HBASE-7349 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Attachments: 7349-build-improve-javadoc-warnings.0.diff > > > The javadoc check should look for an increase in the number of warnings. It > can do so by running javadoc against trunk before running it for the patch. > This will increase build times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531871#comment-13531871
]
Ted Yu commented on HBASE-7352:
---
bq. something failed internally but was allowed to continue to do the dir
rename at some point.
This is not desirable. Renaming directory should only take place after all
regions are copied.
> clone operation from HBaseAdmin can hang forever.
> -
>
> Key: HBASE-7352
> URL: https://issues.apache.org/jira/browse/HBASE-7352
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0
>
>
> Sometimes the clone operation from the hbase shell can hang. The table has
> been created (it shows up in the web ui), but does not have any entries in
> META.
> There don't seem to be any clone, snapshot, enable or disable found in the
> master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x7f782800d000 nid=0x25c waiting on condition
> [0x7f782f9bf000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
> at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> ... (more jruby stack) ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7350) Flakey tests make CI unreliable
[
https://issues.apache.org/jira/browse/HBASE-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531872#comment-13531872
]
Hadoop QA commented on HBASE-7350:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12560873/7350-build-warn-on-terminate.0.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
104 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 23 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3537//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3537//console
This message is automatically generated.
> Flakey tests make CI unreliable
> ---
>
> Key: HBASE-7350
> URL: https://issues.apache.org/jira/browse/HBASE-7350
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: Nick Dimiduk
> Attachments: 7350-build-reduce-parallelism.0.diff,
> 7350-build-warn-on-terminate.0.diff
>
>
> Most PreCommit and PostCommit builds are failing these days. Keeping an eye
> on usual suspects reveals them to be mostly longer-running tests. Either the
> tests need to me made more rigorous or the erroneous Jenkins configuration
> needs addressed.
> The usual suspects:
> - org.apache.hadoop.hbase.client.TestMultiParallel
> - org.apache.hadoop.hbase.TestDrainingServer
> - org.apache.hadoop.hbase.regionserver.TestSplitTransaction
> - org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsNoAbort
> - org.apache.hadoop.hbase.replication.TestReplication
> - org.apache.hadoop.hbase.util.TestHBaseFsck
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7205) Coprocessor classloader is replicated for all regions in the HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7205: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Coprocessor classloader is replicated for all regions in the HRegionServer > -- > > Key: HBASE-7205 > URL: https://issues.apache.org/jira/browse/HBASE-7205 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.92.2, 0.94.2 >Reporter: Adrian Muraru >Assignee: Ted Yu >Priority: Critical > Fix For: 0.96.0, 0.94.4 > > Attachments: 7205-0.94.txt, 7205-v10.txt, 7205-v1.txt, 7205-v3.txt, > 7205-v4.txt, 7205-v5.txt, 7205-v6.txt, 7205-v7.txt, 7205-v8.txt, 7205-v9.txt, > HBASE-7205_v2.patch > > > HBASE-6308 introduced a new custom CoprocessorClassLoader to load the > coprocessor classes and a new instance of this CL is created for each single > HRegion opened. This leads to OOME-PermGen when the number of regions go > above hundres / region server. > Having the table coprocessor jailed in a separate classloader is good however > we should create only one for all regions of a table in each HRS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[
https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531878#comment-13531878
]
Hadoop QA commented on HBASE-7268:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12560876/HBASE-7268-v2-plus-masterTs.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3538//console
This message is automatically generated.
> correct local region location cache information can be overwritten w/stale
> information from an old server
> -
>
> Key: HBASE-7268
> URL: https://issues.apache.org/jira/browse/HBASE-7268
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-7268-v0.patch, HBASE-7268-v0.patch,
> HBASE-7268-v1.patch, HBASE-7268-v2.patch, HBASE-7268-v2-plus-masterTs.patch
>
>
> Discovered via HBASE-7250; related to HBASE-5877.
> Test is writing from multiple threads.
> Server A has region R; client knows that.
> R gets moved from A to server B.
> B gets killed.
> R gets moved by master to server C.
> ~15 seconds later, client tries to write to it (on A?).
> Multiple client threads report from RegionMoved exception processing logic "R
> moved from C to B", even though such transition never happened (neither in
> nor before the sequence described below). Not quite sure how the client
> learned of the transition to C, I assume it's from meta from some other
> thread...
> Then, put fails (it may fail due to accumulated errors that are not logged,
> which I am investigating... but the bogus cache update is there
> nonwithstanding).
> I have a patch but not sure if it works, test still fails locally for yet
> unknown reason.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7353) [shell] have list and list_snapshot return jruby string arrays.
Jonathan Hsieh created HBASE-7353:
-
Summary: [shell] have list and list_snapshot return jruby string
arrays.
Key: HBASE-7353
URL: https://issues.apache.org/jira/browse/HBASE-7353
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Hsieh
It is really convenient to allow comamnds like list and list_snapshots return a
jruby array of values in the hbase shell.
it allows for nice things like this:
{code}
# drop all tables starting with foo
list("foo.*").map { |t| disable t; drop t }
{code}
or
{code}
# clone all tables that start with bar
list_snapshots("bar.*").map { |s| clone_snapshot s, s + "-table"}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7353) [shell] have list and list_snapshot return jruby string arrays.
[
https://issues.apache.org/jira/browse/HBASE-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7353:
--
Component/s: shell
Affects Version/s: hbase-6055
> [shell] have list and list_snapshot return jruby string arrays.
> ---
>
> Key: HBASE-7353
> URL: https://issues.apache.org/jira/browse/HBASE-7353
> Project: HBase
> Issue Type: Sub-task
> Components: Client, shell
>Affects Versions: hbase-6055
>Reporter: Jonathan Hsieh
>
> It is really convenient to allow comamnds like list and list_snapshots return
> a jruby array of values in the hbase shell.
> it allows for nice things like this:
> {code}
> # drop all tables starting with foo
> list("foo.*").map { |t| disable t; drop t }
> {code}
> or
> {code}
> # clone all tables that start with bar
> list_snapshots("bar.*").map { |s| clone_snapshot s, s + "-table"}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7353) [shell] have list and list_snapshot return jruby string arrays.
[
https://issues.apache.org/jira/browse/HBASE-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7353:
--
Attachment: hbase-7353.patch
> [shell] have list and list_snapshot return jruby string arrays.
> ---
>
> Key: HBASE-7353
> URL: https://issues.apache.org/jira/browse/HBASE-7353
> Project: HBase
> Issue Type: Sub-task
> Components: Client, shell
>Affects Versions: hbase-6055
>Reporter: Jonathan Hsieh
> Attachments: hbase-7353.patch
>
>
> It is really convenient to allow comamnds like list and list_snapshots return
> a jruby array of values in the hbase shell.
> it allows for nice things like this:
> {code}
> # drop all tables starting with foo
> list("foo.*").map { |t| disable t; drop t }
> {code}
> or
> {code}
> # clone all tables that start with bar
> list_snapshots("bar.*").map { |s| clone_snapshot s, s + "-table"}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[ https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7268: Attachment: HBASE-7268-v2-plus-masterTs.patch > correct local region location cache information can be overwritten w/stale > information from an old server > - > > Key: HBASE-7268 > URL: https://issues.apache.org/jira/browse/HBASE-7268 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7268-v0.patch, HBASE-7268-v0.patch, > HBASE-7268-v1.patch, HBASE-7268-v2.patch, HBASE-7268-v2-plus-masterTs.patch, > HBASE-7268-v2-plus-masterTs.patch > > > Discovered via HBASE-7250; related to HBASE-5877. > Test is writing from multiple threads. > Server A has region R; client knows that. > R gets moved from A to server B. > B gets killed. > R gets moved by master to server C. > ~15 seconds later, client tries to write to it (on A?). > Multiple client threads report from RegionMoved exception processing logic "R > moved from C to B", even though such transition never happened (neither in > nor before the sequence described below). Not quite sure how the client > learned of the transition to C, I assume it's from meta from some other > thread... > Then, put fails (it may fail due to accumulated errors that are not logged, > which I am investigating... but the bogus cache update is there > nonwithstanding). > I have a patch but not sure if it works, test still fails locally for yet > unknown reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7353) [shell] have list and list_snapshot return jruby string arrays.
[
https://issues.apache.org/jira/browse/HBASE-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7353:
--
Status: Patch Available (was: Open)
> [shell] have list and list_snapshot return jruby string arrays.
> ---
>
> Key: HBASE-7353
> URL: https://issues.apache.org/jira/browse/HBASE-7353
> Project: HBase
> Issue Type: Sub-task
> Components: Client, shell
>Affects Versions: hbase-6055
>Reporter: Jonathan Hsieh
> Attachments: hbase-7353.patch
>
>
> It is really convenient to allow comamnds like list and list_snapshots return
> a jruby array of values in the hbase shell.
> it allows for nice things like this:
> {code}
> # drop all tables starting with foo
> list("foo.*").map { |t| disable t; drop t }
> {code}
> or
> {code}
> # clone all tables that start with bar
> list_snapshots("bar.*").map { |s| clone_snapshot s, s + "-table"}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7353) [shell] have list and list_snapshot return jruby string arrays.
[
https://issues.apache.org/jira/browse/HBASE-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531883#comment-13531883
]
Jonathan Hsieh commented on HBASE-7353:
---
here's what it looks like (also, the commands in the descrition work).
{code}
hbase(main):011:0> list
TABLE
TestTable
pe-1-table
pe-10-table
pe-11-table2
pe-11-table3
pe-12-table
pe-13-table
pe-14-table
pe-15-table
pe-16-table
pe-17-table
pe-18-table
pe-19-table
pe-20-table
pe-21-table
pe-22-table
pe-23-table
pe-24-table
pe-25-table
pe-26-table
pe-27-table
pe-28-table
pe-29-table
23 row(s) in 0.0800 seconds
=> ["TestTable", "pe-1-table", "pe-10-table", "pe-11-table2", "pe-11-table3",
"pe-12-table", "pe-13-table", "pe-14-table", "pe-15-table", "pe-16-table",
"pe-17-table", "pe-18-table", "pe-19-table", "pe-20-table", "pe-21-table",
"pe-22-table", "pe-23-table", "pe-24-table", "pe-25-table", "pe-26-table",
"pe-27-table", "pe-28-table", "pe-29-table"]
hbase(main):012:0>
{code}
{code}
hbase(main):012:0> list_snapshots
SNAPSHOT TABLE + CREATION TIME
pe-1TestTable (Thu Dec 13 15:37:56 -0800 2012)
pe-10 TestTable (Thu Dec 13 15:38:33 -0800 2012)
pe-11 TestTable (Thu Dec 13 15:38:38 -0800 2012)
pe-12 TestTable (Thu Dec 13 15:38:43 -0800 2012)
pe-13 TestTable (Thu Dec 13 15:38:48 -0800 2012)
pe-14 TestTable (Thu Dec 13 15:38:53 -0800 2012)
pe-15 TestTable (Thu Dec 13 15:38:58 -0800 2012)
pe-16 TestTable (Thu Dec 13 15:39:03 -0800 2012)
pe-17 TestTable (Thu Dec 13 15:39:08 -0800 2012)
p
[jira] [Commented] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531894#comment-13531894
]
Jonathan Hsieh commented on HBASE-7352:
---
I agree. Hence I have filed this bug.
> clone operation from HBaseAdmin can hang forever.
> -
>
> Key: HBASE-7352
> URL: https://issues.apache.org/jira/browse/HBASE-7352
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0
>
>
> Sometimes the clone operation from the hbase shell can hang. The table has
> been created (it shows up in the web ui), but does not have any entries in
> META.
> There don't seem to be any clone, snapshot, enable or disable found in the
> master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x7f782800d000 nid=0x25c waiting on condition
> [0x7f782f9bf000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
> at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> ... (more jruby stack) ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531857#comment-13531857
]
Jonathan Hsieh edited comment on HBASE-7352 at 12/14/12 1:27 AM:
-
The snapshot can be cloned from another shell instance but must write to a
different name.
If you attempt to clone the snapshot (pe-11) to the same target table
(pe-11-table), you get:
{code}
hbase(main):006:0> clone_snapshot 'pe-11', 'pe-11-table'
ERROR: org.apache.hadoop.hbase.snapshot.exception.RestoreSnapshotException:
org.apache.hadoop.hbase.snapshot.exception.RestoreSnapshotExcept
ion: Couldn't clone the snapshot=name: "pe-11"
table: "TestTable"
creationTime: 1355441918484
type: FLUSH
version: 0
on table=pe-11-table
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:558)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.restoreSnapshot(SnapshotManager.java:597)
at
org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2528)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Server.call(ProtobufRpcEngine.java:356)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1816)
Caused by: org.apache.hadoop.hbase.TableExistsException: pe-11-table
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.(CreateTableHandler.java:96)
at
org.apache.hadoop.hbase.master.snapshot.CloneSnapshotHandler.(CloneSnapshotHandler.java:65)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:551)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:558)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.restoreSnapshot(SnapshotManager.java:597)
at
org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2528)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Server.call(ProtobufRpcEngine.java:356)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1816)
Caused by: org.apache.hadoop.hbase.TableExistsException: pe-11-table
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.(CreateTableHandler.java:96)
at
org.apache.hadoop.hbase.master.snapshot.CloneSnapshotHandler.(CloneSnapshotHandler.java:65)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:551)
... 7 more
{code}
The dir for seems to be present for table pe-11, but there seems to be a large
number of missing files.
In this particular case, the snapshot has 16 regions, while the failed attempt
to restore has 12 regions moved into real table position. This suggests that
something failed internally but was allowed to continue to do the dir rename at
some point.
If one removes the data bad data from the hdfs we still cannot clone to the
same pe-11-table name because there is some in memory state that blocks this.
(grammar fixes)
was (Author: jmhsieh):
The snapshot can be cloned from another shell instance but must write to a
different name.
If you attempt to clone the snapshot (pe-11) to the same target table
(pe-11-table), you get:
{code}
hbase(main):006:0> clone_snapshot 'pe-11', 'pe-11-table'
ERROR: org.apache.hadoop.hbase.snapshot.exception.RestoreSnapshotException:
org.apache.hadoop.hbase.snapshot.exception.RestoreSnapshotExcept
ion: Couldn't clone the snapshot=name: "pe-11"
table: "TestTable"
creationTime: 1355441918484
type: FLUSH
version: 0
on table=pe-11-table
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:558)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.restoreSnapshot(SnapshotManager.java:597)
at
org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2528)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Server.call(ProtobufRpcEngine.java:356)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1816)
Caused by: org.apache.hadoop.hbase.Table
[jira] [Commented] (HBASE-7353) [shell] have list and list_snapshot return jruby string arrays.
[
https://issues.apache.org/jira/browse/HBASE-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531903#comment-13531903
]
Hadoop QA commented on HBASE-7353:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12560893/hbase-7353.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3539//console
This message is automatically generated.
> [shell] have list and list_snapshot return jruby string arrays.
> ---
>
> Key: HBASE-7353
> URL: https://issues.apache.org/jira/browse/HBASE-7353
> Project: HBase
> Issue Type: Sub-task
> Components: Client, shell
>Affects Versions: hbase-6055
>Reporter: Jonathan Hsieh
> Attachments: hbase-7353.patch
>
>
> It is really convenient to allow comamnds like list and list_snapshots return
> a jruby array of values in the hbase shell.
> it allows for nice things like this:
> {code}
> # drop all tables starting with foo
> list("foo.*").map { |t| disable t; drop t }
> {code}
> or
> {code}
> # clone all tables that start with bar
> list_snapshots("bar.*").map { |s| clone_snapshot s, s + "-table"}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7349) Jenkins build should compare trunk vs patch for Javadoc warnings
[ https://issues.apache.org/jira/browse/HBASE-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531917#comment-13531917 ] Hudson commented on HBASE-7349: --- Integrated in HBase-TRUNK #3620 (See [https://builds.apache.org/job/HBase-TRUNK/3620/]) HBASE-7349 Jenkins build should compare trunk vs patch for Javadoc warnings (Revision 1421629) Result = FAILURE stack : Files : * /hbase/trunk/dev-support/test-patch.properties * /hbase/trunk/dev-support/test-patch.sh > Jenkins build should compare trunk vs patch for Javadoc warnings > > > Key: HBASE-7349 > URL: https://issues.apache.org/jira/browse/HBASE-7349 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Attachments: 7349-build-improve-javadoc-warnings.0.diff > > > The javadoc check should look for an increase in the number of warnings. It > can do so by running javadoc against trunk before running it for the patch. > This will increase build times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7350) Flakey tests make CI unreliable
[ https://issues.apache.org/jira/browse/HBASE-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531916#comment-13531916 ] Hudson commented on HBASE-7350: --- Integrated in HBase-TRUNK #3620 (See [https://builds.apache.org/job/HBase-TRUNK/3620/]) HBASE-7350 Flakey tests make CI unreliable (Revision 1421623) Result = FAILURE stack : Files : * /hbase/trunk/dev-support/test-patch.sh > Flakey tests make CI unreliable > --- > > Key: HBASE-7350 > URL: https://issues.apache.org/jira/browse/HBASE-7350 > Project: HBase > Issue Type: Bug > Components: build >Reporter: Nick Dimiduk > Attachments: 7350-build-reduce-parallelism.0.diff, > 7350-build-warn-on-terminate.0.diff > > > Most PreCommit and PostCommit builds are failing these days. Keeping an eye > on usual suspects reveals them to be mostly longer-running tests. Either the > tests need to me made more rigorous or the erroneous Jenkins configuration > needs addressed. > The usual suspects: > - org.apache.hadoop.hbase.client.TestMultiParallel > - org.apache.hadoop.hbase.TestDrainingServer > - org.apache.hadoop.hbase.regionserver.TestSplitTransaction > - org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsNoAbort > - org.apache.hadoop.hbase.replication.TestReplication > - org.apache.hadoop.hbase.util.TestHBaseFsck -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7315) Remove support for client-side RowLocks
[ https://issues.apache.org/jira/browse/HBASE-7315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-7315: -- Attachment: HBASE-7315-v4.patch > Remove support for client-side RowLocks > --- > > Key: HBASE-7315 > URL: https://issues.apache.org/jira/browse/HBASE-7315 > Project: HBase > Issue Type: Sub-task > Components: Transactions/MVCC >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 0.96.0 > > Attachments: HBASE-7315.patch, HBASE-7315-v2.patch, > HBASE-7315-v3.patch, HBASE-7315-v4.patch > > > See comments in HBASE-7263. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531923#comment-13531923 ] Sergey Shelukhin commented on HBASE-5416: - Hmm, on reruns, I cannot repro the gains above on small datasets. However I see difference in IO activity during slow and joined scanner :) > Improve performance of scans with some kind of filters. > --- > > Key: HBASE-5416 > URL: https://issues.apache.org/jira/browse/HBASE-5416 > Project: HBase > Issue Type: Improvement > Components: Filters, Performance, regionserver >Affects Versions: 0.90.4 >Reporter: Max Lapan >Assignee: Max Lapan > Fix For: 0.96.0 > > Attachments: 5416-Filtered_scans_v6.patch, 5416-v5.txt, 5416-v6.txt, > Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch, > Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch, > Filtered_scans_v7.patch > > > When the scan is performed, whole row is loaded into result list, after that > filter (if exists) is applied to detect that row is needed. > But when scan is performed on several CFs and filter checks only data from > the subset of these CFs, data from CFs, not checked by a filter is not needed > on a filter stage. Only when we decided to include current row. And in such > case we can significantly reduce amount of IO performed by a scan, by loading > only values, actually checked by a filter. > For example, we have two CFs: flags and snap. Flags is quite small (bunch of > megabytes) and is used to filter large entries from snap. Snap is very large > (10s of GB) and it is quite costly to scan it. If we needed only rows with > some flag specified, we use SingleColumnValueFilter to limit result to only > small subset of region. But current implementation is loading both CFs to > perform scan, when only small subset is needed. > Attached patch adds one routine to Filter interface to allow filter to > specify which CF is needed to it's operation. In HRegion, we separate all > scanners into two groups: needed for filter and the rest (joined). When new > row is considered, only needed data is loaded, filter applied, and only if > filter accepts the row, rest of data is loaded. At our data, this speeds up > such kind of scans 30-50 times. Also, this gives us the way to better > normalize the data into separate columns by optimizing the scans performed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[
https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531937#comment-13531937
]
Hadoop QA commented on HBASE-7268:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12560894/HBASE-7268-v2-plus-masterTs.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 21 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool generates 1 additional
warning messages with this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 22 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3540//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3540//console
This message is automatically generated.
> correct local region location cache information can be overwritten w/stale
> information from an old server
> -
>
> Key: HBASE-7268
> URL: https://issues.apache.org/jira/browse/HBASE-7268
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-7268-v0.patch, HBASE-7268-v0.patch,
> HBASE-7268-v1.patch, HBASE-7268-v2.patch, HBASE-7268-v2-plus-masterTs.patch,
> HBASE-7268-v2-plus-masterTs.patch
>
>
> Discovered via HBASE-7250; related to HBASE-5877.
> Test is writing from multiple threads.
> Server A has region R; client knows that.
> R gets moved from A to server B.
> B gets killed.
> R gets moved by master to server C.
> ~15 seconds later, client tries to write to it (on A?).
> Multiple client threads report from RegionMoved exception processing logic "R
> moved from C to B", even though such transition never happened (neither in
> nor before the sequence described below). Not quite sure how the client
> learned of the transition to C, I assume it's from meta from some other
> thread...
> Then, put fails (it may fail due to accumulated errors that are not logged,
> which I am investigating... but the bogus cache update is there
> nonwithstanding).
> I have a patch but not sure if it works, test still fails locally for yet
> unknown reason.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7342) Split operation without split key incorrectly finds the middle key in off-by-one error
[
https://issues.apache.org/jira/browse/HBASE-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531947#comment-13531947
]
ramkrishna.s.vasudevan commented on HBASE-7342:
---
@Stack
Sorry Stack.. I did not mean to -1 it..Actually we once faced exactly the same
issue and we decided on something which i dont remember now :(
> Split operation without split key incorrectly finds the middle key in
> off-by-one error
> --
>
> Key: HBASE-7342
> URL: https://issues.apache.org/jira/browse/HBASE-7342
> Project: HBase
> Issue Type: Bug
> Components: HFile, io
>Affects Versions: 0.94.1, 0.94.2, 0.94.3, 0.96.0
>Reporter: Aleksandr Shulman
>Assignee: Aleksandr Shulman
>Priority: Minor
> Fix For: 0.96.0, 0.94.4
>
> Attachments: HBASE-7342-v1.patch, HBASE-7342-v2.patch
>
>
> I took a deeper look into issues I was having using region splitting when
> specifying a region (but not a key for splitting).
> The midkey calculation is off by one and when there are 2 rows, will pick the
> 0th one. This causes the firstkey to be the same as midkey and the split will
> fail. Removing the -1 causes it work correctly, as per the test I've added.
> Looking into the code here is what goes on:
> 1. Split takes the largest storefile
> 2. It puts all the keys into a 2-dimensional array called blockKeys[][]. Key
> i resides as blockKeys[i]
> 3. Getting the middle root-level index should yield the key in the middle of
> the storefile
> 4. In step 3, we see that there is a possible erroneous (-1) to adjust for
> the 0-offset indexing.
> 5. In a result with where there are only 2 blockKeys, this yields the 0th
> block key.
> 6. Unfortunately, this is the same block key that 'firstKey' will be.
> 7. This yields the result in HStore.java:1873 ("cannot split because midkey
> is the same as first or last row")
> 8. Removing the -1 solves the problem (in this case).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-5416: Attachment: HBASE-5416-v7-rebased.patch Ok, here's the rebased patch, mvn test appears to pass at least small tests, rest are running now. Can you please sanity check? I modified the perf test to run within normal test time and to run scan 10 times (at least on my laptop most of the time is taken by load, due to compactions probably). When it settles after 1-2 iterations, I see perf improvement, about 20-25% on small dataset. Tomorrow I will probably look at config/more tests. > Improve performance of scans with some kind of filters. > --- > > Key: HBASE-5416 > URL: https://issues.apache.org/jira/browse/HBASE-5416 > Project: HBase > Issue Type: Improvement > Components: Filters, Performance, regionserver >Affects Versions: 0.90.4 >Reporter: Max Lapan >Assignee: Max Lapan > Fix For: 0.96.0 > > Attachments: 5416-Filtered_scans_v6.patch, 5416-v5.txt, 5416-v6.txt, > Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch, > Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch, > Filtered_scans_v7.patch, HBASE-5416-v7-rebased.patch > > > When the scan is performed, whole row is loaded into result list, after that > filter (if exists) is applied to detect that row is needed. > But when scan is performed on several CFs and filter checks only data from > the subset of these CFs, data from CFs, not checked by a filter is not needed > on a filter stage. Only when we decided to include current row. And in such > case we can significantly reduce amount of IO performed by a scan, by loading > only values, actually checked by a filter. > For example, we have two CFs: flags and snap. Flags is quite small (bunch of > megabytes) and is used to filter large entries from snap. Snap is very large > (10s of GB) and it is quite costly to scan it. If we needed only rows with > some flag specified, we use SingleColumnValueFilter to limit result to only > small subset of region. But current implementation is loading both CFs to > perform scan, when only small subset is needed. > Attached patch adds one routine to Filter interface to allow filter to > specify which CF is needed to it's operation. In HRegion, we separate all > scanners into two groups: needed for filter and the rest (joined). When new > row is considered, only needed data is loaded, filter applied, and only if > filter accepts the row, rest of data is loaded. At our data, this speeds up > such kind of scans 30-50 times. Also, this gives us the way to better > normalize the data into separate columns by optimizing the scans performed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7315) Remove support for client-side RowLocks
[
https://issues.apache.org/jira/browse/HBASE-7315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531962#comment-13531962
]
Hadoop QA commented on HBASE-7315:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12560905/HBASE-7315-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 88 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
additional warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 21 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3541//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3541//console
This message is automatically generated.
> Remove support for client-side RowLocks
> ---
>
> Key: HBASE-7315
> URL: https://issues.apache.org/jira/browse/HBASE-7315
> Project: HBase
> Issue Type: Sub-task
> Components: Transactions/MVCC
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Fix For: 0.96.0
>
> Attachments: HBASE-7315.patch, HBASE-7315-v2.patch,
> HBASE-7315-v3.patch, HBASE-7315-v4.patch
>
>
> See comments in HBASE-7263.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[
https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13532037#comment-13532037
]
Ted Yu commented on HBASE-5416:
---
Based on patch v7, I got the following result on MacBook:
{code}
grep 'scanner finished in' ../testJoinedScanners-output.txt
2012-12-13 20:09:26,809 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 112.792634 seconds, got 100 rows
2012-12-13 20:10:15,726 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 48.915989 seconds, got 100 rows
2012-12-13 20:10:33,006 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 17.280432 seconds, got 100 rows
2012-12-13 20:10:38,514 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 5.508207 seconds, got 100 rows
2012-12-13 20:10:51,095 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 12.580323 seconds, got 100 rows
2012-12-13 20:11:00,517 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 9.422024 seconds, got 100 rows
2012-12-13 20:11:22,650 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 22.132854 seconds, got 100 rows
2012-12-13 20:11:31,890 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 9.23955 seconds, got 100 rows
2012-12-13 20:11:34,421 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 2.531598 seconds, got 100 rows
2012-12-13 20:11:36,694 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 2.272578 seconds, got 100 rows
2012-12-13 20:11:39,197 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 2.502777 seconds, got 100 rows
2012-12-13 20:11:58,269 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 19.071438 seconds, got 100 rows
2012-12-13 20:12:01,043 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 2.774262 seconds, got 100 rows
2012-12-13 20:12:03,317 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 2.273745 seconds, got 100 rows
2012-12-13 20:12:05,981 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 2.664124 seconds, got 100 rows
2012-12-13 20:12:08,574 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 2.593234 seconds, got 100 rows
2012-12-13 20:12:11,130 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 2.555977 seconds, got 100 rows
2012-12-13 20:12:13,381 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 2.250275 seconds, got 100 rows
2012-12-13 20:12:15,721 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 2.340003 seconds, got 100 rows
2012-12-13 20:12:18,075 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 2.354218 seconds, got 100 rows
{code}
I am running the test on Linux.
Will take another look at the patch and test result tomorrow.
> Improve performance of scans with some kind of filters.
> ---
>
> Key: HBASE-5416
> URL: https://issues.apache.org/jira/browse/HBASE-5416
> Project: HBase
> Issue Type: Improvement
> Components: Filters, Performance, regionserver
>Affects Versions: 0.90.4
>Reporter: Max Lapan
>Assignee: Max Lapan
> Fix For: 0.96.0
>
> Attachments: 5416-Filtered_scans_v6.patch, 5416-v5.txt, 5416-v6.txt,
> Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch,
> Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch,
> Filtered_scans_v7.patch, HBASE-5416-v7-rebased.patch
>
>
> When the scan is performed, whole row is loaded into result list, after that
> filter (if exists) is applied to detect that row is needed.
> But when scan is performed on several CFs and filter checks only data from
> the subset of these CFs, data from CFs, not checked by a filter is not needed
> on a filter stage. Only when we decided to include current row. And in such
> case we can significantly reduce amount of IO performed by a scan, by loading
> only values, actually checked by a filter.
> For example, we have two CFs: flags and snap. Flags is quite small (bunch of
> megabytes) and is used to filter large entries from snap. Snap is very large
> (10s of GB) and it is quite costly to scan it. If we needed only rows with
> some flag specified, we use SingleColumnValueFilter to limit result to only
> small subset of region. But current implementation is loading both CFs to
> perform scan, when only small subset is needed.
> Attached patch adds one routine to Filter interface to allow filter to
> specify which CF is needed to it's operation. In HRegio
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[
https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13532040#comment-13532040
]
Ted Yu commented on HBASE-5416:
---
Here is test result from Linux:
{code}
grep 'scanner finished in' testJoinedScanners-output.txt
2012-12-13 20:28:36,780 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 29.421479079 seconds, got 100 rows
2012-12-13 20:28:47,617 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 10.836890451 seconds, got 100 rows
2012-12-13 20:28:58,637 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 11.019543361 seconds, got 100 rows
2012-12-13 20:29:07,865 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 9.227820454 seconds, got 100 rows
2012-12-13 20:29:17,690 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 9.824966218 seconds, got 100 rows
2012-12-13 20:29:26,317 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 8.626794601 seconds, got 100 rows
2012-12-13 20:29:36,288 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 9.97033987 seconds, got 100 rows
2012-12-13 20:29:45,033 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 8.745137076 seconds, got 100 rows
2012-12-13 20:29:55,023 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 9.989630848 seconds, got 100 rows
2012-12-13 20:30:03,416 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 8.392952897 seconds, got 100 rows
2012-12-13 20:30:12,267 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 8.850649054 seconds, got 100 rows
2012-12-13 20:30:20,985 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 8.718266736 seconds, got 100 rows
2012-12-13 20:30:30,108 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 9.122057799 seconds, got 100 rows
2012-12-13 20:30:38,669 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 8.561782079 seconds, got 100 rows
2012-12-13 20:30:47,898 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 9.228045508 seconds, got 100 rows
2012-12-13 20:30:57,057 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 9.158965127 seconds, got 100 rows
2012-12-13 20:31:07,428 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 10.370526135 seconds, got 100 rows
2012-12-13 20:31:16,586 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 9.157627332 seconds, got 100 rows
2012-12-13 20:31:25,612 INFO [main] regionserver.TestJoinedScanners(172): Slow
scanner finished in 9.026821302 seconds, got 100 rows
2012-12-13 20:31:34,553 INFO [main] regionserver.TestJoinedScanners(172):
Joined scanner finished in 8.93992941 seconds, got 100 rows
{code}
> Improve performance of scans with some kind of filters.
> ---
>
> Key: HBASE-5416
> URL: https://issues.apache.org/jira/browse/HBASE-5416
> Project: HBase
> Issue Type: Improvement
> Components: Filters, Performance, regionserver
>Affects Versions: 0.90.4
>Reporter: Max Lapan
>Assignee: Max Lapan
> Fix For: 0.96.0
>
> Attachments: 5416-Filtered_scans_v6.patch, 5416-v5.txt, 5416-v6.txt,
> Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch,
> Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch,
> Filtered_scans_v7.patch, HBASE-5416-v7-rebased.patch
>
>
> When the scan is performed, whole row is loaded into result list, after that
> filter (if exists) is applied to detect that row is needed.
> But when scan is performed on several CFs and filter checks only data from
> the subset of these CFs, data from CFs, not checked by a filter is not needed
> on a filter stage. Only when we decided to include current row. And in such
> case we can significantly reduce amount of IO performed by a scan, by loading
> only values, actually checked by a filter.
> For example, we have two CFs: flags and snap. Flags is quite small (bunch of
> megabytes) and is used to filter large entries from snap. Snap is very large
> (10s of GB) and it is quite costly to scan it. If we needed only rows with
> some flag specified, we use SingleColumnValueFilter to limit result to only
> small subset of region. But current implementation is loading both CFs to
> perform scan, when only small subset is needed.
> Attached patch adds one routine to Filter interface to allow filter to
> specify which CF is needed to it's operation. In HRegion, we separate all
> scanners into two groups: needed for filter an
[jira] [Commented] (HBASE-7337) SingleColumnValueFilter seems to get unavailble data
[
https://issues.apache.org/jira/browse/HBASE-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13532042#comment-13532042
]
Anoop Sam John commented on HBASE-7337:
---
bq. then i set latestVersionOnly to false, the result does no change.
latestVersionOnly in SCVF is not for determining what will get returned by the
Scan. This specifies which value of a row column to be checked in case of
multiple versions available for the cell. When latestVersionOnly = true, only
the latest version value will get checked and so in your case only if the
latest version value is "2" that row will get selected. When latestVersionOnly
=false, it will check all the versions and if any of the version value is "2"
that row will get selected.
BTW how is the Scan object? I guess you have set in Scan to get all the
versions.
SCVF will not decide how many versions of a cell to be send back to client. It
just checks the cell value with the value specified.
> SingleColumnValueFilter seems to get unavailble data
>
>
> Key: HBASE-7337
> URL: https://issues.apache.org/jira/browse/HBASE-7337
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.94.3, 0.96.0
> Environment: 0.94
>Reporter: Zhou wenjian
>Assignee: Zhou wenjian
> Fix For: 0.96.0, 0.94.4
>
>
> put multi versions of a row.
> r1 cf:q version:1 value:1
> r1 cf:q version:2 value:3
> r1 cf:q version:3 value:2
> the filter in scan is set as below:
> SingleColumnValueFilter valueF = new SingleColumnValueFilter(
> family,qualifier,CompareOp.EQUAL,new BinaryComparator(Bytes
> .toBytes("2")));
> then i found all of the three versions will be emmitted, then i set
> latestVersionOnly to false, the result does no change.
> public ReturnCode filterKeyValue(KeyValue keyValue) {
> // System.out.println("REMOVE KEY=" + keyValue.toString() + ", value=" +
> Bytes.toString(keyValue.getValue()));
> if (this.matchedColumn) {
> // We already found and matched the single column, all keys now pass
> return ReturnCode.INCLUDE;
> } else if (this.latestVersionOnly && this.foundColumn) {
> // We found but did not match the single column, skip to next row
> return ReturnCode.NEXT_ROW;
> }
> if (!keyValue.matchingColumn(this.columnFamily, this.columnQualifier)) {
> return ReturnCode.INCLUDE;
> }
> foundColumn = true;
> if (filterColumnValue(keyValue.getBuffer(),
> keyValue.getValueOffset(), keyValue.getValueLength())) {
> return this.latestVersionOnly? ReturnCode.NEXT_ROW: ReturnCode.INCLUDE;
> }
> this.matchedColumn = true;
> return ReturnCode.INCLUDE;
> }
> From the code above, it seeems that version 3 will be first emmited, and set
> matchedColumn to true, which leads the following version 2 and 1 emmited too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7342) Split operation without split key incorrectly finds the middle key in off-by-one error
[
https://issues.apache.org/jira/browse/HBASE-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13532043#comment-13532043
]
stack commented on HBASE-7342:
--
[~ram_krish] No problem boss. Thats interesting that you saw this below.
Would be good to know what you fellas decided? Good on you.
> Split operation without split key incorrectly finds the middle key in
> off-by-one error
> --
>
> Key: HBASE-7342
> URL: https://issues.apache.org/jira/browse/HBASE-7342
> Project: HBase
> Issue Type: Bug
> Components: HFile, io
>Affects Versions: 0.94.1, 0.94.2, 0.94.3, 0.96.0
>Reporter: Aleksandr Shulman
>Assignee: Aleksandr Shulman
>Priority: Minor
> Fix For: 0.96.0, 0.94.4
>
> Attachments: HBASE-7342-v1.patch, HBASE-7342-v2.patch
>
>
> I took a deeper look into issues I was having using region splitting when
> specifying a region (but not a key for splitting).
> The midkey calculation is off by one and when there are 2 rows, will pick the
> 0th one. This causes the firstkey to be the same as midkey and the split will
> fail. Removing the -1 causes it work correctly, as per the test I've added.
> Looking into the code here is what goes on:
> 1. Split takes the largest storefile
> 2. It puts all the keys into a 2-dimensional array called blockKeys[][]. Key
> i resides as blockKeys[i]
> 3. Getting the middle root-level index should yield the key in the middle of
> the storefile
> 4. In step 3, we see that there is a possible erroneous (-1) to adjust for
> the 0-offset indexing.
> 5. In a result with where there are only 2 blockKeys, this yields the 0th
> block key.
> 6. Unfortunately, this is the same block key that 'firstKey' will be.
> 7. This yields the result in HStore.java:1873 ("cannot split because midkey
> is the same as first or last row")
> 8. Removing the -1 solves the problem (in this case).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13532055#comment-13532055 ] Ted Yu commented on HBASE-5416: --- There is significant performance gain for Joined scanner in the first few iterations. The test generates data and then runs scans alternately using normal and joined scanners. I think we should test scenarios where write and scan operations interleave. > Improve performance of scans with some kind of filters. > --- > > Key: HBASE-5416 > URL: https://issues.apache.org/jira/browse/HBASE-5416 > Project: HBase > Issue Type: Improvement > Components: Filters, Performance, regionserver >Affects Versions: 0.90.4 >Reporter: Max Lapan >Assignee: Max Lapan > Fix For: 0.96.0 > > Attachments: 5416-Filtered_scans_v6.patch, 5416-v5.txt, 5416-v6.txt, > Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch, > Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch, > Filtered_scans_v7.patch, HBASE-5416-v7-rebased.patch > > > When the scan is performed, whole row is loaded into result list, after that > filter (if exists) is applied to detect that row is needed. > But when scan is performed on several CFs and filter checks only data from > the subset of these CFs, data from CFs, not checked by a filter is not needed > on a filter stage. Only when we decided to include current row. And in such > case we can significantly reduce amount of IO performed by a scan, by loading > only values, actually checked by a filter. > For example, we have two CFs: flags and snap. Flags is quite small (bunch of > megabytes) and is used to filter large entries from snap. Snap is very large > (10s of GB) and it is quite costly to scan it. If we needed only rows with > some flag specified, we use SingleColumnValueFilter to limit result to only > small subset of region. But current implementation is loading both CFs to > perform scan, when only small subset is needed. > Attached patch adds one routine to Filter interface to allow filter to > specify which CF is needed to it's operation. In HRegion, we separate all > scanners into two groups: needed for filter and the rest (joined). When new > row is considered, only needed data is loaded, filter applied, and only if > filter accepts the row, rest of data is loaded. At our data, this speeds up > such kind of scans 30-50 times. Also, this gives us the way to better > normalize the data into separate columns by optimizing the scans performed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7337) SingleColumnValueFilter seems to get unavailble data
[
https://issues.apache.org/jira/browse/HBASE-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13532066#comment-13532066
]
Zhou wenjian commented on HBASE-7337:
-
[~anoopsamjohn]
actually i did get all the versions.
when using SCVF, it is not allowed to set version in the scan for it will get
data againset SCVF,do you mean it?
> SingleColumnValueFilter seems to get unavailble data
>
>
> Key: HBASE-7337
> URL: https://issues.apache.org/jira/browse/HBASE-7337
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.94.3, 0.96.0
> Environment: 0.94
>Reporter: Zhou wenjian
>Assignee: Zhou wenjian
> Fix For: 0.96.0, 0.94.4
>
>
> put multi versions of a row.
> r1 cf:q version:1 value:1
> r1 cf:q version:2 value:3
> r1 cf:q version:3 value:2
> the filter in scan is set as below:
> SingleColumnValueFilter valueF = new SingleColumnValueFilter(
> family,qualifier,CompareOp.EQUAL,new BinaryComparator(Bytes
> .toBytes("2")));
> then i found all of the three versions will be emmitted, then i set
> latestVersionOnly to false, the result does no change.
> public ReturnCode filterKeyValue(KeyValue keyValue) {
> // System.out.println("REMOVE KEY=" + keyValue.toString() + ", value=" +
> Bytes.toString(keyValue.getValue()));
> if (this.matchedColumn) {
> // We already found and matched the single column, all keys now pass
> return ReturnCode.INCLUDE;
> } else if (this.latestVersionOnly && this.foundColumn) {
> // We found but did not match the single column, skip to next row
> return ReturnCode.NEXT_ROW;
> }
> if (!keyValue.matchingColumn(this.columnFamily, this.columnQualifier)) {
> return ReturnCode.INCLUDE;
> }
> foundColumn = true;
> if (filterColumnValue(keyValue.getBuffer(),
> keyValue.getValueOffset(), keyValue.getValueLength())) {
> return this.latestVersionOnly? ReturnCode.NEXT_ROW: ReturnCode.INCLUDE;
> }
> this.matchedColumn = true;
> return ReturnCode.INCLUDE;
> }
> From the code above, it seeems that version 3 will be first emmited, and set
> matchedColumn to true, which leads the following version 2 and 1 emmited too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6788) Convert AuthenticationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13532069#comment-13532069 ] stack commented on HBASE-6788: -- I went over the patch again. I think I follow what is going on. This AuthenticationTokenIdentifier implements the Hadoop TokenIdentifier. Its a Writable and nought you can do about it, not at moment, so you are outputting pb when write and expecting pb when you readFrom. That seems to make sense given the constraint. Is this needed anymore? public static final long VERSION = 0L; Why do this Gary? +return AuthenticationProtos.AuthenticationService.newReflectiveService(this); Why can't we just return 'this'? Else patch looks good (again) > Convert AuthenticationProtocol to protocol buffer service > - > > Key: HBASE-6788 > URL: https://issues.apache.org/jira/browse/HBASE-6788 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6788.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. > AccessControllerProtocol was converted as part of HBASE-5448, but the > authentication token provider still needs to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7337) SingleColumnValueFilter seems to get unavailble data
[
https://issues.apache.org/jira/browse/HBASE-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13532082#comment-13532082
]
Anoop Sam John commented on HBASE-7337:
---
In Scan you can set how many version you need or you need all the versions or
only latest version.
This is irrespective of whether SCVF or any other filters are set on the Scan.
In fact the filter will get to know all the versions. (in filterKeyValue)
> SingleColumnValueFilter seems to get unavailble data
>
>
> Key: HBASE-7337
> URL: https://issues.apache.org/jira/browse/HBASE-7337
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.94.3, 0.96.0
> Environment: 0.94
>Reporter: Zhou wenjian
>Assignee: Zhou wenjian
> Fix For: 0.96.0, 0.94.4
>
>
> put multi versions of a row.
> r1 cf:q version:1 value:1
> r1 cf:q version:2 value:3
> r1 cf:q version:3 value:2
> the filter in scan is set as below:
> SingleColumnValueFilter valueF = new SingleColumnValueFilter(
> family,qualifier,CompareOp.EQUAL,new BinaryComparator(Bytes
> .toBytes("2")));
> then i found all of the three versions will be emmitted, then i set
> latestVersionOnly to false, the result does no change.
> public ReturnCode filterKeyValue(KeyValue keyValue) {
> // System.out.println("REMOVE KEY=" + keyValue.toString() + ", value=" +
> Bytes.toString(keyValue.getValue()));
> if (this.matchedColumn) {
> // We already found and matched the single column, all keys now pass
> return ReturnCode.INCLUDE;
> } else if (this.latestVersionOnly && this.foundColumn) {
> // We found but did not match the single column, skip to next row
> return ReturnCode.NEXT_ROW;
> }
> if (!keyValue.matchingColumn(this.columnFamily, this.columnQualifier)) {
> return ReturnCode.INCLUDE;
> }
> foundColumn = true;
> if (filterColumnValue(keyValue.getBuffer(),
> keyValue.getValueOffset(), keyValue.getValueLength())) {
> return this.latestVersionOnly? ReturnCode.NEXT_ROW: ReturnCode.INCLUDE;
> }
> this.matchedColumn = true;
> return ReturnCode.INCLUDE;
> }
> From the code above, it seeems that version 3 will be first emmited, and set
> matchedColumn to true, which leads the following version 2 and 1 emmited too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7337) SingleColumnValueFilter seems to get unavailble data
[
https://issues.apache.org/jira/browse/HBASE-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-7337:
--
Description:
put multi versions of a row.
r1 cf:q version:1 value:1
r1 cf:q version:2 value:3
r1 cf:q version:3 value:2
the filter in scan is set as below:
SingleColumnValueFilter valueF = new SingleColumnValueFilter(
family,qualifier,CompareOp.EQUAL,new BinaryComparator(Bytes
.toBytes("2")));
then i found all of the three versions will be emmitted, then i set
latestVersionOnly to false, the result does no change.
{code}
public ReturnCode filterKeyValue(KeyValue keyValue) {
// System.out.println("REMOVE KEY=" + keyValue.toString() + ", value=" +
Bytes.toString(keyValue.getValue()));
if (this.matchedColumn) {
// We already found and matched the single column, all keys now pass
return ReturnCode.INCLUDE;
} else if (this.latestVersionOnly && this.foundColumn) {
// We found but did not match the single column, skip to next row
return ReturnCode.NEXT_ROW;
}
if (!keyValue.matchingColumn(this.columnFamily, this.columnQualifier)) {
return ReturnCode.INCLUDE;
}
foundColumn = true;
if (filterColumnValue(keyValue.getBuffer(),
keyValue.getValueOffset(), keyValue.getValueLength())) {
return this.latestVersionOnly? ReturnCode.NEXT_ROW: ReturnCode.INCLUDE;
}
this.matchedColumn = true;
return ReturnCode.INCLUDE;
}
{code}
>From the code above, it seeems that version 3 will be first emmited, and set
>matchedColumn to true, which leads the following version 2 and 1 emmited too.
was:
put multi versions of a row.
r1 cf:q version:1 value:1
r1 cf:q version:2 value:3
r1 cf:q version:3 value:2
the filter in scan is set as below:
SingleColumnValueFilter valueF = new SingleColumnValueFilter(
family,qualifier,CompareOp.EQUAL,new BinaryComparator(Bytes
.toBytes("2")));
then i found all of the three versions will be emmitted, then i set
latestVersionOnly to false, the result does no change.
public ReturnCode filterKeyValue(KeyValue keyValue) {
// System.out.println("REMOVE KEY=" + keyValue.toString() + ", value=" +
Bytes.toString(keyValue.getValue()));
if (this.matchedColumn) {
// We already found and matched the single column, all keys now pass
return ReturnCode.INCLUDE;
} else if (this.latestVersionOnly && this.foundColumn) {
// We found but did not match the single column, skip to next row
return ReturnCode.NEXT_ROW;
}
if (!keyValue.matchingColumn(this.columnFamily, this.columnQualifier)) {
return ReturnCode.INCLUDE;
}
foundColumn = true;
if (filterColumnValue(keyValue.getBuffer(),
keyValue.getValueOffset(), keyValue.getValueLength())) {
return this.latestVersionOnly? ReturnCode.NEXT_ROW: ReturnCode.INCLUDE;
}
this.matchedColumn = true;
return ReturnCode.INCLUDE;
}
>From the code above, it seeems that version 3 will be first emmited, and set
>matchedColumn to true, which leads the following version 2 and 1 emmited too.
> SingleColumnValueFilter seems to get unavailble data
>
>
> Key: HBASE-7337
> URL: https://issues.apache.org/jira/browse/HBASE-7337
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.94.3, 0.96.0
> Environment: 0.94
>Reporter: Zhou wenjian
>Assignee: Zhou wenjian
> Fix For: 0.96.0, 0.94.4
>
>
> put multi versions of a row.
> r1 cf:q version:1 value:1
> r1 cf:q version:2 value:3
> r1 cf:q version:3 value:2
> the filter in scan is set as below:
> SingleColumnValueFilter valueF = new SingleColumnValueFilter(
> family,qualifier,CompareOp.EQUAL,new BinaryComparator(Bytes
> .toBytes("2")));
> then i found all of the three versions will be emmitted, then i set
> latestVersionOnly to false, the result does no change.
> {code}
> public ReturnCode filterKeyValue(KeyValue keyValue) {
> // System.out.println("REMOVE KEY=" + keyValue.toString() + ", value=" +
> Bytes.toString(keyValue.getValue()));
> if (this.matchedColumn) {
> // We already found and matched the single column, all keys now pass
> return ReturnCode.INCLUDE;
> } else if (this.latestVersionOnly && this.foundColumn) {
> // We found but did not match the single column, skip to next row
> return ReturnCode.NEXT_ROW;
> }
> if (!keyValue.matchingColumn(this.columnFamily, this.columnQualifier)) {
> return ReturnCode.INCLUDE;
> }
> foundColumn = true;
> if (filterColumnValue(keyValue.getBuffer(),
> keyValue.getValueOffset(), keyValue.getValueLength())) {
> return this.latestVersionOnly? ReturnCode.NEXT_ROW: ReturnCode.INCLUDE;
> }
> this.matchedColumn = true;
>
[jira] [Commented] (HBASE-6788) Convert AuthenticationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13532100#comment-13532100 ] Gary Helmling commented on HBASE-6788: -- [~saint@gmail.com] Thanks for the review. Sorry to update it again just after you've taken a look. bq. This AuthenticationTokenIdentifier implements the Hadoop TokenIdentifier. Yes, AuthenticationTokenIdentifier still needs to retain the Writable interface since Token (Hadoop class) uses this internally, and we rely on Token to tie in transparently to the Hadoop UGI credentials. bq. public static final long VERSION = 0L; Nice catch, no longer needed. bq. + return AuthenticationProtos.AuthenticationService.newReflectiveService(this); Since TokenProvider implements AuthenticationService.Interface, this translates that back into a Service instance. > Convert AuthenticationProtocol to protocol buffer service > - > > Key: HBASE-6788 > URL: https://issues.apache.org/jira/browse/HBASE-6788 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6788.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. > AccessControllerProtocol was converted as part of HBASE-5448, but the > authentication token provider still needs to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6788) Convert AuthenticationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-6788: - Status: Open (was: Patch Available) > Convert AuthenticationProtocol to protocol buffer service > - > > Key: HBASE-6788 > URL: https://issues.apache.org/jira/browse/HBASE-6788 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6788_2.patch, HBASE-6788.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. > AccessControllerProtocol was converted as part of HBASE-5448, but the > authentication token provider still needs to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6788) Convert AuthenticationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-6788: - Attachment: HBASE-6788_2.patch Updated patch addressing Nick's comment on required PB fields, Stack's comment on VERSION and adding a real test of token auth to TestTokenAuthentication. > Convert AuthenticationProtocol to protocol buffer service > - > > Key: HBASE-6788 > URL: https://issues.apache.org/jira/browse/HBASE-6788 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6788_2.patch, HBASE-6788.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. > AccessControllerProtocol was converted as part of HBASE-5448, but the > authentication token provider still needs to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6788) Convert AuthenticationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-6788: - Status: Patch Available (was: Open) > Convert AuthenticationProtocol to protocol buffer service > - > > Key: HBASE-6788 > URL: https://issues.apache.org/jira/browse/HBASE-6788 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6788_2.patch, HBASE-6788.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. > AccessControllerProtocol was converted as part of HBASE-5448, but the > authentication token provider still needs to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7247) Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening
[ https://issues.apache.org/jira/browse/HBASE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7247: - Status: Open (was: Patch Available) > Assignment performances decreased by 50% because of > regionserver.OpenRegionHandler#tickleOpening > > > Key: HBASE-7247 > URL: https://issues.apache.org/jira/browse/HBASE-7247 > Project: HBase > Issue Type: Improvement > Components: master, Region Assignment, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal >Priority: Critical > Fix For: 0.96.0 > > Attachments: 7247.v1.patch > > > The regionserver.OpenRegionHandler#tickleOpening updates the region znode as > "Do this so master doesn't timeout this region-in-transition.". > However, on the usual test, this makes the assignment time of 1500 regions > goes from 70s to 100s, that is, we're 50% slower because of this. > More generally, ZooKeper commits to disk all the data update, and this takes > time. Using it to provide a keep alive seems overkill. At the very list, it > could be made asynchronous. > I'm not sure how necessary these updates are required (I need to go deeper in > the internal, feedback welcome), but it seems very important to optimize > this... The trival fix would be to make this optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[
https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13532110#comment-13532110
]
Hadoop QA commented on HBASE-5416:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12560912/HBASE-5416-v7-rebased.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
additional warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 23 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3542//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3542//console
This message is automatically generated.
> Improve performance of scans with some kind of filters.
> ---
>
> Key: HBASE-5416
> URL: https://issues.apache.org/jira/browse/HBASE-5416
> Project: HBase
> Issue Type: Improvement
> Components: Filters, Performance, regionserver
>Affects Versions: 0.90.4
>Reporter: Max Lapan
>Assignee: Max Lapan
> Fix For: 0.96.0
>
> Attachments: 5416-Filtered_scans_v6.patch, 5416-v5.txt, 5416-v6.txt,
> Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch,
> Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch,
> Filtered_scans_v7.patch, HBASE-5416-v7-rebased.patch
>
>
> When the scan is performed, whole row is loaded into result list, after that
> filter (if exists) is applied to detect that row is needed.
> But when scan is performed on several CFs and filter checks only data from
> the subset of these CFs, data from CFs, not checked by a filter is not needed
> on a filter stage. Only when we decided to include current row. And in such
> case we can significantly reduce amount of IO performed by a scan, by loading
> only values, actually checked by a filter.
> For example, we have two CFs: flags and snap. Flags is quite small (bunch of
> megabytes) and is used to filter large entries from snap. Snap is very large
> (10s of GB) and it is quite costly to scan it. If we needed only rows with
> some flag specified, we use SingleColumnValueFilter to limit result to only
> small subset of region. But current implementation is loading both CFs to
> perform scan, when only small subset is needed.
> Attached patch adds one routine to Filter interface to allow filter to
> specify which CF is needed to it's operation. In HRegion, we separate all
> scanners into two groups: needed for filter and the rest (joined). When new
> row is considered, only needed data is loaded, filter applied, and only if
> filter accepts the row, rest of data is loaded. At our data, this speeds up
> such kind of scans 30-50 times. Also, this gives us the way to better
> normalize the data into separate columns by optimizing the scans performed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/sof
[jira] [Updated] (HBASE-6789) Convert test CoprocessorProtocol implementations to protocol buffer services
[ https://issues.apache.org/jira/browse/HBASE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6789: - Attachment: 6789.txt Here is a start converting over PingProtocol. About half done. Stuck at moment because specifying a row range that would seem to specify a single region only, the second of three, is returning results from the first region when I do table.coprocessorService(PingProtos.PingService.class, start, end, new Batch... I seem to be passing the proper rows... > Convert test CoprocessorProtocol implementations to protocol buffer services > > > Key: HBASE-6789 > URL: https://issues.apache.org/jira/browse/HBASE-6789 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling > Fix For: 0.96.0 > > Attachments: 6789.txt > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. > Several CoprocessorProtocol implementations are defined for tests: > * ColumnAggregationProtocol > * GenericProtocol > * TestServerCustomProtocol.PingProtocol > These should either be converted to PB services or removed if they duplicate > other tests/are no longer necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
