[jira] [Created] (HBASE-14509) Configurable sparse indexes?
Lars Hofhansl created HBASE-14509: - Summary: Configurable sparse indexes? Key: HBASE-14509 URL: https://issues.apache.org/jira/browse/HBASE-14509 Project: HBase Issue Type: Brainstorming Reporter: Lars Hofhansl This idea just popped up today and I wanted to record it for discussion: What if we kept sparse column indexes per region or HFile or per configurable range? I.e. For any given CQ we record the lowest and highest value for a particular range (HFile, Region, or a custom range like the Phoenix guide post). By tweaking the size of these ranges we can control the size of the index, vs its selectivity. For example if we kept it by HFile we can almost instantly decide whether we need scan a particular HFile at all to find a particular value in a Cell. We can also collect min/max values for each n MB of data, for example when we can the region the first time. Assuming ranges are large enough we can always keep the index in memory together with the region. Kind of a sparse local index. Might much easier than the buddy region stuff we've been discussing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Thinking of HBase 1.1.3 [was HBASE-14317 and 1.1.3]
Status update. A patch for HBASE-14474 was committed, but the issue has been reopened as verification is ongoing. HBASE-14475 looks very close. On HBASE-14394, I reviewed Jerry's addendum; it satisfies the compatibility report and has my +1. All other issues I've kicked out of 1.1.3 release. If your ticket is close (can commit in the next 48 hours) and I've removed it, please let me know and we'll bring it back in. Thanks, Nick On Wed, Sep 23, 2015 at 6:01 PM, Enis Söztutarwrote: > Agreed that we should not change the declared interface for TRR in patch > releases. Ugly, but we can rethrow as RuntimeException or ignore in 1.1 and > before. > > I think this is also a blocker: > https://issues.apache.org/jira/browse/HBASE-14474 > > Enis > > On Wed, Sep 23, 2015 at 3:50 PM, Nick Dimiduk wrote: > >> I've run the compatibility checking tool [0] between branch-1.1 >> (0bf97bac2ed564994a0bcda5f1993260bf0b448f) and 1.1.0 >> (e860c66d41ddc8231004b646098a58abca7fb523). There has been a little bit of >> drift, but nothing that I think is release-blocking. However, I'd like to >> bring it to your attention here, before it sinks an RC. You can compare >> this to the run between 1.1.0 and 1.1.2RC2, which became 1.1.2 [1]. Notice >> we've added a handful of methods, which is acceptable according to our >> guidelines [2].The question I have is about adding throws IOException >> to TableRecordReader.close(). IOException is in the interface declaration >> of the super type, but this will require a source code change for anyone >> consuming our type directly. I believe, according to [2], this breaks our >> guidelines for a patch release. >> >> I've also sent a note over to HBASE-14394 [3] regarding the added public >> and undocumented method to TableRecordReader, so there's potentially two >> addendum's required for this patch. >> >> How would the community like to proceed? >> >> [0]: >> http://people.apache.org/~ndimiduk/1.1.0_branch-1.1_compat_report.html >> [1]: http://people.apache.org/~ndimiduk/1.1.0_1.1.2RC2_compat_report.html >> [2]: http://hbase.apache.org/book.html#hbase.versioning >> [3]: >> >> https://issues.apache.org/jira/browse/HBASE-14394?focusedCommentId=14905429=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14905429 >> >> On Mon, Sep 21, 2015 at 3:18 PM, Nick Dimiduk >> wrote: >> >> > Hi folks, >> > >> > It's that time again, I'm looking at spinning 1.1.3 bit this week, with >> > hopes that we can get a release out in early October. The only issue I'm >> > actively tracking as a must for this release is HBASE-14374, the back >> port >> > for HBASE-14317. Is there anything else you're planning to get in for >> this >> > one that's not been committed yet? Please speak up. I'll be starting my >> > pre-release validations tomorrow or Wednesday. >> > >> > Thanks, >> > Nick >> > >> > On Fri, Sep 4, 2015 at 4:08 PM, Andrew Purtell >> > wrote: >> > >> >> > PMC: do you have bandwidth to test yet another round of RC's? >> >> >> >> Yes, absolutely, and if you'd also like help making the RCs mail me >> >> privately. >> >> >> >> On Fri, Sep 4, 2015 at 8:11 AM, Nick Dimiduk >> wrote: >> >> >> >> > Hi folks, >> >> > >> >> > I know we just got through voting periods on three patch releases, >> but >> >> > HBASE-14317 is looking pretty bad by my eye. Given we have a fix on >> our >> >> > end, I'm up for spinning 1.1.3 a couple weeks early. How does the >> >> community >> >> > feel about it? Users: do you need this patch immediately? PMC: do you >> >> have >> >> > bandwidth to test yet another round of RC's? I'm not on JIRA yet this >> >> > morning; is there other nastiness we should get fixed in an >> accelerated >> >> .3 >> >> > as well? >> >> > >> >> > Thanks for your thoughts and your time. >> >> > -n >> >> > >> >> >> >> >> >> >> >> -- >> >> Best regards, >> >> >> >>- Andy >> >> >> >> Problems worthy of attack prove their worth by hitting back. - Piet >> Hein >> >> (via Tom White) >> >> >> > >> > >> > >
[jira] [Created] (HBASE-14508) Hbase scan not returning all rows when setting heigher value in scan.setCaching(cacheRow)
Lav Mudgal created HBASE-14508: -- Summary: Hbase scan not returning all rows when setting heigher value in scan.setCaching(cacheRow) Key: HBASE-14508 URL: https://issues.apache.org/jira/browse/HBASE-14508 Project: HBase Issue Type: Bug Affects Versions: 1.0.0 Reporter: Lav Mudgal Scan s = new Scan(); s.addFamily(Bytes.toBytes("cf1")); s.setCaching(cacheRows); s.setCacheBlocks(false); s.setStartRow("30.0.2.2\01441756800\0"); s.setStopRow("30.0.2.3\01441756800\0"); ResultScanner scanner = table.getScanner(s); long rows = 0; try { for (Result rr = scanner.next(); rr != null; rr = scanner.next()) { rows++; } } finally { scanner.close(); } System.out.println("Total no of rows = " + rows); When I run above code with cacheRows = 100 or 1 it prints Total no of rows = 48 When I run above code with cacheRows = 10 it prints Total no of rows = 10090 cacheRows <= 10083 prints 48 cacheRows = 10084 prints 191595 cacheRows = 10085 prints 20169 cacheRows = 10086 prints 20170 cacheRows = 10087 prints 20171 cacheRows = 10088 prints 20172 cacheRows = 10089 prints 20173 cacheRows = 10090 prints 20174 cacheRows >= 10091 prints 10090 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14507) Disable TestDistributedLogSplitting#testWorkerAbort Its flakey with tenuous chance of success
[ https://issues.apache.org/jira/browse/HBASE-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-14507. --- Resolution: Fixed Fix Version/s: 2.0.0 Done. Was part of the patch that went for "HBASE-14495 TestHRegion#testFlushCacheWhileScanning goes zombie". Rather than pull the fix out of there, I just left it. Here is the change that disabled the unit test for TDLS: {code} diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java index 25a5f41..4a43bbd 100644 @@ -169,6 +169,7 @@ public class TestDistributedLogSplitting { 169 conf.setInt(HConstants.REGIONSERVER_INFO_PORT, -1); 169 conf.setInt(HConstants.REGIONSERVER_INFO_PORT, -1); 170 conf.setFloat(HConstants.LOAD_BALANCER_SLOP_KEY, (float) 100.0); // no load balancing 170 conf.setFloat(HConstants.LOAD_BALANCER_SLOP_KEY, (float) 100.0); // no load balancing 171 conf.setInt("hbase.regionserver.wal.max.splitters", 3); 171 conf.setInt("hbase.regionserver.wal.max.splitters", 3); 172 conf.setInt(HConstants.REGION_SERVER_HIGH_PRIORITY_HANDLER_COUNT, 10); 172 TEST_UTIL.shutdownMiniHBaseCluster(); 173 TEST_UTIL.shutdownMiniHBaseCluster(); 173 TEST_UTIL = new HBaseTestingUtility(conf); 174 TEST_UTIL = new HBaseTestingUtility(conf); 174 TEST_UTIL.setDFSCluster(dfsCluster);175 TEST_UTIL.setDFSCluster(dfsCluster); @@ -998,7 +999,7 @@ public class TestDistributedLogSplitting { 998* detects that the region server has aborted.999* detects that the region server has aborted. 999* @throws Exception 1000 * @throws Exception 1000 */ 1001 */ 1001 @Test (timeout=30) 1002 @Ignore ("Disabled because flakey") @Test (timeout=30) 1002 public void testWorkerAbort() throws Exception { 1003 public void testWorkerAbort() throws Exception { 1003LOG.info("testWorkerAbort");1004 LOG.info("testWorkerAbort"); 1004startCluster(3);1005startCluster(3); {code} > Disable TestDistributedLogSplitting#testWorkerAbort Its flakey with tenuous > chance of success > - > > Key: HBASE-14507 > URL: https://issues.apache.org/jira/browse/HBASE-14507 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > > HBASE-14506 is to reenable it. Fails on apache build and locally for me. See > log in HBASE-14506 for detail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14510) Can not set coprocessor from Shell after HBASE-14224
Yerui Sun created HBASE-14510: - Summary: Can not set coprocessor from Shell after HBASE-14224 Key: HBASE-14510 URL: https://issues.apache.org/jira/browse/HBASE-14510 Project: HBase Issue Type: Bug Affects Versions: 0.98.14, 2.0.0, 1.3.0 Reporter: Yerui Sun I was able to set coprocessors for table by shell normally, but it didn't worked now. Here's the shell output (omit really jar path and coprocessor classname): {code} HBase Shell; enter 'help' for list of supported commands. Type "exit" to leave the HBase Shell Version 1.3.0-SNAPSHOT, 642273bc2a5a415eba6f1592a439a6b2b53a70a9, Tue Sep 29 15:58:28 CST 2015 hbase(main):001:0> describe 'test' Table test is ENABLED test COLUMN FAMILIES DESCRIPTION {NAME => 'f', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', B LOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 1 row(s) in 0.4370 seconds hbase(main):002:0> alter 'test', 'coprocessor'=>'hdfs:///some.jar|com.somepackage.SomeObserver|1001' Updating all regions with the new schema... 1/1 regions updated. Done. 0 row(s) in 1.9760 seconds hbase(main):003:0> describe 'test' Table test is ENABLED test, {TABLE_ATTRIBUTES => {coprocessor$1 => '|hdfs:///some.jar|com.somepackage.SomeObserver|1001|1073741823|'} COLUMN FAMILIES DESCRIPTION {NAME => 'f', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', B LOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 1 row(s) in 0.0180 seconds {code} I checked the recent commits and found HBASE-14224 is related to. It's an important improvement, but made a mistake in alter() of admin.rb, line 587. As the value is coprocess spec string but not only class name, here should use htd.setCoprocessorWithSpec instead of htd.setCoprocessor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14512) Cache UGI groups
Elliott Clark created HBASE-14512: - Summary: Cache UGI groups Key: HBASE-14512 URL: https://issues.apache.org/jira/browse/HBASE-14512 Project: HBase Issue Type: Bug Components: Performance, security Affects Versions: 1.2.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 1.2.0, 1.3.0 Right now every call gets a new User object. We should keep the same user for the life of a connection. We should also cache the group names. However we can't cache the groups for forever as that would mean groups don't get refreshed every 5 mins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14513) TestBucketCache runs obnoxious 1k threads in a unit test
stack created HBASE-14513: - Summary: TestBucketCache runs obnoxious 1k threads in a unit test Key: HBASE-14513 URL: https://issues.apache.org/jira/browse/HBASE-14513 Project: HBase Issue Type: Sub-task Components: test Reporter: stack Assignee: stack Attachments: 14513.txt Slightly related to zombie stomping... i was unable to run unadorned on new hardware without getting OOME can't create native thread -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14513) TestBucketCache runs obnoxious 1k threads in a unit test
[ https://issues.apache.org/jira/browse/HBASE-14513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-14513. --- Resolution: Fixed Fix Version/s: 1.1.3 1.0.3 1.3.0 1.2.0 2.0.0 Pushed one liner that changes 1000 threads to 100 in a test to branch-1.0+ > TestBucketCache runs obnoxious 1k threads in a unit test > > > Key: HBASE-14513 > URL: https://issues.apache.org/jira/browse/HBASE-14513 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3 > > Attachments: 14513.txt > > > Slightly related to zombie stomping... i was unable to run unadorned on new > hardware without getting OOME can't create native thread -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14511) StoreFile.Writer Meta Plugin
Vladimir Rodionov created HBASE-14511: - Summary: StoreFile.Writer Meta Plugin Key: HBASE-14511 URL: https://issues.apache.org/jira/browse/HBASE-14511 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov During my work on a new compaction policies (HBASE-14468, HBASE-14477) I had to modify the existing code of a StoreFile.Writer to add additional meta-info required by these new policies. I think that it should be done by means of a new Plugin framework, because this seems to be a general capability/feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14518) Give TestScanEarlyTermination the same treatment as 'HBASE-14378 Get TestAccessController* passing again...' -- up priority handlers
[ https://issues.apache.org/jira/browse/HBASE-14518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-14518. --- Resolution: Fixed Fix Version/s: 1.3.0 1.2.0 2.0.0 Pushed one-liner that sets priority handlers up to 10 from default 3. Here is how the hang looks: {code} "main" prio=10 tid=0x7fa02400a800 nid=0x519f in Object.wait() [0x7fa02c921000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148) - locked <0x0007c778e390> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212) at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1253) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1159) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1130) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1114) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:935) at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83) at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:79) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95) at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(CoprocessorRpcChannel.java:73) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService$BlockingStub.grant(AccessControlProtos.java:10280) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.grant(ProtobufUtil.java:2209) at org.apache.hadoop.hbase.security.access.SecureTestUtil$8.call(SecureTestUtil.java:510) at org.apache.hadoop.hbase.security.access.SecureTestUtil$8.call(SecureTestUtil.java:502) at org.apache.hadoop.hbase.security.access.SecureTestUtil.updateACLs(SecureTestUtil.java:332) at org.apache.hadoop.hbase.security.access.SecureTestUtil.grantOnTable(SecureTestUtil.java:502) at org.apache.hadoop.hbase.security.access.TestScanEarlyTermination.testEarlyScanTermination(TestScanEarlyTermination.java:148) {code} It happened here a few hours ago: https://builds.apache.org/job/PreCommit-HBASE-Build/15815/consoleText > Give TestScanEarlyTermination the same treatment as 'HBASE-14378 Get > TestAccessController* passing again...' -- up priority handlers > > > Key: HBASE-14518 > URL: https://issues.apache.org/jira/browse/HBASE-14518 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14518.txt > > > Up the priority handlers. I see a bunch of clients trying to scan hbase:meta > and then a bunch of occupied priority handlers -- though some seem open. Let > me give this same treatment as was done for other big tests upping priority > handlers to see if that helps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14514) Vulnerability to XSS attack due to printing HTML output
Ted Yu created HBASE-14514: -- Summary: Vulnerability to XSS attack due to printing HTML output Key: HBASE-14514 URL: https://issues.apache.org/jira/browse/HBASE-14514 Project: HBase Issue Type: Bug Reporter: Ted Yu In flink-clients/src/main/java/org/apache/flink/client/web/PlanDisplayServlet.java : {code} 113 writer.println("// register the event handler for the 'run' button and activate zoom Buttons\n" 114 + " activateZoomButtons();" 115 + " $('#run_button').click(function () {\n" + " $('#run_button').remove();\n" 116 + " $.ajax( {" + " url: '/runJob'," + " data: { action: 'runsubmitted', id: '" + uid + "' }," 117 + " success: function () { alert('Job succesfully submitted');" 118 + (this.runtimeVisURL != null ? (" window.location = \"" + this.runtimeVisURL + "\"; },") : " },") {code} Printing HTML output induces XSS vulnerability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14515) Allow spark module unit tests to be skipped with a profile
Sean Busbey created HBASE-14515: --- Summary: Allow spark module unit tests to be skipped with a profile Key: HBASE-14515 URL: https://issues.apache.org/jira/browse/HBASE-14515 Project: HBase Issue Type: Improvement Components: build, spark Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Fix For: 2.0.0 we can skip the tests for individual modules with profile, e.g. skipServerTests, we should have the same for spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[DISCUSS] getting postcommit tests back to green.
Hi Folks! Recently our postcommit tests have been in the pits. Essentially we've been broken since mid to late August[1]. While Stack's been fighting the good fight, I think things are far enough gone that we need to consider more drastic measures. I've been playing with getting things going on Travis-CI. It might be a good stop-gap measure while we wait for ASF Infra to get some additional testing resources. I believe Spark uses it already. It won't help with precommit unless we shift to github PRs (which I'd prefer not to do). Anyone have any other ideas or concerns? [1] ref: https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/ https://builds.apache.org/job/HBase-1.3/ -- Sean
[jira] [Created] (HBASE-14517) Show regionserver's version in master status page
Liu Shaohui created HBASE-14517: --- Summary: Show regionserver's version in master status page Key: HBASE-14517 URL: https://issues.apache.org/jira/browse/HBASE-14517 Project: HBase Issue Type: Improvement Components: monitoring Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor In production env, regionservers may be removed from the cluster for hardware problems and rejoined the cluster after the repair. There is a potential risk that the version of rejoined regionserver may diff from others because the cluster has been upgraded through many versions. To solve this, we can show the all regionservers' version in the server list of master's status page, and highlight the regionserver when its version is different from the master's version, similar to HDFS-3245 Suggestions are welcome~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14518) Give TestScanEarlyTermination the same treatment as 'HBASE-14378 Get TestAccessController* passing again...' -- up priority handlers
stack created HBASE-14518: - Summary: Give TestScanEarlyTermination the same treatment as 'HBASE-14378 Get TestAccessController* passing again...' -- up priority handlers Key: HBASE-14518 URL: https://issues.apache.org/jira/browse/HBASE-14518 Project: HBase Issue Type: Bug Components: test Reporter: stack Assignee: stack Up the priority handlers. I see a bunch of clients trying to scan hbase:meta and then a bunch of occupied priority handlers -- though some seem open. Let me give this same treatment as was done for other big tests upping priority handlers to see if that helps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14516) categorize hadoop-compat tests
Sean Busbey created HBASE-14516: --- Summary: categorize hadoop-compat tests Key: HBASE-14516 URL: https://issues.apache.org/jira/browse/HBASE-14516 Project: HBase Issue Type: Task Components: build, hadoop2, test Reporter: Sean Busbey Assignee: Sean Busbey Priority: Critical the hadoop-compat and hadoop2-compat modules do not rely on the hbase annotations test-jar and their tests aren't categorized. this causes things to fail if you attempt to specify one of our test categories to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)