[jira] [Created] (HBASE-19950) Introduce a ColumnValue Filter

[Reid Chan (JIRA)]

Reid Chan created HBASE-19950:
-

 Summary: Introduce a ColumnValue Filter
 Key: HBASE-19950
 URL: https://issues.apache.org/jira/browse/HBASE-19950
 Project: HBase
  Issue Type: New Feature
  Components: Filters
Reporter: Reid Chan


Different from {{SingleColumnValueFilter}} which returns an entire row when 
specified condition is matched, this new filter will return the matched cell 
only. There're already some discussions in HBASE-19824.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-19927) TestFullLogReconstruction flakey

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-19927.
---
Resolution: Fixed

Reresolving. The addendum seems to have worked. No failures in last 24 hours 
(about 28 runs of the flakies suite). Good stuff.

> TestFullLogReconstruction flakey
> 
>
> Key: HBASE-19927
> URL: https://issues.apache.org/jira/browse/HBASE-19927
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Reporter: stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19927-addendum.patch, HBASE-19927.patch, js, out
>
>
> Fails pretty frequently in hadoopqa builds.
> There is a recent hang in 
> org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68)
> In here... 
> https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/
> ... see here.
> Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0):
>   State: TIMED_WAITING
>   Blocked count: 92
>   Waited count: 278
>   Stack:
> java.lang.Object.wait(Native Method)
> 
> org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133)
> 
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718)
> 
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605)
> 
> org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154)
> 
> org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319)
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531)
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437)
> 
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> We missed a signal? We need to do an interrupt? The log is not all there in 
> hadoopqa builds so hard to see all that is going on. This test is not in the 
> flakey set either



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

QuotaExceededException as a DoNotRetryIOException?

[Huaxiang Sun]

Hi HBase devs,

I found that QuotaExceededException  is a DoNotRetryIOException, which is a 
bit strange from user’s point of view.
For rpc throttling, the exception is retryable and it tells app to slow 
down and retry later.

Any thoughts?

Thanks,
Huaxiang

CFP for Dataworks Summit, San Jose, 2018

[Devaraj Das]

All, Dataworks Summit San Jose 2018 is June 17-21. The call for  abstracts is 
open through February 9th. Tracks like Datawarehousing and Operational Data 
Store might be a good fit for HBase & Phoenix talks. You can submit an abstract 
at https://dataworkssummit.com/san-jose-2018/​​
Thanks,
Devaraj.

[jira] [Created] (HBASE-19949) TestRSGroupsWithACL fails with ExceptionInInitializerError

[Ted Yu (JIRA)]

Ted Yu created HBASE-19949:
--

 Summary: TestRSGroupsWithACL fails with ExceptionInInitializerError
 Key: HBASE-19949
 URL: https://issues.apache.org/jira/browse/HBASE-19949
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu


{code}
org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL  Time elapsed: 0.168 sec  
<<< ERROR!
java.lang.ExceptionInInitializerError
at 
org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL.(TestRSGroupsWithACL.java:60)
{code}
The cause is that HBaseClassTestRule#getTimeoutInSeconds() cannot retrieve 
annotation for test size.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-19948) Since HBASE-19873, HBaseClassTestRule, Small/Medium/Large has different semantic

[stack (JIRA)]

stack created HBASE-19948:
-

 Summary: Since HBASE-19873, HBaseClassTestRule, Small/Medium/Large 
has different semantic
 Key: HBASE-19948
 URL: https://issues.apache.org/jira/browse/HBASE-19948
 Project: HBase
  Issue Type: Bug
Reporter: stack


I was confused on how SmallTest/MediumTest/LargeTest were being interpreted 
since HBASE-19873 where we added HBaseClassTestRule enforcing a ClassRule.

Small/Medium/Large are defined up in the refguide here: 
http://hbase.apache.org/book.html#hbase.unittests

E.g: "Small test cases are executed in a shared JVM and individual test cases 
should run in 15 seconds or less..."

I've always read the above as each method in a test suite/class should take 15 
seconds (see below for finding by [~appy] [1]).

The old CategoryBasedTimeout annotation used to try and enforce a test method 
taking only its designated category amount of time.

The above meant that you could have as many tests as you wanted in a 
class/suite and it could run as along as you liked as along as each individual 
test stayed within its category-based elapsed amount of time (and the whole 
suite completed inside the surefire fork timeout of 15mins).

Then came HBASE-19873 which addressed an awkward issue around accounting for 
time spent in startup/shutdown -- i.e. time taken outside of a test method run 
-- and trying to have a timeout that cuts in before the surefire fork one does. 
It ended up adding a ClassRule that set a timeout on the whole test 
*suite/class* -- Good -- but the timeout set varies dependent upon the test 
category. A suite/class with 60 small tests that each take a second to complete 
now times out if you add one more test to the suite (61 seconds > 60 seconds 
timeout -- give or take vagaries of the platform you run the test on).

This latter change I have trouble with. It changes how small/medium/large have 
classically been understood. I think it will confuse too as now devs must do 
careful counting of test methods per class; one fat one (i.e. 'large') is same 
as N small ones. Could we set a single timeout on the whole test suite/class, 
one that was well less than the surefire fork kill timeout of 900seconds but 
keep the old timeout on each method as we used to have with the category-based 
annotation?

(Am just looking for agreement that we have a problem here and that we want 
categories to be per test method as it used be; how to do it doesn't look easy 
and is for later).

1. @appy pointed out that the actual SmallTest annotation says something other 
than what is in the refguide: "Tag a test as 'small', meaning that the test 
class has the following characteristics: ideally, last less than 15 
seconds" 
https://github.com/apache/hbase/blob/master/hbase-annotations/src/test/java/org/apache/hadoop/hbase/testclassification/SmallTests.java#L22

2. Here is code to show how timeout has changed now... previous the below would 
have 'run' without timing out.

@Category({SmallTests.class})
public class TestTimingOut {
  @ClassRule
  public static final HBaseClassTestRule CLASS_RULE =
  HBaseClassTestRule.forClass(TestTimingOut.class);

  @Test
  public void oneTest() {
Threads.sleep(14000);
  }

  @Test
  public void twoTest() {
Threads.sleep(14000);
  }

  @Test
  public void threeTest() {
Threads.sleep(14000);
  }

  @Test
  public void fourTest() {
Threads.sleep(14000);
  }

  @Test
  public void fiveTest() {
Threads.sleep(14000);
  }
}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-19947) MR jobs using ITU use wrong filesystem

[Mike Drob (JIRA)]

Mike Drob created HBASE-19947:
-

 Summary: MR jobs using ITU use wrong filesystem
 Key: HBASE-19947
 URL: https://issues.apache.org/jira/browse/HBASE-19947
 Project: HBase
  Issue Type: Task
  Components: integration tests
Reporter: stack
Assignee: Mike Drob
 Fix For: 2.0.0-beta-2


Discovered by [~stack] as a result of HBASE-19841

IntegrationTestUtil subclasses HBasteTestUtility which new sets local FS as the 
default. When ITU is run against a mini cluster we reset it to the newly 
created DFS, but when it runs against an already existing distributed cluster, 
we forget to point our conf at the right place.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-19841) Tests against hadoop3 fail with StreamLacksCapabilityException

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob resolved HBASE-19841.
---
Resolution: Fixed

> Tests against hadoop3 fail with StreamLacksCapabilityException
> --
>
> Key: HBASE-19841
> URL: https://issues.apache.org/jira/browse/HBASE-19841
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Mike Drob
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19841.007.patch, 19841.06.patch, 19841.v0.txt, 
> 19841.v1.txt, HBASE-19841.v10.patch, HBASE-19841.v11.patch, 
> HBASE-19841.v11.patch, HBASE-19841.v2.patch, HBASE-19841.v3.patch, 
> HBASE-19841.v4.patch, HBASE-19841.v5.patch, HBASE-19841.v7.patch, 
> HBASE-19841.v8.patch, HBASE-19841.v8.patch, HBASE-19841.v8.patch, 
> HBASE-19841.v9.patch
>
>
> The following can be observed running against hadoop3:
> {code}
> java.io.IOException: cannot get log writer
>   at 
> org.apache.hadoop.hbase.regionserver.TestCompactingMemStore.compactingSetUp(TestCompactingMemStore.java:107)
>   at 
> org.apache.hadoop.hbase.regionserver.TestCompactingMemStore.setUp(TestCompactingMemStore.java:89)
> Caused by: 
> org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: 
> hflush and hsync
>   at 
> org.apache.hadoop.hbase.regionserver.TestCompactingMemStore.compactingSetUp(TestCompactingMemStore.java:107)
>   at 
> org.apache.hadoop.hbase.regionserver.TestCompactingMemStore.setUp(TestCompactingMemStore.java:89)
> {code}
> This was due to hbase-server/src/test/resources/hbase-site.xml not being 
> picked up by Configuration object. Among the configs from this file, the 
> value for "hbase.unsafe.stream.capability.enforce" relaxes check for presence 
> of hflush and hsync. Without this config entry,  
> StreamLacksCapabilityException is thrown.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-19941) Flaky TestCreateTableProcedure times out in nightly, needs to LargeTests

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-19941.
---
Resolution: Fixed

Resolving. Seems to have done the trick -- i.e. removing explict timeouts and 
instead relying on categorization.

> Flaky TestCreateTableProcedure times out in nightly, needs to LargeTests
> 
>
> Key: HBASE-19941
> URL: https://issues.apache.org/jira/browse/HBASE-19941
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Affects Versions: 2.0.0-beta-1
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19941.branch-2.001.patch
>
>
> Currently its categorized as MediumTests but sometimes running all test in 
> this class take more than 180 seconds. Here is the comparison of runtimes 
> between local runs (on my dev machine) and in nightly runs:
> ||Test||Local (seconds)||Nihgtly (seconds)||
> |testSimpleCreateWithSplits|~1.5|~12|
> |testRollbackAndDoubleExecutionOnMobTable|~4.7|~21|
> |testSimpleCreate|~1.7|~11|
> |testRollbackAndDoubleExecution|~4.3|~18|
> |testMRegions|~26.4|Timed out after 90 seconds|
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-19946) TestPerColumnFamilyFlush and TestWalAndCompactingMemStoreFlush fail against h3

[Mike Drob (JIRA)]

Mike Drob created HBASE-19946:
-

 Summary: TestPerColumnFamilyFlush and 
TestWalAndCompactingMemStoreFlush fail against h3
 Key: HBASE-19946
 URL: https://issues.apache.org/jira/browse/HBASE-19946
 Project: HBase
  Issue Type: Task
  Components: test
Reporter: Mike Drob
Assignee: Mike Drob
 Fix For: 2.0.0-beta-2


These tests started failing against hadoop 3 after HBASE-19841



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Moving the stable pointer to 1.3.1

[Ted Yu]

+1 to what Mike proposed.

On Tue, Feb 6, 2018 at 9:51 AM, Mike Drob  wrote:

> If there is an imminent 1.3.2 coming, I'd rather see it go directly there
> instead of pausing at 1.3.1, but otherwise no major concerns.
>
> On Tue, Feb 6, 2018 at 11:45 AM, Andrew Purtell 
> wrote:
>
> > On the thread "​Considering branching for 1.5 and other branch-1 release
> > planning
> > ​" Francis Liu, Zach York, and Ashish Singhi are all in favor of moving
> the
> > stable pointer forward, based on production and scale testing. Testing
> with
> > 1.3 we have done at my place of employment also had a positive result:
> 1.3
> > survived aggressive tests where 1.1 and 1.2 did not (at the time).
> >
> > I believe we have arrived at a consensus, so intend to retarget the
> stable
> > pointer to the latest 1.3 release, 1.3.1.  After Francis runs the 1.3.2​
> > release, we will move it to 1.3.2.
> >
> >
> > Please write in with objections or concerns if you have them and I'll
> hold
> > off until the discussion is settled (or if there is an objection).
> >
> >
> > --
> > Best regards,
> > Andrew
> >
>

Re: Moving the stable pointer to 1.3.1

[Andrew Purtell]

Good point, we can do that.

On Tue, Feb 6, 2018 at 9:51 AM, Mike Drob  wrote:

> If there is an imminent 1.3.2 coming, I'd rather see it go directly there
> instead of pausing at 1.3.1, but otherwise no major concerns.
>
> On Tue, Feb 6, 2018 at 11:45 AM, Andrew Purtell 
> wrote:
>
> > On the thread "​Considering branching for 1.5 and other branch-1 release
> > planning
> > ​" Francis Liu, Zach York, and Ashish Singhi are all in favor of moving
> the
> > stable pointer forward, based on production and scale testing. Testing
> with
> > 1.3 we have done at my place of employment also had a positive result:
> 1.3
> > survived aggressive tests where 1.1 and 1.2 did not (at the time).
> >
> > I believe we have arrived at a consensus, so intend to retarget the
> stable
> > pointer to the latest 1.3 release, 1.3.1.  After Francis runs the 1.3.2​
> > release, we will move it to 1.3.2.
> >
> >
> > Please write in with objections or concerns if you have them and I'll
> hold
> > off until the discussion is settled (or if there is an objection).
> >
> >
> > --
> > Best regards,
> > Andrew
> >
>



-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: Moving the stable pointer to 1.3.1

[Mike Drob]

If there is an imminent 1.3.2 coming, I'd rather see it go directly there
instead of pausing at 1.3.1, but otherwise no major concerns.

On Tue, Feb 6, 2018 at 11:45 AM, Andrew Purtell  wrote:

> On the thread "​Considering branching for 1.5 and other branch-1 release
> planning
> ​" Francis Liu, Zach York, and Ashish Singhi are all in favor of moving the
> stable pointer forward, based on production and scale testing. Testing with
> 1.3 we have done at my place of employment also had a positive result: 1.3
> survived aggressive tests where 1.1 and 1.2 did not (at the time).
>
> I believe we have arrived at a consensus, so intend to retarget the stable
> pointer to the latest 1.3 release, 1.3.1.  After Francis runs the 1.3.2​
> release, we will move it to 1.3.2.
>
>
> Please write in with objections or concerns if you have them and I'll hold
> off until the discussion is settled (or if there is an objection).
>
>
> --
> Best regards,
> Andrew
>

Moving the stable pointer to 1.3.1

[Andrew Purtell]

On the thread "​Considering branching for 1.5 and other branch-1 release
planning
​" Francis Liu, Zach York, and Ashish Singhi are all in favor of moving the
stable pointer forward, based on production and scale testing. Testing with
1.3 we have done at my place of employment also had a positive result: 1.3
survived aggressive tests where 1.1 and 1.2 did not (at the time).

I believe we have arrived at a consensus, so intend to retarget the stable
pointer to the latest 1.3 release, 1.3.1.  After Francis runs the 1.3.2​
release, we will move it to 1.3.2.


Please write in with objections or concerns if you have them and I'll hold
off until the discussion is settled (or if there is an objection).


--
Best regards,
Andrew

[jira] [Created] (HBASE-19945) Separate tests of TestRSGroups into two classes

[Ted Yu (JIRA)]

Ted Yu created HBASE-19945:
--

 Summary: Separate tests of TestRSGroups into two classes
 Key: HBASE-19945
 URL: https://issues.apache.org/jira/browse/HBASE-19945
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu


TestRSGroups is annotated as LargeTests already, however, it still times out on 
Jenkins:

https://builds.apache.org/job/HBase-Trunk_matrix/4537/jdk=JDK%201.8%20(latest),label=(Hadoop%20&&%20!H5)/testReport/junit/org.apache.hadoop.hbase.rsgroup/TestRSGroups/org_apache_hadoop_hbase_rsgroup_TestRSGroups/
{code}
org.junit.runners.model.TestTimedOutException: test timed out after 180 seconds
{code}
The above is reproducible on Linux locally.

The tests from TestRSGroups should be separated into two classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Please invite me slack

[Stack]

On Thu, Feb 1, 2018 at 4:01 AM, Kang Minwoo  wrote:

> Hello, dev users.
>
> I want to join the slack channel.
> Please invite me slack.
>
> Best regards,
> Minwoo Kang
>

Looks like an invite has been sent?
S

[jira] [Created] (HBASE-19944) Fix timeout TestVisibilityLabelsWithCustomVisLabService

[Guanghao Zhang (JIRA)]

Guanghao Zhang created HBASE-19944:
--

 Summary: Fix timeout TestVisibilityLabelsWithCustomVisLabService
 Key: HBASE-19944
 URL: https://issues.apache.org/jira/browse/HBASE-19944
 Project: HBase
  Issue Type: Sub-task
Reporter: Guanghao Zhang


[https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1404/testReport/junit/org.apache.hadoop.hbase.security.visibility/TestVisibilityLabelsWithCustomVisLabService/testVisibilityLabelsOnRSRestart/]

 
h3. Error Message

test timed out after 6 milliseconds
h3. Stacktrace

org.junit.runners.model.TestTimedOutException: test timed out after 6 
milliseconds



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)