[jira] [Created] (HBASE-18250) Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }

2017-06-20 Thread vishal (JIRA) 
vishal created HBASE-18250:
--

 Summary: Failed to move working directory snapshot { 
ss=_UPGRADING_TABLE_SYSTEM.MUTEX table=SYSTEM.MUTEX type=DISABLED }
 Key: HBASE-18250
 URL: https://issues.apache.org/jira/browse/HBASE-18250
 Project: HBase
  Issue Type: Bug
 Environment: HBase version : 1.2.6
Phoenix Version : 4.10.0-HBase-1.2.0
Reporter: vishal


While creating schema or table I am getting this error:
Failed to move working directory snapshot { ss=_UPGRADING_TABLE_SYSTEM.MUTEX 
table=SYSTEM.MUTEX type=DISABLED }
So because of this error it is not creating the schema or table.

java-program:
--
{code:java}
Properties connectionProps = new Properties();
connectionProps.put("phoenix.schema.isNamespaceMappingEnabled", "true");
connectionProps.put("phoenix.schema.mapSystemTablesToNamespace", "true");
connection = 
DriverManager.getConnection("jdbc:phoenix:localhost",connectionProps);
statement = connection.createStatement();
statement.executeUpdate("CREATE SCHEMA MYSCHEMA");
{code}

hdfs-site.xml

{code:java}
 
phoenix.schema.isNamespaceMappingEnabled
true
   

phoenix.schema.mapSystemTablesToNamespace
true
   
{code}

please help me.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [RESULT] [VOTE] First hbase-2.0.0-alpha-1 Release Candidate is available

2017-06-20 Thread Phil Yang 
Or we can just rename 2.0.0 version to alpha-1 and delete the current
alpha-1 and move issues under old alpha-1 to new one?

Thanks,
Phil


2017-06-21 11:20 GMT+08:00 Phil Yang :

> In JIRA there are many resolved issues whose fix version is 2.0.0, they
> are "released" when 2.0.0-alpha-1 released, do we need change them to
> 2.0.0-alpha-1? 2.0.0 should be a formal version after alpha/beta?
>
> Thanks,
> Phil
>
>
> 2017-06-21 1:00 GMT+08:00 Stack :
>
>> Thanks Josh.
>>
>> Sean noticed that I'd not actually pushed the alpha to our release dir.
>> Just did that.
>>
>> St.Ack
>>
>> On Tue, Jun 20, 2017 at 9:58 AM, Josh Elser  wrote:
>>
>> > Done ;)
>> >
>> > There were two issues still tagged as alpha-1 (JIRA didn't want to show
>> me
>> > them before performing the action), but I bumped them to alpha-2.
>> >
>> >
>> > On 6/20/17 12:19 PM, Mike Drob wrote:
>> >
>> >> Hi Stack,
>> >>
>> >> Can you mark jira version 2.0.0-alpha-1 as released?
>> >>
>> >> Thanks,
>> >> Mike
>> >>
>> >> On Sat, Jun 10, 2017 at 10:55 PM, Stack  wrote:
>> >>
>> >> With 4 binding votes and 1 non-binding, vote passes. Let me push out
>> the
>> >>> alpha.
>> >>> Thanks to all who voted.
>> >>> St.Ack
>> >>>
>> >>>
>> >>> On Sat, Jun 10, 2017 at 11:40 AM, Stack  wrote:
>> >>>
>> >>> Great. Thanks Sean. In-line...
>> 
>>  On Fri, Jun 9, 2017 at 11:35 PM, Sean Busbey 
>> wrote:
>> 
>>  +1
>> >
>> > details below, a few things to clean up for later alpha/betas.
>> >
>> > * checksums are all good
>> > * signatures are made with two different keys. should clean up by
>> next
>> > alpha.
>> >
>> > src / bin artifacts are signed with 8ACC93D2, as you mentioned in
>> the
>> > VOTE. However, this key is not in our project's KEYS file.
>> >
>> >
>> > Yes. Will fix.
>> 
>> 
>>  maven staged repository are signed with 30CD0996, which is in our
>> KEYS
>> > file so those check out fine.
>> >
>> > * the tag 2.0.0-alpha-1RC0 does point to
>> > c830a0f47f58d4892dd3300032c8244d6278aecc, which matches the source
>> > artifact after accounting HBASE-13088
>> >
>> > The tag isn't signed; would be good to make sure we do tag signing
>> in
>> > betas so that it's ready to go come GA time.
>> >
>> >
>> > Next time through, I'll update our 'How to RC' doc and will add
>> above.
>> 
>> 
>> 
>>  (side note, it would be good to get a decision on HBASE-13088 for the
>> > beta releases. been over 2 years)
>> >
>> > * the source tarball creates a binary assembly that looks as close
>> to
>> > the posted binary artifact as I've seen branch-1 do.
>> >
>> > idle interest, but our binary convenience artifact has jumped from
>> > ~100MiB in branch-1 release to ~160MiB. If anyone has time to dig in
>> > on why, probably worth checking. (e.g. the docs directory is ~585
>> MiB
>> > when untared.)
>> >
>> >
>> > Yes. Its obnoxious (HBASE-18208).
>> 
>> 
>>  * shaded artifact binaries are a reasonable number of MiB in size
>> > (incorrect build flags will result in ~empty jars)
>> >
>> > * LICENSE/NOTICE spot check looks okay.
>> >
>> > we have a ton of places where we have velocity variables instead of
>> > copyright years, but IIRC that's a problem on branch-1 right now
>> too.
>> >
>> > * CHANGES file hasn't been updated correctly
>> >
>> > currently has details for 0.93.
>> >
>> >
>> > Will fix next time through.
>> 
>>  Thanks Sean,
>>  S
>> 
>> 
>> 
>>  On Fri, Jun 9, 2017 at 12:06 PM, Andrew Purtell > >
>> > wrote:
>> >
>> >> +1
>> >>
>> >> On Thu, Jun 8, 2017 at 12:33 AM, Stack  wrote:
>> >>
>> >> The first release candidate for HBase 2.0.0-alpha-1 is up at:
>> >>>
>> >>>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-
>> >>>
>> >> alpha-1RC0/
>> >>>
>> 
>> >>> Maven artifacts are available from a staging directory here:
>> >>>
>> >>>   https://repository.apache.org/content/repositories/orgapache
>> >>>
>> >> hbase-1169
>> >
>> >>
>> >>> All was signed w/ my key 8ACC93D2
>> >>> 
>> >>>
>> >>> I tagged the RC as 2.0.0-alpha-1RC0
>> >>> (c830a0f47f58d4892dd3300032c8244d6278aecc).
>> >>>
>> >>> hbase-2.0.0-alpha-1 will be our first 2.0.0 release. It is a rough
>> >>>
>> >> cut
>> >>>
>>  ('alpha') not-for-production preview of what hbase-2.0.0 will look
>> >>>
>> >> like. It
>> >
>> >> is what we used to call a 'Developer' release[1] meant mostly for
>> >>>
>> >> devs
>> >>>
>>  and
>> >
>> >> downstreamers to test drive and flag us early if we there are
>> issues
>> >>>
>> >> we’ve
>> >
>> >> missed ahead of our rolling a production-worthy release.
>> >>>

Re: [RESULT] [VOTE] First hbase-2.0.0-alpha-1 Release Candidate is available

2017-06-20 Thread Phil Yang 
In JIRA there are many resolved issues whose fix version is 2.0.0, they are
"released" when 2.0.0-alpha-1 released, do we need change them to
2.0.0-alpha-1? 2.0.0 should be a formal version after alpha/beta?

Thanks,
Phil


2017-06-21 1:00 GMT+08:00 Stack :

> Thanks Josh.
>
> Sean noticed that I'd not actually pushed the alpha to our release dir.
> Just did that.
>
> St.Ack
>
> On Tue, Jun 20, 2017 at 9:58 AM, Josh Elser  wrote:
>
> > Done ;)
> >
> > There were two issues still tagged as alpha-1 (JIRA didn't want to show
> me
> > them before performing the action), but I bumped them to alpha-2.
> >
> >
> > On 6/20/17 12:19 PM, Mike Drob wrote:
> >
> >> Hi Stack,
> >>
> >> Can you mark jira version 2.0.0-alpha-1 as released?
> >>
> >> Thanks,
> >> Mike
> >>
> >> On Sat, Jun 10, 2017 at 10:55 PM, Stack  wrote:
> >>
> >> With 4 binding votes and 1 non-binding, vote passes. Let me push out the
> >>> alpha.
> >>> Thanks to all who voted.
> >>> St.Ack
> >>>
> >>>
> >>> On Sat, Jun 10, 2017 at 11:40 AM, Stack  wrote:
> >>>
> >>> Great. Thanks Sean. In-line...
> 
>  On Fri, Jun 9, 2017 at 11:35 PM, Sean Busbey 
> wrote:
> 
>  +1
> >
> > details below, a few things to clean up for later alpha/betas.
> >
> > * checksums are all good
> > * signatures are made with two different keys. should clean up by
> next
> > alpha.
> >
> > src / bin artifacts are signed with 8ACC93D2, as you mentioned in the
> > VOTE. However, this key is not in our project's KEYS file.
> >
> >
> > Yes. Will fix.
> 
> 
>  maven staged repository are signed with 30CD0996, which is in our KEYS
> > file so those check out fine.
> >
> > * the tag 2.0.0-alpha-1RC0 does point to
> > c830a0f47f58d4892dd3300032c8244d6278aecc, which matches the source
> > artifact after accounting HBASE-13088
> >
> > The tag isn't signed; would be good to make sure we do tag signing in
> > betas so that it's ready to go come GA time.
> >
> >
> > Next time through, I'll update our 'How to RC' doc and will add
> above.
> 
> 
> 
>  (side note, it would be good to get a decision on HBASE-13088 for the
> > beta releases. been over 2 years)
> >
> > * the source tarball creates a binary assembly that looks as close to
> > the posted binary artifact as I've seen branch-1 do.
> >
> > idle interest, but our binary convenience artifact has jumped from
> > ~100MiB in branch-1 release to ~160MiB. If anyone has time to dig in
> > on why, probably worth checking. (e.g. the docs directory is ~585 MiB
> > when untared.)
> >
> >
> > Yes. Its obnoxious (HBASE-18208).
> 
> 
>  * shaded artifact binaries are a reasonable number of MiB in size
> > (incorrect build flags will result in ~empty jars)
> >
> > * LICENSE/NOTICE spot check looks okay.
> >
> > we have a ton of places where we have velocity variables instead of
> > copyright years, but IIRC that's a problem on branch-1 right now too.
> >
> > * CHANGES file hasn't been updated correctly
> >
> > currently has details for 0.93.
> >
> >
> > Will fix next time through.
> 
>  Thanks Sean,
>  S
> 
> 
> 
>  On Fri, Jun 9, 2017 at 12:06 PM, Andrew Purtell 
> > wrote:
> >
> >> +1
> >>
> >> On Thu, Jun 8, 2017 at 12:33 AM, Stack  wrote:
> >>
> >> The first release candidate for HBase 2.0.0-alpha-1 is up at:
> >>>
> >>>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-
> >>>
> >> alpha-1RC0/
> >>>
> 
> >>> Maven artifacts are available from a staging directory here:
> >>>
> >>>   https://repository.apache.org/content/repositories/orgapache
> >>>
> >> hbase-1169
> >
> >>
> >>> All was signed w/ my key 8ACC93D2
> >>> 
> >>>
> >>> I tagged the RC as 2.0.0-alpha-1RC0
> >>> (c830a0f47f58d4892dd3300032c8244d6278aecc).
> >>>
> >>> hbase-2.0.0-alpha-1 will be our first 2.0.0 release. It is a rough
> >>>
> >> cut
> >>>
>  ('alpha') not-for-production preview of what hbase-2.0.0 will look
> >>>
> >> like. It
> >
> >> is what we used to call a 'Developer' release[1] meant mostly for
> >>>
> >> devs
> >>>
>  and
> >
> >> downstreamers to test drive and flag us early if we there are issues
> >>>
> >> we’ve
> >
> >> missed ahead of our rolling a production-worthy release.
> >>>
> >>> hbase-2.0.0 includes a fleet of new features that include a new
> >>>
> >> assignment
> >
> >> manager, means for keeping read and write path off-heap, in-memory
> >>> compactions, and more. I have been keeping a running doc on the
> state
> >>>
> >> of
> >
> >> 2.0.0 here [2]. There is much to do still (see aforementioned doc).

[jira] [Created] (HBASE-18249) Explicitly mark failed small test(s)

2017-06-20 Thread Ted Yu (JIRA) 
Ted Yu created HBASE-18249:
--

 Summary: Explicitly mark failed small test(s)
 Key: HBASE-18249
 URL: https://issues.apache.org/jira/browse/HBASE-18249
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu


In some cases, I would remind contributor to re-attach patch if failure of 
small test(s) prevented medium / large tests from running.
HBASE-18167 was a recent example.

We should explicitly mark failed small test(s) so that people know a big 
portion of unit tests haven't been run.
This would reduce potential test failure in Jenkins builds if committer doesn't 
pay attention (that failed test was small test).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-18210) Implement Table#checkAndDelete()

2017-06-20 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HBASE-18210.

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: HBASE-14850

Thanks for the review, Enis.

> Implement Table#checkAndDelete()
> 
>
> Key: HBASE-18210
> URL: https://issues.apache.org/jira/browse/HBASE-18210
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: HBASE-14850
>
> Attachments: 18210.v1.txt
>
>
> This issue is to implement Table#checkAndDelete() API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18248) Warn if monitored task has been tied up beyond a configurable threshold

2017-06-20 Thread Andrew Purtell (JIRA) 
Andrew Purtell created HBASE-18248:
--

 Summary: Warn if monitored task has been tied up beyond a 
configurable threshold
 Key: HBASE-18248
 URL: https://issues.apache.org/jira/browse/HBASE-18248
 Project: HBase
  Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 2.0.0, 3.0.0, 1.4.0, 1.3.2


Warn if monitored task has been tied up beyond a configurable threshold. We 
especially want to do this for RPC tasks. Use a separate threshold for warning 
about stuck RPC tasks versus other types of tasks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18247) Hbck to fix the case that replica region shows as key in the meta table

2017-06-20 Thread huaxiang sun (JIRA) 
huaxiang sun created HBASE-18247:


 Summary: Hbck to fix the case that replica region shows as key in 
the meta table
 Key: HBASE-18247
 URL: https://issues.apache.org/jira/browse/HBASE-18247
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0-alpha-1
Reporter: huaxiang sun
Assignee: huaxiang sun
Priority: Minor


Recently, we run into one case with read replica, the replica region shows up 
as key in meta table (it is not supposed to happen, we are still working on why 
it showed up in the meta table).

However, hbck always reported the error about the primary region. Please see 
the error attached.

{code}
test,92b0201b,1492546349354_0001.c3e6f235fe7caef75f8b0fb92a012da3. 
column=info:regioninfo, timestamp=1494958820573, value={ENCODED => 
c3e6f235fe7caef75f8b0fb92a012da3, NAME => 
'test,92b0201b,1492546349354_0001.c3e6f235fe7caef75f8b0fb92a012da3.', STARTKEY 
=> '92b0201b', ENDKEY => '92f1a952', REPLICA_ID => 1}

ERROR: Region { meta => 
test,92b0201b,1492546349354.d2c637715f31a072f174e70d407fb458., hdfs => null, 
deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on 
any region server.
{code}

Traced the code, in the following line, it does not consider the case that 
replicaId in regionInfo could be non-default. 

https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java#L985

If it is changed to get replicaId from regionInfo, then hbck should be able to 
fix this by "-fixMeta".

{code}
diff --git 
a/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java 
b/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
index 9eb5111..1649e53 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
@@ -982,7 +982,7 @@ public class MetaTableAccessor {
 List locations = new ArrayList<>(1);
 NavigableMap> familyMap = 
r.getNoVersionMap();
 
-locations.add(getRegionLocation(r, regionInfo, 0));
+locations.add(getRegionLocation(r, regionInfo, regionInfo.getReplicaId()));
 
 NavigableMap infoMap = familyMap.get(getCatalogFamily());
 if (infoMap == null) return new RegionLocations(locations);
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18246) Maintain Data locality in ServerCrashProcedure

2017-06-20 Thread Stephen Yuan Jiang (JIRA) 
Stephen Yuan Jiang created HBASE-18246:
--

 Summary: Maintain Data locality in ServerCrashProcedure
 Key: HBASE-18246
 URL: https://issues.apache.org/jira/browse/HBASE-18246
 Project: HBase
  Issue Type: Sub-task
  Components: Region Assignment
Affects Versions: 2.0.0
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang


Before HBASE-18036, SSH would use round-robin to re-distribute regions during 
processing.  Round-robin assignment would loss data locality.  HBASE-18036 
retains data locality if the dead region server has already restarted when the 
dead RS is processing.  

With Proc-V2 based AM, the change of HBASE-18036 in Apache HBASE 1.x releases 
is no longer possible.  We need to implement the same logic under Proc-V2 based 
AM.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-18205) Output multiple license failures in a single run

2017-06-20 Thread Mike Drob (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-18205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob resolved HBASE-18205.
---
Resolution: Duplicate

HBASE-16351 allows us to print multiple errors per run, so there will be no 
additional work here.

> Output multiple license failures in a single run
> 
>
> Key: HBASE-18205
> URL: https://issues.apache.org/jira/browse/HBASE-18205
> Project: HBase
>  Issue Type: Improvement
>  Components: dependencies
>Reporter: Mike Drob
>
> Currently, when we run into dependency licensing issues the build will fail 
> and output the culprit dependencies one at a time, after which we an fix it, 
> rerun the build, and then see the failure.
> It would be much more efficient if the build could inform us of multiple 
> failures in some way.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18245) Handle failed server in RpcClient

2017-06-20 Thread Ted Yu (JIRA) 
Ted Yu created HBASE-18245:
--

 Summary: Handle failed server in RpcClient
 Key: HBASE-18245
 URL: https://issues.apache.org/jira/browse/HBASE-18245
 Project: HBase
  Issue Type: Sub-task
Reporter: Ted Yu


This task is to add support for failed server in RpcClient#GetConnection().

FailedServers Java class would be ported to C++.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: HBaseCon Update (CfP closes Monday)

2017-06-20 Thread Stack 
Slides from last weeks hbasecon2017 are up and available here:
https://www.slideshare.net/search/slideshow?searchfrom=header&q=hbasecon2017

Enjoy,
HBaseCon Program Committee

On Thu, Apr 20, 2017 at 10:57 AM, Stack  wrote:

> A few things.
>
> 1. CfP closes Monday. Get those talks in [1]!
> 2. Registration is full; we are working on opening the 'waiting list' (if
> you are registered and do not think you can come, please give up your spot
> for another). Please check back on the hbasecon website in next day or so
> [2].
> 3. In case you did not realize, sponsorship (courtesy of our gracious
> hosts) is kinda cool in that you get your logo as sponsor of hbasecon by
> donating to the Apache Software Foundation (one lump of cash, two good
> deeds).
>
> Write me if I can help w/ questions on any of the above or anything else
> related to hbasecon.
>
> Thanks,
> St.Ack
> HBaseCon Secretary
>
> P.S. CfP for HBaseCon Asia is also up [3]. If you can't come out west, you
> can head east in August!
>
> 1. https://easychair.org/cfp/hbasecon2017
> 2. hbasecon.com
> 3. https://easychair.org/cfp/HBaseConAsia2017
>

[jira] [Created] (HBASE-18244) org.apache.hadoop.hbase.client.rsgroup.TestShellRSGroups hangs/fails

2017-06-20 Thread Josh Elser (JIRA) 
Josh Elser created HBASE-18244:
--

 Summary: org.apache.hadoop.hbase.client.rsgroup.TestShellRSGroups 
hangs/fails
 Key: HBASE-18244
 URL: https://issues.apache.org/jira/browse/HBASE-18244
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: Josh Elser
 Fix For: 3.0.0


Sometime in the past couple of weeks, TestShellRSGroups has started 
timing-out/failing for me.

It will get stuck on a call to moveTables()

{noformat}
"main" #1 prio=5 os_prio=31 tid=0x7ff012004800 nid=0x1703 in Object.wait() 
[0x7020d000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at 
org.apache.hadoop.hbase.ipc.BlockingRpcCallback.get(BlockingRpcCallback.java:62)
- locked <0x00078d1003f0> (a 
org.apache.hadoop.hbase.ipc.BlockingRpcCallback)
at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328)
at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:94)
at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:567)
at 
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.execMasterService(MasterProtos.java)
at 
org.apache.hadoop.hbase.client.ConnectionImplementation$3.execMasterService(ConnectionImplementation.java:1500)
at 
org.apache.hadoop.hbase.client.HBaseAdmin$67$1.rpcCall(HBaseAdmin.java:2991)
at 
org.apache.hadoop.hbase.client.HBaseAdmin$67$1.rpcCall(HBaseAdmin.java:2986)
at 
org.apache.hadoop.hbase.client.MasterCallable.call(MasterCallable.java:98)
at 
org.apache.hadoop.hbase.client.HBaseAdmin$67.callExecService(HBaseAdmin.java:2997)
at 
org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callBlockingMethod(SyncCoprocessorRpcChannel.java:69)
at 
org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService$BlockingStub.moveTables(RSGroupAdminProtos.java:13171)
at 
org.apache.hadoop.hbase.rsgroup.RSGroupAdminClient.moveTables(RSGroupAdminClient.java:117)
{noformat}

The server-side end of the RPC is waiting on a procedure to finish:

{noformat}
"RpcServer.default.FPBQ.Fifo.handler=27,queue=0,port=64242" #289 daemon prio=5 
os_prio=31 tid=0x7ff015b7c000 nid=0x1e603 waiting on condition 
[0x7dbc9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at 
org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitFor(ProcedureSyncWait.java:184)
at 
org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitFor(ProcedureSyncWait.java:171)
at 
org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToComplete(ProcedureSyncWait.java:141)
at 
org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.waitForProcedureToCompleteIOE(ProcedureSyncWait.java:130)
at 
org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.submitAndWaitProcedure(ProcedureSyncWait.java:123)
at 
org.apache.hadoop.hbase.master.assignment.AssignmentManager.unassign(AssignmentManager.java:478)
at 
org.apache.hadoop.hbase.master.assignment.AssignmentManager.unassign(AssignmentManager.java:465)
at 
org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveTables(RSGroupAdminServer.java:432)
at 
org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.moveTables(RSGroupAdminEndpoint.java:174)
at 
org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:12786)
at 
org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:673)
at 
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)

   Locked ownable synchronizers:
- None
{noformat}

I don't see anything else running in the thread dump, but I do see that meta 
was closed. Eventually the client (I think) gives up and retries, but then 
fails because meta isn't assigned.

{noformat}
2017-06-20 15:07:45,720 DEBUG [RS_CLOSE_META-hw10447:64242-0] 
assignment.RegionTransitionProcedure(200): Received report CLOSED seqId=-1, 
pid=12, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure 
table=hbase:meta, region=1588230740, server=hw10447.local,64242,1497985650605; 
rit=CLOSING

[jira] [Created] (HBASE-18243) HBase Thrift server lacks logic for renewing kerberos tickets

2017-06-20 Thread Steen Manniche (JIRA) 
Steen Manniche created HBASE-18243:
--

 Summary: HBase Thrift server lacks logic for renewing kerberos 
tickets
 Key: HBASE-18243
 URL: https://issues.apache.org/jira/browse/HBASE-18243
 Project: HBase
  Issue Type: Bug
  Components: Thrift
Affects Versions: 1.1.2, 2.0.0
Reporter: Steen Manniche
Priority: Minor


I have been looking through the hbase-thrift code looking for where
the server performs renewals of kerberos tickets for the provided
principal/keytab. There seems to be no logic in place for renewing tickets.

The hadoop-common provides the class
UserGroupInformation, which exposes the method
{{checkTGTAndReloginFromKeytab}}. I can see that the {{ThriftServerRunner}} 
class
has a handle to the class
(https://github.com/apache/hbase/blob/master/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java#L205),
but I do not see the ticket renewal logic being called anywhere.

A possible workaround is to renew the ticket outside the java process.

The documentation on the {{checkTGTAndReloginFromKeytab}} states that if the 
ticket is still valid, a call to the method is essentially a no-op.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] More Shading

2017-06-20 Thread Josh Elser 

On 6/20/17 1:28 AM, Stack wrote:

On Thu, Apr 13, 2017 at 4:46 PM, Josh Elser  wrote:


...

I think pushing this part forward with some code is the next logical step.
Seems to be consensus about taking our known internal dependencies and
performing this shade magic.



I opened HBASE-18240 "Add hbase-auxillary, a project with hbase utility
including an hbase-shaded-thirdparty module with guava, netty, etc."

It has a tarball attached that bundles the outline of an hbase-auxillary
project (groupId:org.apache.hbase.auxillary). This project is intended to
be standalone, in its own repository, publishing its own artifacts under
the aegis of this project's PMC.

It includes the first instance of an auxillary utility, a module named
hbase-thirdparty-shaded (artifactId:hbase-thirdparty-shaded). Herein we'll
pull down 3rd party libs and republish at an offset; e.g.
com.google.common.* from guava will be at
org.apache.hbase.thirdparty.shaded.com.google.common.*. Currently it builds
a jar that includes a relocated guava 22.0.

I then messed around making hbase-common use it (You have to build the
hbase-auxillary into your local repo). I put up a patch on the issue.
Mostly its mass find-and-replace w/ some clean up of transitive includes of
guava from hadoop-common and some small fixup of methods renamed between
guava 12.0 and 22.0.

Unless objection, I was going to press on. Sean offered to help set up new
repo. We can always undo and delete it if this project fails.

When done, the hope is we are on a modern version of guava and our netty
and protobuf 3 will be be relocated, 'hidden' from downstream (and won't
clash w/ upstream). I hope to also purge the pre-build we have in our
modules that do protobuf moving this hackery out and under
hbase-thirdparty-shaded.

St.Ack


Kudos on the JFDI approach :). I think having something concrete to show 
is the best way to judge success of it.


Will keep an eye on HBASE-18240.

Re: [ANNOUNCE] New HBase committer Huaxiang Sun

2017-06-20 Thread Gary Helmling 
Congratulations and welcome!

On Mon, Jun 19, 2017 at 11:07 PM Huaxiang Sun  wrote:

> Thanks all for the warm welcome, it is a great honor for me!
>
> Huaxiang
>
> > On Jun 19, 2017, at 9:27 PM, Anoop John  wrote:
> >
> > Congratulations Huaxiang
> >
> > -Anoop-
> >
> > On Tue, Jun 20, 2017 at 9:36 AM, ramkrishna vasudevan
> >  wrote:
> >> Congratulations !!!
> >>
> >> On Tue, Jun 20, 2017 at 9:32 AM, Yu Li  wrote:
> >>
> >>> Congratulations and welcome, Huaxiang!
> >>>
> >>> Best Regards,
> >>> Yu
> >>>
> >>> On 20 June 2017 at 11:03, Allan Yang  wrote:
> >>>
>  Congratulations and welcome, Huaxiang!
> 
>  Best Regards
>  Allan Yang
> 
>  2017-06-20 10:32 GMT+08:00 Pankaj kr :
> 
> > Congratulations Huaxiang..!!
> >
> > Thanks & Regards,
> > Pankaj
> >
> > HUAWEI TECHNOLOGIES CO.LTD.
> > Huawei Tecnologies India Pvt. Ltd.
> > Near EPIP Industrial Area, Kundalahalli Village
> > Whitefield, Bangalore-560066
> > www.huawei.com
> > 
> > -
> > This e-mail and its attachments contain confidential information from
> > HUAWEI, which
> > is intended only for the person or entity whose address is listed
> >>> above.
> > Any use of the
> > information contained herein in any way (including, but not limited
> to,
> > total or partial
> > disclosure, reproduction, or dissemination) by persons other than the
> > intended
> > recipient(s) is prohibited. If you receive this e-mail in error,
> please
> > notify the sender by
> > phone or email immediately and delete it!
> >
> >
> > -Original Message-
> > From: Sean Busbey [mailto:bus...@apache.org]
> > Sent: Tuesday, June 20, 2017 3:31 AM
> > To: dev; u...@hbase.apache.org
> > Subject: [ANNOUNCE] New HBase committer Huaxiang Sun
> >
> > On behalf of the Apache HBase PMC, I am pleased to announce that
> >>> Huaxiang
> > Sun has accepted the PMC's invitation to become a committer on the
>  project.
> > We appreciate all of Huaxiang's great work thus far and look forward
> to
> > continued involvement.
> >
> > Please join me in congratulating Huaxiang!
> >
> 
> >>>
>
>

Re: [RESULT] [VOTE] First hbase-2.0.0-alpha-1 Release Candidate is available

2017-06-20 Thread Stack 
Thanks Josh.

Sean noticed that I'd not actually pushed the alpha to our release dir.
Just did that.

St.Ack

On Tue, Jun 20, 2017 at 9:58 AM, Josh Elser  wrote:

> Done ;)
>
> There were two issues still tagged as alpha-1 (JIRA didn't want to show me
> them before performing the action), but I bumped them to alpha-2.
>
>
> On 6/20/17 12:19 PM, Mike Drob wrote:
>
>> Hi Stack,
>>
>> Can you mark jira version 2.0.0-alpha-1 as released?
>>
>> Thanks,
>> Mike
>>
>> On Sat, Jun 10, 2017 at 10:55 PM, Stack  wrote:
>>
>> With 4 binding votes and 1 non-binding, vote passes. Let me push out the
>>> alpha.
>>> Thanks to all who voted.
>>> St.Ack
>>>
>>>
>>> On Sat, Jun 10, 2017 at 11:40 AM, Stack  wrote:
>>>
>>> Great. Thanks Sean. In-line...

 On Fri, Jun 9, 2017 at 11:35 PM, Sean Busbey  wrote:

 +1
>
> details below, a few things to clean up for later alpha/betas.
>
> * checksums are all good
> * signatures are made with two different keys. should clean up by next
> alpha.
>
> src / bin artifacts are signed with 8ACC93D2, as you mentioned in the
> VOTE. However, this key is not in our project's KEYS file.
>
>
> Yes. Will fix.


 maven staged repository are signed with 30CD0996, which is in our KEYS
> file so those check out fine.
>
> * the tag 2.0.0-alpha-1RC0 does point to
> c830a0f47f58d4892dd3300032c8244d6278aecc, which matches the source
> artifact after accounting HBASE-13088
>
> The tag isn't signed; would be good to make sure we do tag signing in
> betas so that it's ready to go come GA time.
>
>
> Next time through, I'll update our 'How to RC' doc and will add above.



 (side note, it would be good to get a decision on HBASE-13088 for the
> beta releases. been over 2 years)
>
> * the source tarball creates a binary assembly that looks as close to
> the posted binary artifact as I've seen branch-1 do.
>
> idle interest, but our binary convenience artifact has jumped from
> ~100MiB in branch-1 release to ~160MiB. If anyone has time to dig in
> on why, probably worth checking. (e.g. the docs directory is ~585 MiB
> when untared.)
>
>
> Yes. Its obnoxious (HBASE-18208).


 * shaded artifact binaries are a reasonable number of MiB in size
> (incorrect build flags will result in ~empty jars)
>
> * LICENSE/NOTICE spot check looks okay.
>
> we have a ton of places where we have velocity variables instead of
> copyright years, but IIRC that's a problem on branch-1 right now too.
>
> * CHANGES file hasn't been updated correctly
>
> currently has details for 0.93.
>
>
> Will fix next time through.

 Thanks Sean,
 S



 On Fri, Jun 9, 2017 at 12:06 PM, Andrew Purtell 
> wrote:
>
>> +1
>>
>> On Thu, Jun 8, 2017 at 12:33 AM, Stack  wrote:
>>
>> The first release candidate for HBase 2.0.0-alpha-1 is up at:
>>>
>>>   https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-
>>>
>> alpha-1RC0/
>>>

>>> Maven artifacts are available from a staging directory here:
>>>
>>>   https://repository.apache.org/content/repositories/orgapache
>>>
>> hbase-1169
>
>>
>>> All was signed w/ my key 8ACC93D2
>>> 
>>>
>>> I tagged the RC as 2.0.0-alpha-1RC0
>>> (c830a0f47f58d4892dd3300032c8244d6278aecc).
>>>
>>> hbase-2.0.0-alpha-1 will be our first 2.0.0 release. It is a rough
>>>
>> cut
>>>
 ('alpha') not-for-production preview of what hbase-2.0.0 will look
>>>
>> like. It
>
>> is what we used to call a 'Developer' release[1] meant mostly for
>>>
>> devs
>>>
 and
>
>> downstreamers to test drive and flag us early if we there are issues
>>>
>> we’ve
>
>> missed ahead of our rolling a production-worthy release.
>>>
>>> hbase-2.0.0 includes a fleet of new features that include a new
>>>
>> assignment
>
>> manager, means for keeping read and write path off-heap, in-memory
>>> compactions, and more. I have been keeping a running doc on the state
>>>
>> of
>
>> 2.0.0 here [2]. There is much to do still (see aforementioned doc).
>>>
>>> The list of features addressed in 2.0.0 so far can be found here [4].
>>>
>> There
>
>> are about 2500. The list of ~500 fixes in 2.0.0 exclusively can be
>>>
>> found
>
>> here [3].
>>>
>>> Please take it for a spin and vote on whether it ok to put out as our
>>>
>> first
>
>> alpha (bar is low for an 'alpha'). Let the VOTE be open for 24 hours.
>>>
>>> Thanks,
>>> St.Ack
>>>
>>> 1. http://hbase.apache.org/book.html#hbase.versioning.pre10
>>> 2.
>>> https://docs.google.com/docum

Re: [RESULT] [VOTE] First hbase-2.0.0-alpha-1 Release Candidate is available

2017-06-20 Thread Josh Elser 

Done ;)

There were two issues still tagged as alpha-1 (JIRA didn't want to show 
me them before performing the action), but I bumped them to alpha-2.


On 6/20/17 12:19 PM, Mike Drob wrote:

Hi Stack,

Can you mark jira version 2.0.0-alpha-1 as released?

Thanks,
Mike

On Sat, Jun 10, 2017 at 10:55 PM, Stack  wrote:


With 4 binding votes and 1 non-binding, vote passes. Let me push out the
alpha.
Thanks to all who voted.
St.Ack


On Sat, Jun 10, 2017 at 11:40 AM, Stack  wrote:


Great. Thanks Sean. In-line...

On Fri, Jun 9, 2017 at 11:35 PM, Sean Busbey  wrote:


+1

details below, a few things to clean up for later alpha/betas.

* checksums are all good
* signatures are made with two different keys. should clean up by next
alpha.

src / bin artifacts are signed with 8ACC93D2, as you mentioned in the
VOTE. However, this key is not in our project's KEYS file.



Yes. Will fix.



maven staged repository are signed with 30CD0996, which is in our KEYS
file so those check out fine.

* the tag 2.0.0-alpha-1RC0 does point to
c830a0f47f58d4892dd3300032c8244d6278aecc, which matches the source
artifact after accounting HBASE-13088

The tag isn't signed; would be good to make sure we do tag signing in
betas so that it's ready to go come GA time.



Next time through, I'll update our 'How to RC' doc and will add above.




(side note, it would be good to get a decision on HBASE-13088 for the
beta releases. been over 2 years)

* the source tarball creates a binary assembly that looks as close to
the posted binary artifact as I've seen branch-1 do.

idle interest, but our binary convenience artifact has jumped from
~100MiB in branch-1 release to ~160MiB. If anyone has time to dig in
on why, probably worth checking. (e.g. the docs directory is ~585 MiB
when untared.)



Yes. Its obnoxious (HBASE-18208).



* shaded artifact binaries are a reasonable number of MiB in size
(incorrect build flags will result in ~empty jars)

* LICENSE/NOTICE spot check looks okay.

we have a ton of places where we have velocity variables instead of
copyright years, but IIRC that's a problem on branch-1 right now too.

* CHANGES file hasn't been updated correctly

currently has details for 0.93.



Will fix next time through.

Thanks Sean,
S




On Fri, Jun 9, 2017 at 12:06 PM, Andrew Purtell 
wrote:

+1

On Thu, Jun 8, 2017 at 12:33 AM, Stack  wrote:


The first release candidate for HBase 2.0.0-alpha-1 is up at:

  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-

alpha-1RC0/


Maven artifacts are available from a staging directory here:

  https://repository.apache.org/content/repositories/orgapache

hbase-1169


All was signed w/ my key 8ACC93D2


I tagged the RC as 2.0.0-alpha-1RC0
(c830a0f47f58d4892dd3300032c8244d6278aecc).

hbase-2.0.0-alpha-1 will be our first 2.0.0 release. It is a rough

cut

('alpha') not-for-production preview of what hbase-2.0.0 will look

like. It

is what we used to call a 'Developer' release[1] meant mostly for

devs

and

downstreamers to test drive and flag us early if we there are issues

we’ve

missed ahead of our rolling a production-worthy release.

hbase-2.0.0 includes a fleet of new features that include a new

assignment

manager, means for keeping read and write path off-heap, in-memory
compactions, and more. I have been keeping a running doc on the state

of

2.0.0 here [2]. There is much to do still (see aforementioned doc).

The list of features addressed in 2.0.0 so far can be found here [4].

There

are about 2500. The list of ~500 fixes in 2.0.0 exclusively can be

found

here [3].

Please take it for a spin and vote on whether it ok to put out as our

first

alpha (bar is low for an 'alpha'). Let the VOTE be open for 24 hours.

Thanks,
St.Ack

1. http://hbase.apache.org/book.html#hbase.versioning.pre10
2.
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
ktczrlKHK8N4SZzs/edit#heading=h.v21r9nz8g01j
3.
https://issues.apache.org/jira/browse/HBASE-17852?jql=
project%20%3D%20HBASE%20%20and%20fixVersion%20%3D%202.
0.0%20and%20fixVersion%20not%20in%20(1.0.0%2C%201.0.1%2C%
201.0.2%2C%201.0.3%2C%201.0.4%2C%201.0.5%2C%201.0.6%2C%201.
1.0%2C%201.1.1%2C%201.1.2%2C%201.1.3%2C%201.1.4%2C%201.1.5%
2C%201.1.6%2C%201.1.7%2C%201.1.8%2C%201.1.9%2C%201.1.10%2C%
201.2.0%2C%201.2.1%2C%201.2.2%2C%201.2.3%2C%201.2.4%2C%201.
2.5%2C%201.2.6%2C%201.3.0%2C%201.3.1%2C%201.4.0)%20and%20%
20(status%20%3D%20Open%20or%20status%20%3D%20%22Patch%

20Available%22)

4.
https://issues.apache.org/jira/browse/HBASE-18191?jql=
project%20%3D%20HBASE%20%20and%20(%20fixVersion%20%3D%
202.0.0)%20and%20(status%20%3D%20Resolved)





--
Best regards,

- Andy

If you are given a choice, you believe you have acted freely. -

Raymond

Teller (via Peter Watts)

Re: join hbase slack

2017-06-20 Thread Sean Busbey 
invite sent!

On Tue, Jun 20, 2017 at 12:17 AM, Karan Mehta  wrote:
> I want an invite too.
>
> Thanks,
> Karan
> ᐧ
>
> On Mon, Jun 19, 2017 at 6:31 PM, 刘超  wrote:
>
>> HI，please send me an invite,I want to join hbase slack,thanks

Re: [RESULT] [VOTE] First hbase-2.0.0-alpha-1 Release Candidate is available

2017-06-20 Thread Mike Drob 
Hi Stack,

Can you mark jira version 2.0.0-alpha-1 as released?

Thanks,
Mike

On Sat, Jun 10, 2017 at 10:55 PM, Stack  wrote:

> With 4 binding votes and 1 non-binding, vote passes. Let me push out the
> alpha.
> Thanks to all who voted.
> St.Ack
>
>
> On Sat, Jun 10, 2017 at 11:40 AM, Stack  wrote:
>
> > Great. Thanks Sean. In-line...
> >
> > On Fri, Jun 9, 2017 at 11:35 PM, Sean Busbey  wrote:
> >
> >> +1
> >>
> >> details below, a few things to clean up for later alpha/betas.
> >>
> >> * checksums are all good
> >> * signatures are made with two different keys. should clean up by next
> >> alpha.
> >>
> >> src / bin artifacts are signed with 8ACC93D2, as you mentioned in the
> >> VOTE. However, this key is not in our project's KEYS file.
> >>
> >>
> > Yes. Will fix.
> >
> >
> >> maven staged repository are signed with 30CD0996, which is in our KEYS
> >> file so those check out fine.
> >>
> >> * the tag 2.0.0-alpha-1RC0 does point to
> >> c830a0f47f58d4892dd3300032c8244d6278aecc, which matches the source
> >> artifact after accounting HBASE-13088
> >>
> >> The tag isn't signed; would be good to make sure we do tag signing in
> >> betas so that it's ready to go come GA time.
> >>
> >>
> > Next time through, I'll update our 'How to RC' doc and will add above.
> >
> >
> >
> >> (side note, it would be good to get a decision on HBASE-13088 for the
> >> beta releases. been over 2 years)
> >>
> >> * the source tarball creates a binary assembly that looks as close to
> >> the posted binary artifact as I've seen branch-1 do.
> >>
> >> idle interest, but our binary convenience artifact has jumped from
> >> ~100MiB in branch-1 release to ~160MiB. If anyone has time to dig in
> >> on why, probably worth checking. (e.g. the docs directory is ~585 MiB
> >> when untared.)
> >>
> >>
> > Yes. Its obnoxious (HBASE-18208).
> >
> >
> >> * shaded artifact binaries are a reasonable number of MiB in size
> >> (incorrect build flags will result in ~empty jars)
> >>
> >> * LICENSE/NOTICE spot check looks okay.
> >>
> >> we have a ton of places where we have velocity variables instead of
> >> copyright years, but IIRC that's a problem on branch-1 right now too.
> >>
> >> * CHANGES file hasn't been updated correctly
> >>
> >> currently has details for 0.93.
> >>
> >>
> > Will fix next time through.
> >
> > Thanks Sean,
> > S
> >
> >
> >
> >> On Fri, Jun 9, 2017 at 12:06 PM, Andrew Purtell 
> >> wrote:
> >> > +1
> >> >
> >> > On Thu, Jun 8, 2017 at 12:33 AM, Stack  wrote:
> >> >
> >> >> The first release candidate for HBase 2.0.0-alpha-1 is up at:
> >> >>
> >> >>  https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-
> alpha-1RC0/
> >> >>
> >> >> Maven artifacts are available from a staging directory here:
> >> >>
> >> >>  https://repository.apache.org/content/repositories/orgapache
> >> hbase-1169
> >> >>
> >> >> All was signed w/ my key 8ACC93D2
> >> >> 
> >> >>
> >> >> I tagged the RC as 2.0.0-alpha-1RC0
> >> >> (c830a0f47f58d4892dd3300032c8244d6278aecc).
> >> >>
> >> >> hbase-2.0.0-alpha-1 will be our first 2.0.0 release. It is a rough
> cut
> >> >> ('alpha') not-for-production preview of what hbase-2.0.0 will look
> >> like. It
> >> >> is what we used to call a 'Developer' release[1] meant mostly for
> devs
> >> and
> >> >> downstreamers to test drive and flag us early if we there are issues
> >> we’ve
> >> >> missed ahead of our rolling a production-worthy release.
> >> >>
> >> >> hbase-2.0.0 includes a fleet of new features that include a new
> >> assignment
> >> >> manager, means for keeping read and write path off-heap, in-memory
> >> >> compactions, and more. I have been keeping a running doc on the state
> >> of
> >> >> 2.0.0 here [2]. There is much to do still (see aforementioned doc).
> >> >>
> >> >> The list of features addressed in 2.0.0 so far can be found here [4].
> >> There
> >> >> are about 2500. The list of ~500 fixes in 2.0.0 exclusively can be
> >> found
> >> >> here [3].
> >> >>
> >> >> Please take it for a spin and vote on whether it ok to put out as our
> >> first
> >> >> alpha (bar is low for an 'alpha'). Let the VOTE be open for 24 hours.
> >> >>
> >> >> Thanks,
> >> >> St.Ack
> >> >>
> >> >> 1. http://hbase.apache.org/book.html#hbase.versioning.pre10
> >> >> 2.
> >> >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> >> >> ktczrlKHK8N4SZzs/edit#heading=h.v21r9nz8g01j
> >> >> 3.
> >> >> https://issues.apache.org/jira/browse/HBASE-17852?jql=
> >> >> project%20%3D%20HBASE%20%20and%20fixVersion%20%3D%202.
> >> >> 0.0%20and%20fixVersion%20not%20in%20(1.0.0%2C%201.0.1%2C%
> >> >> 201.0.2%2C%201.0.3%2C%201.0.4%2C%201.0.5%2C%201.0.6%2C%201.
> >> >> 1.0%2C%201.1.1%2C%201.1.2%2C%201.1.3%2C%201.1.4%2C%201.1.5%
> >> >> 2C%201.1.6%2C%201.1.7%2C%201.1.8%2C%201.1.9%2C%201.1.10%2C%
> >> >> 201.2.0%2C%201.2.1%2C%201.2.2%2C%201.2.3%2C%201.2.4%2C%201.
> >> >> 2.5%2C%201.2.6%2C%201.3.0%2C%201.3.1%2C%201.4.0)%20and%20%
> >> >> 20(status%20

Re: Problem with IntegrationTestRegionReplicaReplication

2017-06-20 Thread Peter Somogyi 
I made some testing and found an interesting behavior that you might be
able to comment on.

When running the test against apache/branch-1.1 and apache/branch-1.2 using
the following command the tests consistently failed for me:
`mvn -pl hbase-it -am -Dtest=NoUnitTests
-Dit.test=IntegrationTestRegionReplicaReplication verify`

If I remove line 103 from the test then the test passes on both apache
branch and CDH based on v.1.2.
conf.setLong(HConstants.HREGION_MEMSTORE_FLUSH_SIZE, 1024L * 1024 * 4);
// flush every 4 MB

Do you know why setting hbase.hregion.memstore.flush.size is needed? As far
as I understand the test verifies that async WAL replication works. Don't
we bypass that functionality if we flush too frequently?

Thanks,
Peter

On Mon, Jun 19, 2017 at 2:55 AM, Devaraj Das  wrote:

> If it is failing consistently I'd suspect we have introduced a bug in the
> 1.2 line or something. We do run the same test with a version based on
> 1.1.2 (HDP-2.3 and beyond) and it works fine
>
>
>
>
> On Sun, Jun 18, 2017 at 8:26 AM -0700, "Peter Somogyi" <
> psomo...@cloudera.com> wrote:
>
>
> I'm using hbase based on 1.2 version.
>
> On Sat, Jun 17, 2017 at 4:00 PM, Devaraj Das  wrote:
>
> > Peter which version of HBase are tou testing with?
> >
> >
> >
> >
> > On Thu, Jun 15, 2017 at 11:57 PM -0700, "Peter Somogyi" <
> > psomo...@cloudera.com> wrote:
> >
> >
> > I tried with those parameters but the test still failed.
> > I noticed that some of the rows were not replicated to the replicas just
> > after I called flush manually. I think memstore replication is not
> working
> > on my system even though it is enabled in the configuration.
> > I will look into it today.
> >
> > On Fri, Jun 16, 2017 at 7:09 AM, Devaraj Das  wrote:
> >
> > > Peter, do have a look at IntegrationTestRegionReplicaReplication.java
> ..
> > > At the top of the file, the ways to specify the options are documented
> ..
> > > You need to add something like -DIntegrationTestRegionReplicaR
> > eplication.read_delay_ms
> > > ..
> > > 
> > > From: Josh Elser
> > > Sent: Thursday, June 15, 2017 10:40 AM
> > > To: dev@hbase.apache.org
> > > Subject: Re: Problem with IntegrationTestRegionReplicaReplication
> > >
> > > I'd start trying a read_delay_ms=6, region_replication=2,
> > > num_keys_per_server=5000, num_regions_per_server=5 with a maybe 10's of
> > > reader and writer threads.
> > >
> > > Again, this can be quite dependent on the kind of hardware you have.
> > > You'll definitely have to tweak ;)
> > >
> > > On 6/15/17 4:44 AM, Peter Somogyi wrote:
> > > > Thanks Josh and Devaraj!
> > > >
> > > > I will try to increase the timeouts. Devaraj, could you share the
> > > > parameters you used for this test which worked?
> > > >
> > > > On Thu, Jun 15, 2017 at 6:44 AM, Devaraj Das
> > > wrote:
> > > >
> > > >> That sounds about right, Josh. Peter, in our internal testing we
> have
> > > seen
> > > >> this test failing and increasing timeouts (look at the test code
> > > options to
> > > >> do with increasing timeout) helped quite some.
> > > >> 
> > > >> From: Josh Elser
> > > >> Sent: Wednesday, June 14, 2017 3:17 PM
> > > >> To: dev@hbase.apache.org
> > > >> Subject: Re: Problem with IntegrationTestRegionReplicaReplication
> > > >>
> > > >> On 6/14/17 3:53 AM, Peter Somogyi wrote:
> > > >>> Hi,
> > > >>>
> > > >>> As one of my first task with HBase I started to look into
> > > >>> why IntegrationTestRegionReplicaReplication fails. I would like to
> > get
> > > >> some
> > > >>> suggestions from you.
> > > >>>
> > > >>> I noticed when I run the test using normal cluster or minicluster I
> > get
> > > >> the
> > > >>> same error messages: "Error checking data for key [null], no data
> > > >>> returned". I looked into the code and here are my conclusions.
> > > >>>
> > > >>> There are multiple threads writing data parallel which are read by
> > > >> multiple
> > > >>> reader threads simultaneously. Each writer gets a portion of the
> keys
> > > to
> > > >>> write (e.g. 0-2000) and these keys are added to a
> ConstantDelayQueue.
> > > >>> The reader threads get the elements (e.g. key=1000) from the queue
> > and
> > > >>> these reader threads assume that all the keys up to this are
> already
> > in
> > > >> the
> > > >>> database. Since we're using multiple writers it can happen that
> > another
> > > >>> thread has not yet written key=500 and verifying these keys will
> > cause
> > > >> the
> > > >>> test failure.
> > > >>>
> > > >>> Do you think my assumption is correct?
> > > >>
> > > >> Hi Peter,
> > > >>
> > > >> No, as my memory serves, this is not correct. Readers are not made
> > aware
> > > >> of keys to verify until the write occur plus some delay. The delay
> is
> > > >> used to provide enough time for the internal region replication to
> > take
> > > >> effect.
> > > >>
> > > >> So: primary-write, pause, [region replicati

Fwd: Encryption of exisiting data in Stripe Compaction

2017-06-20 Thread ramkrishna vasudevan 
Hi all

Interesting case with Stripe compactions and Encryptions. Does any one has
any suggestion for Karthick's case? The initial mail was targetted to
issues@ and so forwarding it to dev@ and user@.


Regards
Ram

-- Forwarded message --
From: ramkrishna vasudevan 
Date: Tue, Jun 20, 2017 at 4:51 PM
Subject: Re: Encryption of exisiting data in Stripe Compaction
To: Karthick Ram 


I am not aware of any other mechanism. I just noticed that you had fwded
the message to issues@ and not to dev@ and users@. Let me forward it to
those mailing address. Thanks Karthick.

Regards
Ram

On Mon, Jun 19, 2017 at 1:07 PM, Karthick Ram 
wrote:

> Hi,
> Yes we are doing exactly the same. We altered the table with
> exploringcompaction and triggered a major compaction. But when it comes to
> key rotation, which we do very often, we have to manually alter the table
> and rollback to previous compaction policy. Currently we have a cron job
> for this. Is there any other way to automate this?
>
> Regards
> Karthick R
>
> On Thu, Jun 15, 2017 at 9:55 AM, ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
>> Hi
>> Very interesting case. Ya Stripe compaction does not need to under go a
>> major compaction if it already running under stripe compaction (reading the
>> docs I get this).
>> Since you have enable encryption at a later point of time you face this
>> issue I believe. The naive workaround I can think of is that do a alter
>> table with default compaction and it will do a major compaction and once
>> that is done again move back to Stripe compaction?  Will that work?
>>
>> I would like to hear opinion of others who have experience with Strip
>> compaction.
>>
>> Regards
>> Ram
>>
>> On Wed, Jun 14, 2017 at 10:25 AM, Karthick Ram 
>> wrote:
>>
>>> We have a table which has time series data with Stripe Compaction
>>> enabled.
>>> After encryption has been enabled for this table the newer entries are
>>> encrypted and inserted. However to encrypt the existing data in the
>>> table,
>>> a major compaction has to run. Since, stripe compaction doesn't allow a
>>> major compaction to run, we are unable to encrypt the previous data.
>>> Please
>>> suggest some ways to rectify this problem.
>>>
>>> Regards,
>>> Karthick R
>>>
>>
>>
>

[jira] [Created] (HBASE-18242) Client#openRegion, deleted files under regionencodeName/.tmp

2017-06-20 Thread Bo Cui (JIRA) 
Bo Cui created HBASE-18242:
--

 Summary: Client#openRegion, deleted files under 
regionencodeName/.tmp
 Key: HBASE-18242
 URL: https://issues.apache.org/jira/browse/HBASE-18242
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.10
Reporter: Bo Cui
Priority: Minor


There are two ways to open region:
1, RS#openRegion:flush, compact, split, merge, scan, get, etc.
2, Client#openRegion: only scan, get
Either way, all files under /hbase/data/default/TableName/RegionEncodeName/.tmp 
will be deleted when region is initialized
For method 2, this should not happen: the client opening region can only read 
data; there is no possibility of compaction; flush; if RS is processing 
compact, and client removes the TMP file, the RS needs to be re executed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)