[jira] [Resolved] (HBASE-16796) Correct RpcServer responseSize and requestSize to account for cellScanner payload

2016-12-07 Thread Jerry He (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-16796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-16796.
--
Resolution: Duplicate

> Correct RpcServer responseSize and requestSize to account for cellScanner 
> payload
> -
>
> Key: HBASE-16796
> URL: https://issues.apache.org/jira/browse/HBASE-16796
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>
> The reponseSize and requestSize on the RpcServer don't seem to account for 
> the cellScanner payload.  We should correct them.
> The metrics numbers and the logging for reponseTooLarge  and TooSlow won't  
> show up correctly otherwise.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17279) TestSerialReplication#testRegionSplit is flay

2016-12-07 Thread Ted Yu (JIRA) 
Ted Yu created HBASE-17279:
--

 Summary: TestSerialReplication#testRegionSplit is flay
 Key: HBASE-17279
 URL: https://issues.apache.org/jira/browse/HBASE-17279
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


>From 
>https://builds.apache.org/job/PreCommit-HBASE-Build/4832/artifact/patchprocess/patch-unit-hbase-server.txt
> :
{code}
testRegionSplit(org.apache.hadoop.hbase.replication.TestSerialReplication)  
Time elapsed: 24.247 sec  <<< FAILURE!
java.lang.AssertionError: expected:<9> but was:<4>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at 
org.apache.hadoop.hbase.replication.TestSerialReplication.testRegionSplit(TestSerialReplication.java:256)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17278) Cell Scanner Implementation to be used by ResultScanner

2016-12-07 Thread Sudeep Sunthankar (JIRA) 
Sudeep Sunthankar created HBASE-17278:
-

 Summary: Cell Scanner Implementation to be used by ResultScanner
 Key: HBASE-17278
 URL: https://issues.apache.org/jira/browse/HBASE-17278
 Project: HBase
  Issue Type: Sub-task
Reporter: Sudeep Sunthankar






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-15902) Scan Object

2016-12-07 Thread Enis Soztutar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar resolved HBASE-15902.
---
   Resolution: Fixed
Fix Version/s: HBASE-14850

Committed this to the branch. 

> Scan Object
> ---
>
> Key: HBASE-15902
> URL: https://issues.apache.org/jira/browse/HBASE-15902
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Fix For: HBASE-14850
>
> Attachments: HBASE-15902.HBASE-14850.patch, 
> HBASE-15902.HBASE-14850.v2.patch, HBASE-15902.HBASE-14850.v3.patch, 
> HBASE-15902.HBASE-14850.v4.patch
>
>
> Patch for creating Scan objects. Scan objects thus created can be used by 
> Table implementation to fetch results for a given row.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Successful: HBase Generate Website

2016-12-07 Thread Apache Jenkins Server 
Build status: Successful

If successful, the website and docs have been generated. To update the live 
site, follow the instructions below. If failed, skip to the bottom of this 
email.

Use the following commands to download the patch and apply it to a clean branch 
based on origin/asf-site. If you prefer to keep the hbase-site repo around 
permanently, you can skip the clone step.

  git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git

  cd hbase-site
  wget -O- 
https://builds.apache.org/job/hbase_generate_website/430/artifact/website.patch.zip
 | funzip > 6f25f838c0af80d2deaec91ed13248ce5024bd29.patch
  git fetch
  git checkout -b asf-site-6f25f838c0af80d2deaec91ed13248ce5024bd29 
origin/asf-site
  git am --whitespace=fix 6f25f838c0af80d2deaec91ed13248ce5024bd29.patch

At this point, you can preview the changes by opening index.html or any of the 
other HTML pages in your local 
asf-site-6f25f838c0af80d2deaec91ed13248ce5024bd29 branch.

There are lots of spurious changes, such as timestamps and CSS styles in 
tables, so a generic git diff is not very useful. To see a list of files that 
have been added, deleted, renamed, changed type, or are otherwise interesting, 
use the following command:

  git diff --name-status --diff-filter=ADCRTXUB origin/asf-site

To see only files that had 100 or more lines changed:

  git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'

When you are satisfied, publish your changes to origin/asf-site using these 
commands:

  git commit --allow-empty -m "Empty commit" # to work around a current ASF 
INFRA bug
  git push origin asf-site-6f25f838c0af80d2deaec91ed13248ce5024bd29:asf-site
  git checkout asf-site
  git branch -D asf-site-6f25f838c0af80d2deaec91ed13248ce5024bd29

Changes take a couple of minutes to be propagated. You can verify whether they 
have been propagated by looking at the Last Published date at the bottom of 
http://hbase.apache.org/. It should match the date in the index.html on the 
asf-site branch in Git.

As a courtesy- reply-all to this email to let other committers know you pushed 
the site.



If failed, see https://builds.apache.org/job/hbase_generate_website/430/console

[jira] [Created] (HBASE-17277) Allow alternate BufferedMutator implemenation

2016-12-07 Thread stack (JIRA) 
stack created HBASE-17277:
-

 Summary: Allow alternate BufferedMutator implemenation
 Key: HBASE-17277
 URL: https://issues.apache.org/jira/browse/HBASE-17277
 Project: HBase
  Issue Type: Sub-task
Reporter: stack
Assignee: stack


Allow being able to supply alternate BufferedMutator implementation. For 
example, the parent issue would like to spool writes to the filesystem if hbase 
is down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Successful: HBase Generate Website

2016-12-07 Thread Stack 
FYI, I pushed this patch just now.
St.Ack

On Wed, Dec 7, 2016 at 6:57 AM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build status: Successful
>
> If successful, the website and docs have been generated. To update the
> live site, follow the instructions below. If failed, skip to the bottom of
> this email.
>
> Use the following commands to download the patch and apply it to a clean
> branch based on origin/asf-site. If you prefer to keep the hbase-site repo
> around permanently, you can skip the clone step.
>
>   git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
>
>   cd hbase-site
>   wget -O- https://builds.apache.org/job/hbase_generate_website/429/
> artifact/website.patch.zip | funzip > 61220e4d7c8d7e5fb8ed3bbe2469bc
> 86632c48de.patch
>   git fetch
>   git checkout -b asf-site-61220e4d7c8d7e5fb8ed3bbe2469bc86632c48de
> origin/asf-site
>   git am --whitespace=fix 61220e4d7c8d7e5fb8ed3bbe2469bc86632c48de.patch
>
> At this point, you can preview the changes by opening index.html or any of
> the other HTML pages in your local 
> asf-site-61220e4d7c8d7e5fb8ed3bbe2469bc86632c48de
> branch.
>
> There are lots of spurious changes, such as timestamps and CSS styles in
> tables, so a generic git diff is not very useful. To see a list of files
> that have been added, deleted, renamed, changed type, or are otherwise
> interesting, use the following command:
>
>   git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
>
> To see only files that had 100 or more lines changed:
>
>   git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
>
> When you are satisfied, publish your changes to origin/asf-site using
> these commands:
>
>   git commit --allow-empty -m "Empty commit" # to work around a current
> ASF INFRA bug
>   git push origin asf-site-61220e4d7c8d7e5fb8ed3bbe2469bc
> 86632c48de:asf-site
>   git checkout asf-site
>   git branch -D asf-site-61220e4d7c8d7e5fb8ed3bbe2469bc86632c48de
>
> Changes take a couple of minutes to be propagated. You can verify whether
> they have been propagated by looking at the Last Published date at the
> bottom of http://hbase.apache.org/. It should match the date in the
> index.html on the asf-site branch in Git.
>
> As a courtesy- reply-all to this email to let other committers know you
> pushed the site.
>
>
>
> If failed, see https://builds.apache.org/job/hbase_generate_website/429/
> console

[jira] [Resolved] (HBASE-17272) Doc how to run Standalone HBase over an HDFS instance; all daemons in one JVM but persisting to an HDFS instance

2016-12-07 Thread stack (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-17272.
---
   Resolution: Fixed
 Assignee: stack
Fix Version/s: 2.0.0

I pushed the last patch. Will update the site separately.

> Doc how to run Standalone HBase over an HDFS instance; all daemons in one JVM 
> but persisting to an HDFS instance
> 
>
> Key: HBASE-17272
> URL: https://issues.apache.org/jira/browse/HBASE-17272
> Project: HBase
>  Issue Type: Task
>  Components: documentation
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 17272.txt, HBASE-17272.master.001.patch, 
> HBASE-17272.master.002.patch
>
>
> It is possible to run hbase in standalone mode -- all daemons in the one JVM 
> (Master+RegionServers+ZooKeeper) -- but persisting to HDFS rather than the 
> usual local filesystem. That this works was news to me. This configuration is 
> wanted for the default deploy of the timeline server v2. The standalone mode 
> hbase shoudl be good enough for light loading. The persistence to HDFS makes 
> it so data can live across the coming and goings of nodes.
> This issue is about doc'ing how to setup this config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17276) Reduce log spam from WrongRegionException in large multi()'s

2016-12-07 Thread Josh Elser (JIRA) 
Josh Elser created HBASE-17276:
--

 Summary: Reduce log spam from WrongRegionException in large 
multi()'s
 Key: HBASE-17276
 URL: https://issues.apache.org/jira/browse/HBASE-17276
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Minor
 Fix For: 2.0.0


The following spam drives me up a wall in the regionserver log:

{noformat}
2016-12-05 05:53:05,085 WARN  
[RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16020] 
regionserver.HRegion: Batch mutation had a row that does not belong to this 
region
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of 
range for doMiniBatchMutation on HRegion 
IntegrationTestReplicationSinkRestart,L\xCC\xCC\xCC\xCC\xCC\xCC\xC8,1480916713541.caab3310166699287b54b72b35b29431.,
 startKey='L\xCC\xCC\xCC\xCC\xCC\xCC\xC8', 
getEndKey()='Y\x99\x99\x99\x99\x99\x99\x94', 
row='\x0C\xD2\xA5\xA3\x99\xC7\xE0Q!\x15^\xA6\x90\x1E\xA3\xAD'
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5211)
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkAndPrepareMutation(HRegion.java:3879)
at 
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3040)
at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2933)
at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:717)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:679)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2056)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2141)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
2016-12-05 05:53:05,086 WARN  
[RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16020] 
regionserver.HRegion: Batch mutation had a row that does not belong to this 
region
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of 
range for doMiniBatchMutation on HRegion 
IntegrationTestReplicationSinkRestart,L\xCC\xCC\xCC\xCC\xCC\xCC\xC8,1480916713541.caab3310166699287b54b72b35b29431.,
 startKey='L\xCC\xCC\xCC\xCC\xCC\xCC\xC8', 
getEndKey()='Y\x99\x99\x99\x99\x99\x99\x94', 
row='\x0E\xE7\xFA[\x8D\x93;\xF4\xC7F\xF9\x85\x84\x85\xF3\x0E'
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5211)
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkAndPrepareMutation(HRegion.java:3879)
at 
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3040)
at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2933)
at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:717)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:679)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2056)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2141)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
2016-12-05 05:53:05,087 WARN  
[RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16020] 
regionserver.HRegion: Batch mutation had a row that does not belong to this 
region
org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out of 
range for doMiniBatchMutation on HRegion 
IntegrationTestReplicationSinkRestart,L\xCC\xCC\xCC\xCC\xCC\xCC\xC8,1480916713541.caab3310166699287b54b72b35b29431.,
 startKey='L\xCC\xCC\xCC\xCC\xCC\xCC\xC8', 
getEndKey()='Y\x99\x99\x99\x99\x99\x99\x94', 
row='\x16-\xFC\x99\xF5c\x08\xFA\x1D\x84\x86\xD2\x18\xB1\x03q'
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5211)
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkAndPrepareMutation(HRegion.java:3879)
at 
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3040)
at 
org.apache.hadoop.hbase.r

Successful: HBase Generate Website

2016-12-07 Thread Apache Jenkins Server 
Build status: Successful

If successful, the website and docs have been generated. To update the live 
site, follow the instructions below. If failed, skip to the bottom of this 
email.

Use the following commands to download the patch and apply it to a clean branch 
based on origin/asf-site. If you prefer to keep the hbase-site repo around 
permanently, you can skip the clone step.

  git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git

  cd hbase-site
  wget -O- 
https://builds.apache.org/job/hbase_generate_website/429/artifact/website.patch.zip
 | funzip > 61220e4d7c8d7e5fb8ed3bbe2469bc86632c48de.patch
  git fetch
  git checkout -b asf-site-61220e4d7c8d7e5fb8ed3bbe2469bc86632c48de 
origin/asf-site
  git am --whitespace=fix 61220e4d7c8d7e5fb8ed3bbe2469bc86632c48de.patch

At this point, you can preview the changes by opening index.html or any of the 
other HTML pages in your local 
asf-site-61220e4d7c8d7e5fb8ed3bbe2469bc86632c48de branch.

There are lots of spurious changes, such as timestamps and CSS styles in 
tables, so a generic git diff is not very useful. To see a list of files that 
have been added, deleted, renamed, changed type, or are otherwise interesting, 
use the following command:

  git diff --name-status --diff-filter=ADCRTXUB origin/asf-site

To see only files that had 100 or more lines changed:

  git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'

When you are satisfied, publish your changes to origin/asf-site using these 
commands:

  git commit --allow-empty -m "Empty commit" # to work around a current ASF 
INFRA bug
  git push origin asf-site-61220e4d7c8d7e5fb8ed3bbe2469bc86632c48de:asf-site
  git checkout asf-site
  git branch -D asf-site-61220e4d7c8d7e5fb8ed3bbe2469bc86632c48de

Changes take a couple of minutes to be propagated. You can verify whether they 
have been propagated by looking at the Last Published date at the bottom of 
http://hbase.apache.org/. It should match the date in the index.html on the 
asf-site branch in Git.

As a courtesy- reply-all to this email to let other committers know you pushed 
the site.



If failed, see https://builds.apache.org/job/hbase_generate_website/429/console

Re: meta region written by 2.0 cannot be opened by 1.x

2016-12-07 Thread Ted Yu 
I agree. 

Just marked 15265 and 10800 incompatible change. 

> On Dec 7, 2016, at 2:38 AM, 张铎  wrote:
> 
> I think the imcompatible change is HBASE-15265? HBASE-14949 aims to fix the
> incompatible... Just like HBASE-16189 to HBASE-10800.
> 
> I think we should mark HBASE-15265 and HBASE-10800 as Incompatible Change?
> 
> 2016-12-07 18:29 GMT+08:00 Ted Yu :
> 
>> Shouldn't HBASE-14949 be marked Incompatible Change ?
>> 
>>> On Wed, Dec 7, 2016 at 2:17 AM, 张铎  wrote:
>>> 
>>> See this one
>>> 
>>> https://issues.apache.org/jira/browse/HBASE-14949
>>> 
>>> 2016-12-07 18:09 GMT+08:00 Ted Yu :
>>> 
 Which other features ?
 Can you name them ?
 
 Thanks
 
> On Wed, Dec 7, 2016 at 2:06 AM, 张铎  wrote:
> 
> I believe there are other features that require upgrading to a
>> specific
> version in each 1.x release before rolling upgrading to 2.0? I think
>>> this
> is a typical trick to add incompatible changes...
> 
> 2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com>:
> 
>> If its going to be a pain then we need to support writing the older
 class
>> name in the FFT (trailer) and then use our internal mapping.
>> 
>>> On Wed, Dec 7, 2016 at 3:22 PM, Anoop John 
>> wrote:
>> 
>>> I read the comments in the issue 16189..  This concern was raised
>>> there already..   This is what I replied there.
>>> "Ya we may need to make such a need. User on 1.x has to first
>>> upgrade
>>> (rolling upgrade supported) to latest version in 1.x and then to
>>> 2.0.
>>> This was discussed some other place also. I believe in the mail
 thread
>>> where we discussed abt rolling upgrade support in 2.0"
>>> 
>>> Not able to find the original mail thread which discussed abt
>>> rolling
>>> upgrade support in 2.0Pls correct if took some thing wrong
>>> way..
>>> Ya I agree that this way of indirection might be inconvenience
>> for
>>> users.
>>> 
>>> 
>>> -Anoop-
>>> 
>>> On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu 
>>> wrote:
 bq. write old name only and within code we will have to map it
>>> with
> new
 name.
 
 This is a viable approach.
 Even if we document upgrading to 1.2.3+, some users may not
>>> perform
>> this
 step before upgrading to 2.0
 
 On Tue, Dec 6, 2016 at 10:04 PM, Anoop John <
>>> anoop.hb...@gmail.com
> 
>>> wrote:
 
> Ya that bug raised by Enis was very valid..  So this means
>> when
> rolling upgrade happens, if some of the other RSs with version
> <1.2.3
> , which is not having this fix,  this issue might come up!
> How to address then?   Do we need to enforce a 1.2.3+ upgrade
>>> 1st
> and
> then only 2.0 rolling upgrade?
> 
> Or else we will need to fix in 2.0..  We write the new
>>> Comparator
> class name in FFT (trunk code)   To fix, we might have to
>> write
 old
> name only and within code we will have to map it with new
>> name.
>>> It
> will be ugly! It can be fixed only in 3.0 may be..  But that
>> can
> make
> the rolling upgrade story easier for users..   Just saying the
> possibilities..
> 
> Thanks Ted to bring it up again.
> 
> -Anoop-
> 
> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
>  wrote:
>> I think when Enis reported the issue (
>> https://issues.apache.org/jira/browse/HBASE-16189), in
>>> rolling
>>> upgrade
>> regions could move around. So a region created in 2.0 moved
>>> to a
>>> server
>> with 1.x.
>> 
>> Regards
>> Ram
>> 
>> 
>> On Wed, Dec 7, 2016 at 1:27 AM, Stack 
>>> wrote:
>> 
>>> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu <
>> yuzhih...@gmail.com
 
>> wrote:
>>> 
 Looking at http://hbase.apache.org/book.
> html#executing.the.0.96.
> upgrade
>>> ,
 there is step of running "bin/hbase upgrade -check"
 
 How about adding a sample hfile which contains
 CellComparator$MetaCellComparator
 so that the upgrade check can read and verify ?
>>> 
>>> We don't support downgrade. Never have.
>>> St.Ack
>>> 
>>> 
>>> 
 On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu <
>>> yuzhih...@gmail.com>
>>> wrote:
 
> The build I used yesterday didn't include HBASE-16189
> 
> 
> Once it is included, the cluster can be downgraded
>> fine.
> 
> I wonder how users would know that their existing
 deployment
>> has
> HBASE-16189 >

Re: meta region written by 2.0 cannot be opened by 1.x

2016-12-07 Thread 张铎 
I think the imcompatible change is HBASE-15265? HBASE-14949 aims to fix the
incompatible... Just like HBASE-16189 to HBASE-10800.

I think we should mark HBASE-15265 and HBASE-10800 as Incompatible Change?

2016-12-07 18:29 GMT+08:00 Ted Yu :

> Shouldn't HBASE-14949 be marked Incompatible Change ?
>
> On Wed, Dec 7, 2016 at 2:17 AM, 张铎  wrote:
>
> > See this one
> >
> > https://issues.apache.org/jira/browse/HBASE-14949
> >
> > 2016-12-07 18:09 GMT+08:00 Ted Yu :
> >
> > > Which other features ?
> > > Can you name them ?
> > >
> > > Thanks
> > >
> > > On Wed, Dec 7, 2016 at 2:06 AM, 张铎  wrote:
> > >
> > > > I believe there are other features that require upgrading to a
> specific
> > > > version in each 1.x release before rolling upgrading to 2.0? I think
> > this
> > > > is a typical trick to add incompatible changes...
> > > >
> > > > 2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan <
> > > > ramkrishna.s.vasude...@gmail.com>:
> > > >
> > > > > If its going to be a pain then we need to support writing the older
> > > class
> > > > > name in the FFT (trailer) and then use our internal mapping.
> > > > >
> > > > > On Wed, Dec 7, 2016 at 3:22 PM, Anoop John 
> > > > wrote:
> > > > >
> > > > > > I read the comments in the issue 16189..  This concern was raised
> > > > > > there already..   This is what I replied there.
> > > > > > "Ya we may need to make such a need. User on 1.x has to first
> > upgrade
> > > > > > (rolling upgrade supported) to latest version in 1.x and then to
> > 2.0.
> > > > > > This was discussed some other place also. I believe in the mail
> > > thread
> > > > > > where we discussed abt rolling upgrade support in 2.0"
> > > > > >
> > > > > > Not able to find the original mail thread which discussed abt
> > rolling
> > > > > > upgrade support in 2.0Pls correct if took some thing wrong
> > way..
> > > > > > Ya I agree that this way of indirection might be inconvenience
> for
> > > > > > users.
> > > > > >
> > > > > >
> > > > > > -Anoop-
> > > > > >
> > > > > > On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu 
> > wrote:
> > > > > > > bq. write old name only and within code we will have to map it
> > with
> > > > new
> > > > > > > name.
> > > > > > >
> > > > > > > This is a viable approach.
> > > > > > > Even if we document upgrading to 1.2.3+, some users may not
> > perform
> > > > > this
> > > > > > > step before upgrading to 2.0
> > > > > > >
> > > > > > > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John <
> > anoop.hb...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> Ya that bug raised by Enis was very valid..  So this means
> when
> > > > > > >> rolling upgrade happens, if some of the other RSs with version
> > > > <1.2.3
> > > > > > >> , which is not having this fix,  this issue might come up!
> > > > > > >> How to address then?   Do we need to enforce a 1.2.3+ upgrade
> > 1st
> > > > and
> > > > > > >> then only 2.0 rolling upgrade?
> > > > > > >>
> > > > > > >> Or else we will need to fix in 2.0..  We write the new
> > Comparator
> > > > > > >> class name in FFT (trunk code)   To fix, we might have to
> write
> > > old
> > > > > > >> name only and within code we will have to map it with new
> name.
> > It
> > > > > > >> will be ugly! It can be fixed only in 3.0 may be..  But that
> can
> > > > make
> > > > > > >> the rolling upgrade story easier for users..   Just saying the
> > > > > > >> possibilities..
> > > > > > >>
> > > > > > >> Thanks Ted to bring it up again.
> > > > > > >>
> > > > > > >> -Anoop-
> > > > > > >>
> > > > > > >> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
> > > > > > >>  wrote:
> > > > > > >> > I think when Enis reported the issue (
> > > > > > >> > https://issues.apache.org/jira/browse/HBASE-16189), in
> > rolling
> > > > > > upgrade
> > > > > > >> > regions could move around. So a region created in 2.0 moved
> > to a
> > > > > > server
> > > > > > >> > with 1.x.
> > > > > > >> >
> > > > > > >> > Regards
> > > > > > >> > Ram
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Dec 7, 2016 at 1:27 AM, Stack 
> > wrote:
> > > > > > >> >
> > > > > > >> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu <
> yuzhih...@gmail.com
> > >
> > > > > wrote:
> > > > > > >> >>
> > > > > > >> >> > Looking at http://hbase.apache.org/book.
> > > > html#executing.the.0.96.
> > > > > > >> upgrade
> > > > > > >> >> ,
> > > > > > >> >> > there is step of running "bin/hbase upgrade -check"
> > > > > > >> >> >
> > > > > > >> >> > How about adding a sample hfile which contains
> > > > > > >> >> > CellComparator$MetaCellComparator
> > > > > > >> >> > so that the upgrade check can read and verify ?
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >> >> We don't support downgrade. Never have.
> > > > > > >> >> St.Ack
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu <
> > yuzhih...@gmail.com>
> > > > > > wrote:
> > > > > > >> >> >
> > > > > > >> >> > > The build I used yesterday didn't i

Re: meta region written by 2.0 cannot be opened by 1.x

2016-12-07 Thread Ted Yu 
Shouldn't HBASE-14949 be marked Incompatible Change ?

On Wed, Dec 7, 2016 at 2:17 AM, 张铎  wrote:

> See this one
>
> https://issues.apache.org/jira/browse/HBASE-14949
>
> 2016-12-07 18:09 GMT+08:00 Ted Yu :
>
> > Which other features ?
> > Can you name them ?
> >
> > Thanks
> >
> > On Wed, Dec 7, 2016 at 2:06 AM, 张铎  wrote:
> >
> > > I believe there are other features that require upgrading to a specific
> > > version in each 1.x release before rolling upgrading to 2.0? I think
> this
> > > is a typical trick to add incompatible changes...
> > >
> > > 2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan <
> > > ramkrishna.s.vasude...@gmail.com>:
> > >
> > > > If its going to be a pain then we need to support writing the older
> > class
> > > > name in the FFT (trailer) and then use our internal mapping.
> > > >
> > > > On Wed, Dec 7, 2016 at 3:22 PM, Anoop John 
> > > wrote:
> > > >
> > > > > I read the comments in the issue 16189..  This concern was raised
> > > > > there already..   This is what I replied there.
> > > > > "Ya we may need to make such a need. User on 1.x has to first
> upgrade
> > > > > (rolling upgrade supported) to latest version in 1.x and then to
> 2.0.
> > > > > This was discussed some other place also. I believe in the mail
> > thread
> > > > > where we discussed abt rolling upgrade support in 2.0"
> > > > >
> > > > > Not able to find the original mail thread which discussed abt
> rolling
> > > > > upgrade support in 2.0Pls correct if took some thing wrong
> way..
> > > > > Ya I agree that this way of indirection might be inconvenience for
> > > > > users.
> > > > >
> > > > >
> > > > > -Anoop-
> > > > >
> > > > > On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu 
> wrote:
> > > > > > bq. write old name only and within code we will have to map it
> with
> > > new
> > > > > > name.
> > > > > >
> > > > > > This is a viable approach.
> > > > > > Even if we document upgrading to 1.2.3+, some users may not
> perform
> > > > this
> > > > > > step before upgrading to 2.0
> > > > > >
> > > > > > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John <
> anoop.hb...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Ya that bug raised by Enis was very valid..  So this means when
> > > > > >> rolling upgrade happens, if some of the other RSs with version
> > > <1.2.3
> > > > > >> , which is not having this fix,  this issue might come up!
> > > > > >> How to address then?   Do we need to enforce a 1.2.3+ upgrade
> 1st
> > > and
> > > > > >> then only 2.0 rolling upgrade?
> > > > > >>
> > > > > >> Or else we will need to fix in 2.0..  We write the new
> Comparator
> > > > > >> class name in FFT (trunk code)   To fix, we might have to write
> > old
> > > > > >> name only and within code we will have to map it with new name.
> It
> > > > > >> will be ugly! It can be fixed only in 3.0 may be..  But that can
> > > make
> > > > > >> the rolling upgrade story easier for users..   Just saying the
> > > > > >> possibilities..
> > > > > >>
> > > > > >> Thanks Ted to bring it up again.
> > > > > >>
> > > > > >> -Anoop-
> > > > > >>
> > > > > >> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
> > > > > >>  wrote:
> > > > > >> > I think when Enis reported the issue (
> > > > > >> > https://issues.apache.org/jira/browse/HBASE-16189), in
> rolling
> > > > > upgrade
> > > > > >> > regions could move around. So a region created in 2.0 moved
> to a
> > > > > server
> > > > > >> > with 1.x.
> > > > > >> >
> > > > > >> > Regards
> > > > > >> > Ram
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Dec 7, 2016 at 1:27 AM, Stack 
> wrote:
> > > > > >> >
> > > > > >> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu  >
> > > > wrote:
> > > > > >> >>
> > > > > >> >> > Looking at http://hbase.apache.org/book.
> > > html#executing.the.0.96.
> > > > > >> upgrade
> > > > > >> >> ,
> > > > > >> >> > there is step of running "bin/hbase upgrade -check"
> > > > > >> >> >
> > > > > >> >> > How about adding a sample hfile which contains
> > > > > >> >> > CellComparator$MetaCellComparator
> > > > > >> >> > so that the upgrade check can read and verify ?
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >> We don't support downgrade. Never have.
> > > > > >> >> St.Ack
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu <
> yuzhih...@gmail.com>
> > > > > wrote:
> > > > > >> >> >
> > > > > >> >> > > The build I used yesterday didn't include HBASE-16189
> > > > > >> >> > > 
> > > > > >> >> > >
> > > > > >> >> > > Once it is included, the cluster can be downgraded fine.
> > > > > >> >> > >
> > > > > >> >> > > I wonder how users would know that their existing
> > deployment
> > > > has
> > > > > >> >> > > HBASE-16189  > > jira/browse/HBASE-16189
> > > > >
> > > > > >> before
> > > > > >> >> > > upgrading to 2.0 release.
> > > > > >> >> > >
> > > > > >> >> > > On Tue, Dec 6, 2016 at 2:29 AM, ramkrishna v

Re: meta region written by 2.0 cannot be opened by 1.x

2016-12-07 Thread 张铎 
And a bad news is that this patch is not contained in 1.2.x and 1.1.x which
means it is not safe to rolling upgrade from 1.2.x or 1.1.x to 2.0 if we
change the default wal implementation to AsyncFSWA, or a user set the wal
implementation to asyncfs manually...

2016-12-07 18:17 GMT+08:00 张铎 :

> See this one
>
> https://issues.apache.org/jira/browse/HBASE-14949
>
> 2016-12-07 18:09 GMT+08:00 Ted Yu :
>
>> Which other features ?
>> Can you name them ?
>>
>> Thanks
>>
>> On Wed, Dec 7, 2016 at 2:06 AM, 张铎  wrote:
>>
>> > I believe there are other features that require upgrading to a specific
>> > version in each 1.x release before rolling upgrading to 2.0? I think
>> this
>> > is a typical trick to add incompatible changes...
>> >
>> > 2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan <
>> > ramkrishna.s.vasude...@gmail.com>:
>> >
>> > > If its going to be a pain then we need to support writing the older
>> class
>> > > name in the FFT (trailer) and then use our internal mapping.
>> > >
>> > > On Wed, Dec 7, 2016 at 3:22 PM, Anoop John 
>> > wrote:
>> > >
>> > > > I read the comments in the issue 16189..  This concern was raised
>> > > > there already..   This is what I replied there.
>> > > > "Ya we may need to make such a need. User on 1.x has to first
>> upgrade
>> > > > (rolling upgrade supported) to latest version in 1.x and then to
>> 2.0.
>> > > > This was discussed some other place also. I believe in the mail
>> thread
>> > > > where we discussed abt rolling upgrade support in 2.0"
>> > > >
>> > > > Not able to find the original mail thread which discussed abt
>> rolling
>> > > > upgrade support in 2.0Pls correct if took some thing wrong way..
>> > > > Ya I agree that this way of indirection might be inconvenience for
>> > > > users.
>> > > >
>> > > >
>> > > > -Anoop-
>> > > >
>> > > > On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu  wrote:
>> > > > > bq. write old name only and within code we will have to map it
>> with
>> > new
>> > > > > name.
>> > > > >
>> > > > > This is a viable approach.
>> > > > > Even if we document upgrading to 1.2.3+, some users may not
>> perform
>> > > this
>> > > > > step before upgrading to 2.0
>> > > > >
>> > > > > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John <
>> anoop.hb...@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > >> Ya that bug raised by Enis was very valid..  So this means when
>> > > > >> rolling upgrade happens, if some of the other RSs with version
>> > <1.2.3
>> > > > >> , which is not having this fix,  this issue might come up!
>> > > > >> How to address then?   Do we need to enforce a 1.2.3+ upgrade 1st
>> > and
>> > > > >> then only 2.0 rolling upgrade?
>> > > > >>
>> > > > >> Or else we will need to fix in 2.0..  We write the new Comparator
>> > > > >> class name in FFT (trunk code)   To fix, we might have to write
>> old
>> > > > >> name only and within code we will have to map it with new name.
>> It
>> > > > >> will be ugly! It can be fixed only in 3.0 may be..  But that can
>> > make
>> > > > >> the rolling upgrade story easier for users..   Just saying the
>> > > > >> possibilities..
>> > > > >>
>> > > > >> Thanks Ted to bring it up again.
>> > > > >>
>> > > > >> -Anoop-
>> > > > >>
>> > > > >> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
>> > > > >>  wrote:
>> > > > >> > I think when Enis reported the issue (
>> > > > >> > https://issues.apache.org/jira/browse/HBASE-16189), in rolling
>> > > > upgrade
>> > > > >> > regions could move around. So a region created in 2.0 moved to
>> a
>> > > > server
>> > > > >> > with 1.x.
>> > > > >> >
>> > > > >> > Regards
>> > > > >> > Ram
>> > > > >> >
>> > > > >> >
>> > > > >> > On Wed, Dec 7, 2016 at 1:27 AM, Stack 
>> wrote:
>> > > > >> >
>> > > > >> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu 
>> > > wrote:
>> > > > >> >>
>> > > > >> >> > Looking at http://hbase.apache.org/book.
>> > html#executing.the.0.96.
>> > > > >> upgrade
>> > > > >> >> ,
>> > > > >> >> > there is step of running "bin/hbase upgrade -check"
>> > > > >> >> >
>> > > > >> >> > How about adding a sample hfile which contains
>> > > > >> >> > CellComparator$MetaCellComparator
>> > > > >> >> > so that the upgrade check can read and verify ?
>> > > > >> >> >
>> > > > >> >> >
>> > > > >> >>
>> > > > >> >> We don't support downgrade. Never have.
>> > > > >> >> St.Ack
>> > > > >> >>
>> > > > >> >>
>> > > > >> >>
>> > > > >> >> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu > >
>> > > > wrote:
>> > > > >> >> >
>> > > > >> >> > > The build I used yesterday didn't include HBASE-16189
>> > > > >> >> > > 
>> > > > >> >> > >
>> > > > >> >> > > Once it is included, the cluster can be downgraded fine.
>> > > > >> >> > >
>> > > > >> >> > > I wonder how users would know that their existing
>> deployment
>> > > has
>> > > > >> >> > > HBASE-16189 > > jira/browse/HBASE-16189
>> > > >
>> > > > >> before
>> > > > >> >> > > upgrading to 2.0 release.
>> > > > >> >> > >
>> > > > >> >> > >

Re: meta region written by 2.0 cannot be opened by 1.x

2016-12-07 Thread 张铎 
See this one

https://issues.apache.org/jira/browse/HBASE-14949

2016-12-07 18:09 GMT+08:00 Ted Yu :

> Which other features ?
> Can you name them ?
>
> Thanks
>
> On Wed, Dec 7, 2016 at 2:06 AM, 张铎  wrote:
>
> > I believe there are other features that require upgrading to a specific
> > version in each 1.x release before rolling upgrading to 2.0? I think this
> > is a typical trick to add incompatible changes...
> >
> > 2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com>:
> >
> > > If its going to be a pain then we need to support writing the older
> class
> > > name in the FFT (trailer) and then use our internal mapping.
> > >
> > > On Wed, Dec 7, 2016 at 3:22 PM, Anoop John 
> > wrote:
> > >
> > > > I read the comments in the issue 16189..  This concern was raised
> > > > there already..   This is what I replied there.
> > > > "Ya we may need to make such a need. User on 1.x has to first upgrade
> > > > (rolling upgrade supported) to latest version in 1.x and then to 2.0.
> > > > This was discussed some other place also. I believe in the mail
> thread
> > > > where we discussed abt rolling upgrade support in 2.0"
> > > >
> > > > Not able to find the original mail thread which discussed abt rolling
> > > > upgrade support in 2.0Pls correct if took some thing wrong way..
> > > > Ya I agree that this way of indirection might be inconvenience for
> > > > users.
> > > >
> > > >
> > > > -Anoop-
> > > >
> > > > On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu  wrote:
> > > > > bq. write old name only and within code we will have to map it with
> > new
> > > > > name.
> > > > >
> > > > > This is a viable approach.
> > > > > Even if we document upgrading to 1.2.3+, some users may not perform
> > > this
> > > > > step before upgrading to 2.0
> > > > >
> > > > > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John  >
> > > > wrote:
> > > > >
> > > > >> Ya that bug raised by Enis was very valid..  So this means when
> > > > >> rolling upgrade happens, if some of the other RSs with version
> > <1.2.3
> > > > >> , which is not having this fix,  this issue might come up!
> > > > >> How to address then?   Do we need to enforce a 1.2.3+ upgrade 1st
> > and
> > > > >> then only 2.0 rolling upgrade?
> > > > >>
> > > > >> Or else we will need to fix in 2.0..  We write the new Comparator
> > > > >> class name in FFT (trunk code)   To fix, we might have to write
> old
> > > > >> name only and within code we will have to map it with new name. It
> > > > >> will be ugly! It can be fixed only in 3.0 may be..  But that can
> > make
> > > > >> the rolling upgrade story easier for users..   Just saying the
> > > > >> possibilities..
> > > > >>
> > > > >> Thanks Ted to bring it up again.
> > > > >>
> > > > >> -Anoop-
> > > > >>
> > > > >> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
> > > > >>  wrote:
> > > > >> > I think when Enis reported the issue (
> > > > >> > https://issues.apache.org/jira/browse/HBASE-16189), in rolling
> > > > upgrade
> > > > >> > regions could move around. So a region created in 2.0 moved to a
> > > > server
> > > > >> > with 1.x.
> > > > >> >
> > > > >> > Regards
> > > > >> > Ram
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Dec 7, 2016 at 1:27 AM, Stack  wrote:
> > > > >> >
> > > > >> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu 
> > > wrote:
> > > > >> >>
> > > > >> >> > Looking at http://hbase.apache.org/book.
> > html#executing.the.0.96.
> > > > >> upgrade
> > > > >> >> ,
> > > > >> >> > there is step of running "bin/hbase upgrade -check"
> > > > >> >> >
> > > > >> >> > How about adding a sample hfile which contains
> > > > >> >> > CellComparator$MetaCellComparator
> > > > >> >> > so that the upgrade check can read and verify ?
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >> >> We don't support downgrade. Never have.
> > > > >> >> St.Ack
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu 
> > > > wrote:
> > > > >> >> >
> > > > >> >> > > The build I used yesterday didn't include HBASE-16189
> > > > >> >> > > 
> > > > >> >> > >
> > > > >> >> > > Once it is included, the cluster can be downgraded fine.
> > > > >> >> > >
> > > > >> >> > > I wonder how users would know that their existing
> deployment
> > > has
> > > > >> >> > > HBASE-16189  > jira/browse/HBASE-16189
> > > >
> > > > >> before
> > > > >> >> > > upgrading to 2.0 release.
> > > > >> >> > >
> > > > >> >> > > On Tue, Dec 6, 2016 at 2:29 AM, ramkrishna vasudevan <
> > > > >> >> > > ramkrishna.s.vasude...@gmail.com> wrote:
> > > > >> >> > >
> > > > >> >> > >> @Ted
> > > > >> >> > >> Does your version have this fix
> > > > >> >> > >> https://issues.apache.org/jira/browse/HBASE-16189
> > > > >> >> > >>
> > > > >> >> > >> Regards
> > > > >> >> > >> Ram
> > > > >> >> > >>
> > > > >> >> > >> On Tue, Dec 6, 2016 at 3:56 PM, Ted Yu <
> yuzhih...@gmail.com
> > >
> > > > >> wrote:
> > > > >> >>

Re: meta region written by 2.0 cannot be opened by 1.x

2016-12-07 Thread Ted Yu 
Which other features ?
Can you name them ?

Thanks

On Wed, Dec 7, 2016 at 2:06 AM, 张铎  wrote:

> I believe there are other features that require upgrading to a specific
> version in each 1.x release before rolling upgrading to 2.0? I think this
> is a typical trick to add incompatible changes...
>
> 2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com>:
>
> > If its going to be a pain then we need to support writing the older class
> > name in the FFT (trailer) and then use our internal mapping.
> >
> > On Wed, Dec 7, 2016 at 3:22 PM, Anoop John 
> wrote:
> >
> > > I read the comments in the issue 16189..  This concern was raised
> > > there already..   This is what I replied there.
> > > "Ya we may need to make such a need. User on 1.x has to first upgrade
> > > (rolling upgrade supported) to latest version in 1.x and then to 2.0.
> > > This was discussed some other place also. I believe in the mail thread
> > > where we discussed abt rolling upgrade support in 2.0"
> > >
> > > Not able to find the original mail thread which discussed abt rolling
> > > upgrade support in 2.0Pls correct if took some thing wrong way..
> > > Ya I agree that this way of indirection might be inconvenience for
> > > users.
> > >
> > >
> > > -Anoop-
> > >
> > > On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu  wrote:
> > > > bq. write old name only and within code we will have to map it with
> new
> > > > name.
> > > >
> > > > This is a viable approach.
> > > > Even if we document upgrading to 1.2.3+, some users may not perform
> > this
> > > > step before upgrading to 2.0
> > > >
> > > > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John 
> > > wrote:
> > > >
> > > >> Ya that bug raised by Enis was very valid..  So this means when
> > > >> rolling upgrade happens, if some of the other RSs with version
> <1.2.3
> > > >> , which is not having this fix,  this issue might come up!
> > > >> How to address then?   Do we need to enforce a 1.2.3+ upgrade 1st
> and
> > > >> then only 2.0 rolling upgrade?
> > > >>
> > > >> Or else we will need to fix in 2.0..  We write the new Comparator
> > > >> class name in FFT (trunk code)   To fix, we might have to write old
> > > >> name only and within code we will have to map it with new name. It
> > > >> will be ugly! It can be fixed only in 3.0 may be..  But that can
> make
> > > >> the rolling upgrade story easier for users..   Just saying the
> > > >> possibilities..
> > > >>
> > > >> Thanks Ted to bring it up again.
> > > >>
> > > >> -Anoop-
> > > >>
> > > >> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
> > > >>  wrote:
> > > >> > I think when Enis reported the issue (
> > > >> > https://issues.apache.org/jira/browse/HBASE-16189), in rolling
> > > upgrade
> > > >> > regions could move around. So a region created in 2.0 moved to a
> > > server
> > > >> > with 1.x.
> > > >> >
> > > >> > Regards
> > > >> > Ram
> > > >> >
> > > >> >
> > > >> > On Wed, Dec 7, 2016 at 1:27 AM, Stack  wrote:
> > > >> >
> > > >> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu 
> > wrote:
> > > >> >>
> > > >> >> > Looking at http://hbase.apache.org/book.
> html#executing.the.0.96.
> > > >> upgrade
> > > >> >> ,
> > > >> >> > there is step of running "bin/hbase upgrade -check"
> > > >> >> >
> > > >> >> > How about adding a sample hfile which contains
> > > >> >> > CellComparator$MetaCellComparator
> > > >> >> > so that the upgrade check can read and verify ?
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >> >> We don't support downgrade. Never have.
> > > >> >> St.Ack
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu 
> > > wrote:
> > > >> >> >
> > > >> >> > > The build I used yesterday didn't include HBASE-16189
> > > >> >> > > 
> > > >> >> > >
> > > >> >> > > Once it is included, the cluster can be downgraded fine.
> > > >> >> > >
> > > >> >> > > I wonder how users would know that their existing deployment
> > has
> > > >> >> > > HBASE-16189  jira/browse/HBASE-16189
> > >
> > > >> before
> > > >> >> > > upgrading to 2.0 release.
> > > >> >> > >
> > > >> >> > > On Tue, Dec 6, 2016 at 2:29 AM, ramkrishna vasudevan <
> > > >> >> > > ramkrishna.s.vasude...@gmail.com> wrote:
> > > >> >> > >
> > > >> >> > >> @Ted
> > > >> >> > >> Does your version have this fix
> > > >> >> > >> https://issues.apache.org/jira/browse/HBASE-16189
> > > >> >> > >>
> > > >> >> > >> Regards
> > > >> >> > >> Ram
> > > >> >> > >>
> > > >> >> > >> On Tue, Dec 6, 2016 at 3:56 PM, Ted Yu  >
> > > >> wrote:
> > > >> >> > >>
> > > >> >> > >> > Is the assumption that hbase:meta would not split ?
> > > >> >> > >> >
> > > >> >> > >> > In other thread, Francis Liu was proposing supporting
> > > splittable
> > > >> >> > >> > hbase:meta in 2.0 release.
> > > >> >> > >> >
> > > >> >> > >> > Cheers
> > > >> >> > >> >
> > > >> >> > >> > > On Dec 6, 2016, at 2:20 AM, 张铎 
> > > wrote:
> > > >> >> > >> > >
> > > >> >> > >>

Re: meta region written by 2.0 cannot be opened by 1.x

2016-12-07 Thread 张铎 
I believe there are other features that require upgrading to a specific
version in each 1.x release before rolling upgrading to 2.0? I think this
is a typical trick to add incompatible changes...

2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com>:

> If its going to be a pain then we need to support writing the older class
> name in the FFT (trailer) and then use our internal mapping.
>
> On Wed, Dec 7, 2016 at 3:22 PM, Anoop John  wrote:
>
> > I read the comments in the issue 16189..  This concern was raised
> > there already..   This is what I replied there.
> > "Ya we may need to make such a need. User on 1.x has to first upgrade
> > (rolling upgrade supported) to latest version in 1.x and then to 2.0.
> > This was discussed some other place also. I believe in the mail thread
> > where we discussed abt rolling upgrade support in 2.0"
> >
> > Not able to find the original mail thread which discussed abt rolling
> > upgrade support in 2.0Pls correct if took some thing wrong way..
> > Ya I agree that this way of indirection might be inconvenience for
> > users.
> >
> >
> > -Anoop-
> >
> > On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu  wrote:
> > > bq. write old name only and within code we will have to map it with new
> > > name.
> > >
> > > This is a viable approach.
> > > Even if we document upgrading to 1.2.3+, some users may not perform
> this
> > > step before upgrading to 2.0
> > >
> > > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John 
> > wrote:
> > >
> > >> Ya that bug raised by Enis was very valid..  So this means when
> > >> rolling upgrade happens, if some of the other RSs with version <1.2.3
> > >> , which is not having this fix,  this issue might come up!
> > >> How to address then?   Do we need to enforce a 1.2.3+ upgrade 1st and
> > >> then only 2.0 rolling upgrade?
> > >>
> > >> Or else we will need to fix in 2.0..  We write the new Comparator
> > >> class name in FFT (trunk code)   To fix, we might have to write old
> > >> name only and within code we will have to map it with new name. It
> > >> will be ugly! It can be fixed only in 3.0 may be..  But that can make
> > >> the rolling upgrade story easier for users..   Just saying the
> > >> possibilities..
> > >>
> > >> Thanks Ted to bring it up again.
> > >>
> > >> -Anoop-
> > >>
> > >> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
> > >>  wrote:
> > >> > I think when Enis reported the issue (
> > >> > https://issues.apache.org/jira/browse/HBASE-16189), in rolling
> > upgrade
> > >> > regions could move around. So a region created in 2.0 moved to a
> > server
> > >> > with 1.x.
> > >> >
> > >> > Regards
> > >> > Ram
> > >> >
> > >> >
> > >> > On Wed, Dec 7, 2016 at 1:27 AM, Stack  wrote:
> > >> >
> > >> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu 
> wrote:
> > >> >>
> > >> >> > Looking at http://hbase.apache.org/book.html#executing.the.0.96.
> > >> upgrade
> > >> >> ,
> > >> >> > there is step of running "bin/hbase upgrade -check"
> > >> >> >
> > >> >> > How about adding a sample hfile which contains
> > >> >> > CellComparator$MetaCellComparator
> > >> >> > so that the upgrade check can read and verify ?
> > >> >> >
> > >> >> >
> > >> >>
> > >> >> We don't support downgrade. Never have.
> > >> >> St.Ack
> > >> >>
> > >> >>
> > >> >>
> > >> >> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu 
> > wrote:
> > >> >> >
> > >> >> > > The build I used yesterday didn't include HBASE-16189
> > >> >> > > 
> > >> >> > >
> > >> >> > > Once it is included, the cluster can be downgraded fine.
> > >> >> > >
> > >> >> > > I wonder how users would know that their existing deployment
> has
> > >> >> > > HBASE-16189  >
> > >> before
> > >> >> > > upgrading to 2.0 release.
> > >> >> > >
> > >> >> > > On Tue, Dec 6, 2016 at 2:29 AM, ramkrishna vasudevan <
> > >> >> > > ramkrishna.s.vasude...@gmail.com> wrote:
> > >> >> > >
> > >> >> > >> @Ted
> > >> >> > >> Does your version have this fix
> > >> >> > >> https://issues.apache.org/jira/browse/HBASE-16189
> > >> >> > >>
> > >> >> > >> Regards
> > >> >> > >> Ram
> > >> >> > >>
> > >> >> > >> On Tue, Dec 6, 2016 at 3:56 PM, Ted Yu 
> > >> wrote:
> > >> >> > >>
> > >> >> > >> > Is the assumption that hbase:meta would not split ?
> > >> >> > >> >
> > >> >> > >> > In other thread, Francis Liu was proposing supporting
> > splittable
> > >> >> > >> > hbase:meta in 2.0 release.
> > >> >> > >> >
> > >> >> > >> > Cheers
> > >> >> > >> >
> > >> >> > >> > > On Dec 6, 2016, at 2:20 AM, 张铎 
> > wrote:
> > >> >> > >> > >
> > >> >> > >> > > Could this be solved by hosting meta only on master?
> > >> >> > >> > >
> > >> >> > >> > > BTW, MetaCellComparator is introduced in HBASE-10800.
> > >> >> > >> > >
> > >> >> > >> > > Thanks.
> > >> >> > >> > >
> > >> >> > >> > > 2016-12-06 17:44 GMT+08:00 Ted Yu :
> > >> >> > >> > >
> > >> >> > >> > >> Hi,
> > >> >> > >> > >> When I restarted a cluster with 1.1 , I

[jira] [Created] (HBASE-17275) Assign timeout cause region unassign forever

2016-12-07 Thread Allan Yang (JIRA) 
Allan Yang created HBASE-17275:
--

 Summary: Assign timeout cause region unassign forever
 Key: HBASE-17275
 URL: https://issues.apache.org/jira/browse/HBASE-17275
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Affects Versions: 1.1.7, 1.2.3
Reporter: Allan Yang
Assignee: Allan Yang


This is a real cased happened in my test cluster.
I have more 8000 regions to assign when I restart a cluster, but I only started 
one regionserver. That means master need to assign these 8000 regions to a 
single server(I know it is not right, but just for testing).

The rs recevied the open region rpc and began to open regions. But the due to 
the hugh number of regions, , master timeout the rpc call(but actually some 
region had already opened) after 1 mins, as you can see from log 1.
{noformat}
1. 2016-11-22 10:17:32,285 INFO  [example.org:30001.activeMasterManager] 
master.AssignmentManager: Unable to communicate with 
example.org,30003,1479780976834 in order to assign regions,
java.io.IOException: Call to /example.org:30003 failed on local exception: 
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=1, waitTime=60001, 
operationTimeout=6 expired.
at 
org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1338)
at 
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272)
at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)
at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:290)
at 
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:30177)
at 
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:1000)
at 
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1719)
at 
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2828)
at 
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2775)
at 
org.apache.hadoop.hbase.master.AssignmentManager.assignAllUserRegions(AssignmentManager.java:2876)
at 
org.apache.hadoop.hbase.master.AssignmentManager.processDeadServersAndRegionsInTransition(AssignmentManager.java:646)
at 
org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:493)
at 
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:796)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:188)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1711)
at java.lang.Thread.run(Thread.java:756)
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=1, 
waitTime=60001, operationTimeout=6 expired.
at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:81)
at 
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1246)
... 14 more  
{noformat}
for the region 7e9aee32eb98a6fc9d503b99fc5f9615(like many others), after 
timeout, master use a pool to re-assign them, as in 2
{noformat}
2. 2016-11-22 10:17:32,303 DEBUG [AM.-pool1-t26] master.AssignmentManager: 
Force region state offline {7e9aee32eb98a6fc9d503b99fc5f9615 
state=PENDING_OPEN, ts=1479780992078, server=example.org,30003,1479780976834}  
{noformat}

But, this region was actually opened on the rs, but (maybe) due to the hugh 
pressure, the OPENED zk event recevied by master , as you can tell from 3, 
"which is more than 15 seconds late"
{noformat}
3. 2016-11-22 10:17:32,304 DEBUG [AM.ZK.Worker-pool2-t3] 
master.AssignmentManager: Handling RS_ZK_REGION_OPENED, 
server=example.org,30003,1479780976834, 
region=7e9aee32eb98a6fc9d503b99fc5f9615, which is more than 15 seconds late, 
current_state={7e9aee32eb98a6fc9d503b99fc5f9615 state=PENDING_OPEN, 
ts=1479780992078, server=example.org,30003,1479780976834}
{noformat}

In the meantime, master still try to re-assign this region in another thread. 
Master first close this region in case of multi assign, then change the state 
of this region change from PENDING_OPEN >OFFLINE>PENDING_OPEN. Its RIT node in 
zk was also transitioned to OFFLINE, as in 4,5,6,7
{noformat}
4. 2016-11-22 10:17:32,321 DEBUG [AM.-pool1-t26] master.AssignmentManager: Sent 
CLOSE to example.org,30003,1479780976834 for region 
test,P7HQ55,1475985973151.7e9aee32eb98a6fc9d503b99fc5f9615.
5. 2016-11-22 10:17:32,461 INFO  [AM.-pool1-t26] master.RegionStates: 
Transition {7e9aee32eb98a6fc9d503b99fc5f9615 state=PENDING_OPEN, 
ts=1479781052344, server=example.org,30003,1479780976834} to 
{7e9aee32eb98a6fc9d503b99fc5f9615 state=OFFLINE, ts=1479781052461, 
server=example.org,30003,1479780976834}
6. 2016-11-22 10:17:32,469 DEBUG [AM.-poo

Re: meta region written by 2.0 cannot be opened by 1.x

2016-12-07 Thread ramkrishna vasudevan 
If its going to be a pain then we need to support writing the older class
name in the FFT (trailer) and then use our internal mapping.

On Wed, Dec 7, 2016 at 3:22 PM, Anoop John  wrote:

> I read the comments in the issue 16189..  This concern was raised
> there already..   This is what I replied there.
> "Ya we may need to make such a need. User on 1.x has to first upgrade
> (rolling upgrade supported) to latest version in 1.x and then to 2.0.
> This was discussed some other place also. I believe in the mail thread
> where we discussed abt rolling upgrade support in 2.0"
>
> Not able to find the original mail thread which discussed abt rolling
> upgrade support in 2.0Pls correct if took some thing wrong way..
> Ya I agree that this way of indirection might be inconvenience for
> users.
>
>
> -Anoop-
>
> On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu  wrote:
> > bq. write old name only and within code we will have to map it with new
> > name.
> >
> > This is a viable approach.
> > Even if we document upgrading to 1.2.3+, some users may not perform this
> > step before upgrading to 2.0
> >
> > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John 
> wrote:
> >
> >> Ya that bug raised by Enis was very valid..  So this means when
> >> rolling upgrade happens, if some of the other RSs with version <1.2.3
> >> , which is not having this fix,  this issue might come up!
> >> How to address then?   Do we need to enforce a 1.2.3+ upgrade 1st and
> >> then only 2.0 rolling upgrade?
> >>
> >> Or else we will need to fix in 2.0..  We write the new Comparator
> >> class name in FFT (trunk code)   To fix, we might have to write old
> >> name only and within code we will have to map it with new name. It
> >> will be ugly! It can be fixed only in 3.0 may be..  But that can make
> >> the rolling upgrade story easier for users..   Just saying the
> >> possibilities..
> >>
> >> Thanks Ted to bring it up again.
> >>
> >> -Anoop-
> >>
> >> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
> >>  wrote:
> >> > I think when Enis reported the issue (
> >> > https://issues.apache.org/jira/browse/HBASE-16189), in rolling
> upgrade
> >> > regions could move around. So a region created in 2.0 moved to a
> server
> >> > with 1.x.
> >> >
> >> > Regards
> >> > Ram
> >> >
> >> >
> >> > On Wed, Dec 7, 2016 at 1:27 AM, Stack  wrote:
> >> >
> >> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu  wrote:
> >> >>
> >> >> > Looking at http://hbase.apache.org/book.html#executing.the.0.96.
> >> upgrade
> >> >> ,
> >> >> > there is step of running "bin/hbase upgrade -check"
> >> >> >
> >> >> > How about adding a sample hfile which contains
> >> >> > CellComparator$MetaCellComparator
> >> >> > so that the upgrade check can read and verify ?
> >> >> >
> >> >> >
> >> >>
> >> >> We don't support downgrade. Never have.
> >> >> St.Ack
> >> >>
> >> >>
> >> >>
> >> >> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu 
> wrote:
> >> >> >
> >> >> > > The build I used yesterday didn't include HBASE-16189
> >> >> > > 
> >> >> > >
> >> >> > > Once it is included, the cluster can be downgraded fine.
> >> >> > >
> >> >> > > I wonder how users would know that their existing deployment has
> >> >> > > HBASE-16189 
> >> before
> >> >> > > upgrading to 2.0 release.
> >> >> > >
> >> >> > > On Tue, Dec 6, 2016 at 2:29 AM, ramkrishna vasudevan <
> >> >> > > ramkrishna.s.vasude...@gmail.com> wrote:
> >> >> > >
> >> >> > >> @Ted
> >> >> > >> Does your version have this fix
> >> >> > >> https://issues.apache.org/jira/browse/HBASE-16189
> >> >> > >>
> >> >> > >> Regards
> >> >> > >> Ram
> >> >> > >>
> >> >> > >> On Tue, Dec 6, 2016 at 3:56 PM, Ted Yu 
> >> wrote:
> >> >> > >>
> >> >> > >> > Is the assumption that hbase:meta would not split ?
> >> >> > >> >
> >> >> > >> > In other thread, Francis Liu was proposing supporting
> splittable
> >> >> > >> > hbase:meta in 2.0 release.
> >> >> > >> >
> >> >> > >> > Cheers
> >> >> > >> >
> >> >> > >> > > On Dec 6, 2016, at 2:20 AM, 张铎 
> wrote:
> >> >> > >> > >
> >> >> > >> > > Could this be solved by hosting meta only on master?
> >> >> > >> > >
> >> >> > >> > > BTW, MetaCellComparator is introduced in HBASE-10800.
> >> >> > >> > >
> >> >> > >> > > Thanks.
> >> >> > >> > >
> >> >> > >> > > 2016-12-06 17:44 GMT+08:00 Ted Yu :
> >> >> > >> > >
> >> >> > >> > >> Hi,
> >> >> > >> > >> When I restarted a cluster with 1.1 , I found that
> hbase:meta
> >> >> > region
> >> >> > >> > >> (written to by the previously deployed 2.0) couldn't be
> >> opened:
> >> >> > >> > >>
> >> >> > >> > >> Caused by: java.io.IOException:
> >> >> > >> > >> org.apache.hadoop.hbase.io.hfile.CorruptHFileException:
> >> Problem
> >> >> > >> reading
> >> >> > >> > >> HFile Trailer from file hdfs://yz1.xx.com:8020/apps/
> >> >> > >> hbase/data/data/
> >> >> > >> > >> hbase/meta/1588230740/info/599fc8a37311414e876803312009a9
> 86
> >> >> > >> > >>at
> >> >> > >> > >> org.apache.had

Re: meta region written by 2.0 cannot be opened by 1.x

2016-12-07 Thread Anoop John 
I read the comments in the issue 16189..  This concern was raised
there already..   This is what I replied there.
"Ya we may need to make such a need. User on 1.x has to first upgrade
(rolling upgrade supported) to latest version in 1.x and then to 2.0.
This was discussed some other place also. I believe in the mail thread
where we discussed abt rolling upgrade support in 2.0"

Not able to find the original mail thread which discussed abt rolling
upgrade support in 2.0Pls correct if took some thing wrong way..
Ya I agree that this way of indirection might be inconvenience for
users.


-Anoop-

On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu  wrote:
> bq. write old name only and within code we will have to map it with new
> name.
>
> This is a viable approach.
> Even if we document upgrading to 1.2.3+, some users may not perform this
> step before upgrading to 2.0
>
> On Tue, Dec 6, 2016 at 10:04 PM, Anoop John  wrote:
>
>> Ya that bug raised by Enis was very valid..  So this means when
>> rolling upgrade happens, if some of the other RSs with version <1.2.3
>> , which is not having this fix,  this issue might come up!
>> How to address then?   Do we need to enforce a 1.2.3+ upgrade 1st and
>> then only 2.0 rolling upgrade?
>>
>> Or else we will need to fix in 2.0..  We write the new Comparator
>> class name in FFT (trunk code)   To fix, we might have to write old
>> name only and within code we will have to map it with new name. It
>> will be ugly! It can be fixed only in 3.0 may be..  But that can make
>> the rolling upgrade story easier for users..   Just saying the
>> possibilities..
>>
>> Thanks Ted to bring it up again.
>>
>> -Anoop-
>>
>> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
>>  wrote:
>> > I think when Enis reported the issue (
>> > https://issues.apache.org/jira/browse/HBASE-16189), in rolling upgrade
>> > regions could move around. So a region created in 2.0 moved to a server
>> > with 1.x.
>> >
>> > Regards
>> > Ram
>> >
>> >
>> > On Wed, Dec 7, 2016 at 1:27 AM, Stack  wrote:
>> >
>> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu  wrote:
>> >>
>> >> > Looking at http://hbase.apache.org/book.html#executing.the.0.96.
>> upgrade
>> >> ,
>> >> > there is step of running "bin/hbase upgrade -check"
>> >> >
>> >> > How about adding a sample hfile which contains
>> >> > CellComparator$MetaCellComparator
>> >> > so that the upgrade check can read and verify ?
>> >> >
>> >> >
>> >>
>> >> We don't support downgrade. Never have.
>> >> St.Ack
>> >>
>> >>
>> >>
>> >> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu  wrote:
>> >> >
>> >> > > The build I used yesterday didn't include HBASE-16189
>> >> > > 
>> >> > >
>> >> > > Once it is included, the cluster can be downgraded fine.
>> >> > >
>> >> > > I wonder how users would know that their existing deployment has
>> >> > > HBASE-16189 
>> before
>> >> > > upgrading to 2.0 release.
>> >> > >
>> >> > > On Tue, Dec 6, 2016 at 2:29 AM, ramkrishna vasudevan <
>> >> > > ramkrishna.s.vasude...@gmail.com> wrote:
>> >> > >
>> >> > >> @Ted
>> >> > >> Does your version have this fix
>> >> > >> https://issues.apache.org/jira/browse/HBASE-16189
>> >> > >>
>> >> > >> Regards
>> >> > >> Ram
>> >> > >>
>> >> > >> On Tue, Dec 6, 2016 at 3:56 PM, Ted Yu 
>> wrote:
>> >> > >>
>> >> > >> > Is the assumption that hbase:meta would not split ?
>> >> > >> >
>> >> > >> > In other thread, Francis Liu was proposing supporting splittable
>> >> > >> > hbase:meta in 2.0 release.
>> >> > >> >
>> >> > >> > Cheers
>> >> > >> >
>> >> > >> > > On Dec 6, 2016, at 2:20 AM, 张铎  wrote:
>> >> > >> > >
>> >> > >> > > Could this be solved by hosting meta only on master?
>> >> > >> > >
>> >> > >> > > BTW, MetaCellComparator is introduced in HBASE-10800.
>> >> > >> > >
>> >> > >> > > Thanks.
>> >> > >> > >
>> >> > >> > > 2016-12-06 17:44 GMT+08:00 Ted Yu :
>> >> > >> > >
>> >> > >> > >> Hi,
>> >> > >> > >> When I restarted a cluster with 1.1 , I found that hbase:meta
>> >> > region
>> >> > >> > >> (written to by the previously deployed 2.0) couldn't be
>> opened:
>> >> > >> > >>
>> >> > >> > >> Caused by: java.io.IOException:
>> >> > >> > >> org.apache.hadoop.hbase.io.hfile.CorruptHFileException:
>> Problem
>> >> > >> reading
>> >> > >> > >> HFile Trailer from file hdfs://yz1.xx.com:8020/apps/
>> >> > >> hbase/data/data/
>> >> > >> > >> hbase/meta/1588230740/info/599fc8a37311414e876803312009a986
>> >> > >> > >>at
>> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(
>> >> > >> HStore.java:
>> >> > >> > >> 579)
>> >> > >> > >>at
>> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(
>> >> > >> HStore.java:
>> >> > >> > >> 534)
>> >> > >> > >>at
>> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore.(
>> >> > HStore.java:275)
>> >> > >> > >>at
>> >> > >> > >> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHSto
>> >> >

Re: meta region written by 2.0 cannot be opened by 1.x

2016-12-07 Thread Ted Yu 
bq. write old name only and within code we will have to map it with new
name.

This is a viable approach.
Even if we document upgrading to 1.2.3+, some users may not perform this
step before upgrading to 2.0

On Tue, Dec 6, 2016 at 10:04 PM, Anoop John  wrote:

> Ya that bug raised by Enis was very valid..  So this means when
> rolling upgrade happens, if some of the other RSs with version <1.2.3
> , which is not having this fix,  this issue might come up!
> How to address then?   Do we need to enforce a 1.2.3+ upgrade 1st and
> then only 2.0 rolling upgrade?
>
> Or else we will need to fix in 2.0..  We write the new Comparator
> class name in FFT (trunk code)   To fix, we might have to write old
> name only and within code we will have to map it with new name. It
> will be ugly! It can be fixed only in 3.0 may be..  But that can make
> the rolling upgrade story easier for users..   Just saying the
> possibilities..
>
> Thanks Ted to bring it up again.
>
> -Anoop-
>
> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
>  wrote:
> > I think when Enis reported the issue (
> > https://issues.apache.org/jira/browse/HBASE-16189), in rolling upgrade
> > regions could move around. So a region created in 2.0 moved to a server
> > with 1.x.
> >
> > Regards
> > Ram
> >
> >
> > On Wed, Dec 7, 2016 at 1:27 AM, Stack  wrote:
> >
> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu  wrote:
> >>
> >> > Looking at http://hbase.apache.org/book.html#executing.the.0.96.
> upgrade
> >> ,
> >> > there is step of running "bin/hbase upgrade -check"
> >> >
> >> > How about adding a sample hfile which contains
> >> > CellComparator$MetaCellComparator
> >> > so that the upgrade check can read and verify ?
> >> >
> >> >
> >>
> >> We don't support downgrade. Never have.
> >> St.Ack
> >>
> >>
> >>
> >> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu  wrote:
> >> >
> >> > > The build I used yesterday didn't include HBASE-16189
> >> > > 
> >> > >
> >> > > Once it is included, the cluster can be downgraded fine.
> >> > >
> >> > > I wonder how users would know that their existing deployment has
> >> > > HBASE-16189 
> before
> >> > > upgrading to 2.0 release.
> >> > >
> >> > > On Tue, Dec 6, 2016 at 2:29 AM, ramkrishna vasudevan <
> >> > > ramkrishna.s.vasude...@gmail.com> wrote:
> >> > >
> >> > >> @Ted
> >> > >> Does your version have this fix
> >> > >> https://issues.apache.org/jira/browse/HBASE-16189
> >> > >>
> >> > >> Regards
> >> > >> Ram
> >> > >>
> >> > >> On Tue, Dec 6, 2016 at 3:56 PM, Ted Yu 
> wrote:
> >> > >>
> >> > >> > Is the assumption that hbase:meta would not split ?
> >> > >> >
> >> > >> > In other thread, Francis Liu was proposing supporting splittable
> >> > >> > hbase:meta in 2.0 release.
> >> > >> >
> >> > >> > Cheers
> >> > >> >
> >> > >> > > On Dec 6, 2016, at 2:20 AM, 张铎  wrote:
> >> > >> > >
> >> > >> > > Could this be solved by hosting meta only on master?
> >> > >> > >
> >> > >> > > BTW, MetaCellComparator is introduced in HBASE-10800.
> >> > >> > >
> >> > >> > > Thanks.
> >> > >> > >
> >> > >> > > 2016-12-06 17:44 GMT+08:00 Ted Yu :
> >> > >> > >
> >> > >> > >> Hi,
> >> > >> > >> When I restarted a cluster with 1.1 , I found that hbase:meta
> >> > region
> >> > >> > >> (written to by the previously deployed 2.0) couldn't be
> opened:
> >> > >> > >>
> >> > >> > >> Caused by: java.io.IOException:
> >> > >> > >> org.apache.hadoop.hbase.io.hfile.CorruptHFileException:
> Problem
> >> > >> reading
> >> > >> > >> HFile Trailer from file hdfs://yz1.xx.com:8020/apps/
> >> > >> hbase/data/data/
> >> > >> > >> hbase/meta/1588230740/info/599fc8a37311414e876803312009a986
> >> > >> > >>at
> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(
> >> > >> HStore.java:
> >> > >> > >> 579)
> >> > >> > >>at
> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(
> >> > >> HStore.java:
> >> > >> > >> 534)
> >> > >> > >>at
> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore.(
> >> > HStore.java:275)
> >> > >> > >>at
> >> > >> > >> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHSto
> >> > >> re(HRegion.
> >> > >> > >> java:5150)
> >> > >> > >>at
> >> > >> > >> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.
> >> > >> java:912)
> >> > >> > >>at
> >> > >> > >> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.
> >> > >> java:909)
> >> > >> > >>at java.util.concurrent.FutureTask.run(FutureTask.
> >> java:266)
> >> > >> > >>at
> >> > >> > >> java.util.concurrent.Executors$RunnableAdapter.call(
> >> > >> Executors.java:511)
> >> > >> > >>at java.util.concurrent.FutureTask.run(FutureTask.
> >> java:266)
> >> > >> > >>... 3 more
> >> > >> > >> Caused by: org.apache.hadoop.hbase.io.
> >> hfile.CorruptHFileException:
> >> > >> > Problem
> >> > >> > >> reading HFile Trailer from file hdfs://
> >>

[jira] [Created] (HBASE-17274) Add a server level throttler for replication sources

2016-12-07 Thread Phil Yang (JIRA) 
Phil Yang created HBASE-17274:
-

 Summary: Add a server level throttler for replication sources
 Key: HBASE-17274
 URL: https://issues.apache.org/jira/browse/HBASE-17274
 Project: HBase
  Issue Type: Improvement
Reporter: Phil Yang
Assignee: Phil Yang


If we have many peers or some servers have many recovered queues, the total 
read size per second from WALs will be very high, which will increase the 
pressure of GC, even maybe OOM because we will read entries for 64MB to buffer 
in default.

So we should add a server level throttler to limit the speed of reading WALs, 
and limit the total buffered entries for all ReplicationSources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)