Creating SQL skin on Hbase table

2016-12-22 Thread Mich Talebzadeh 
Hi,

I have used Phoenix to create SQL views on top of Hbase table and also
created covering engines. The issue I have noticed is that all ingestion
has to go through Phoenix otherwise Phoenix indexes will not be updated.

Another alternative has been to try Hive EXTERNAL tables on top of Hbase
table but only EXTERNAL tables are supported. I was wondering the pros and
cons of using Hive or Phoenix tables on Hbase?

Thanks

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

Re: Lease exception

2016-12-22 Thread Phil Yang 
https://github.com/apache/hbase/blob/rel/1.1.1/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L2491

There is a TTL for scanners at server, to prevent client don't close the
scanners and they leak. The TTL is configured by
hbase.client.scanner.timeout.period at server and refreshed when a scan RPC
request comes . The TTLs of all scanners are managed by Lease. Your error
happens when server closes a scanner but in Lease it is already expired. So
I think you can try to increase  hbase.client.scanner.timeout.period at
server or decrease your hbase.client.scanner.timeout.period at client to
prevent the scanner expired before its scanning done.
hbase.client.scanner.timeout.period is used both at client and server, may
be different if you change one of sides.

BTW, I still suggest that you can upgrade your cluster and client. 1.1.1
has some data-loss bugs on scanning.

Thanks,
Phil


2016-12-22 17:26 GMT+08:00 Rajeshkumar J :

> can you please explain what is the cause of this lease exception and is
> there any solve this in current version
>
> Thanks
>
> On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang  wrote:
>
> > In fact at client the rpc timeout of scan request is also
> > hbase.client.scanner.timeout.period which replaces the
> > deprecated hbase.regionserver.lease.period.
> >
> > Your code that throws LeaseException has been removed by HBASE-16604,
> maybe
> > you can try to upgrade your cluster to 1.1.7? Your client can also
> upgrade
> > to 1.1.7 which will ignore UnknowScannerException and retry when the
> lease
> > is expired at server.
> >
> > Thanks,
> > Phil
> >
> >
> > 2016-12-22 16:51 GMT+08:00 Rajeshkumar J :
> >
> > > Also there is a solution what i have found from hbase user guide that
> > > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.
> > > period.
> > > How these two properties plays a part in the above exception. Please
> can
> > > anyone explain?
> > >
> > > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J <
> > > rajeshkumarit8...@gmail.com>
> > > wrote:
> > >
> > > > I am using hbase version 1.1.1
> > > > Also I didn't understand something here. Whenever a scanner.next() is
> > > > called it needs to return rows(based on caching value) within leasing
> > > > period or else scanner client will be closed eventually throwing this
> > > > exception. Correct me as I didn't get the clear understanding of this
> > > issue
> > > >
> > > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu  wrote:
> > > >
> > > >> Which hbase release are you using ?
> > > >>
> > > >> There is heartbeat support when scanning.
> > > >> Looks like the version you use doesn't have this support.
> > > >>
> > > >> Cheers
> > > >>
> > > >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <
> > > rajeshkumarit8...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > Hi,
> > > >> >
> > > >> >   Thanks for the reply. I have properties as below
> > > >> >
> > > >> > 
> > > >> >hbase.regionserver.lease.period
> > > >> >90
> > > >> >  
> > > >> >  
> > > >> >hbase.rpc.timeout
> > > >> >90>/value>
> > > >> >  
> > > >> >
> > > >> >
> > > >> > Correct me If I am wrong.
> > > >> >
> > > >> > I know hbase.regionserver.lease.period, which says how long a
> > scanner
> > > >> > lives between calls to scanner.next().
> > > >> >
> > > >> > As far as I understand when scanner.next() is called it will fetch
> > no
> > > >> > of rows as in *hbase.client.scanner.caching. *When this fetching
> > > >> > process takes more than lease period it will close the scanner
> > object.
> > > >> > so this exception occuring?
> > > >> >
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Rajeshkumar J
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
> > > >> richardstar...@outlook.com
> > > >> >> wrote:
> > > >> >
> > > >> >> It means your lease on a region server has expired during a call
> to
> > > >> >> resultscanner.next(). This happens on a slow call to next(). You
> > can
> > > >> either
> > > >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
> > > >> >> hbase.regionserver.lease.period.
> > > >> >>
> > > >> >> https://richardstartin.com
> > > >> >>
> > > >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J <
> > rajeshkumarit8...@gmail.com
> > > <
> > > >> >> mailto:rajeshkumarit8...@gmail.com>> wrote:
> > > >> >>
> > > >> >> Hi,
> > > >> >>
> > > >> >>  I have faced below issue in our production cluster
> > > >> >>
> > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException:
> > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > '166881'
> > > >> does
> > > >> >> not exist
> > > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > > >> >> removeLease(Leases.java:221)
> > > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > > >> >> cancelLease(Leases.java:206)
> > > >> >> at
> > > >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > > >> >> scan(RSRpcServices.java:2491)
> > > >> >> at
>

Re: Lease exception

2016-12-22 Thread Rajeshkumar J 
can you please explain what is the cause of this lease exception and is
there any solve this in current version

Thanks

On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang  wrote:

> In fact at client the rpc timeout of scan request is also
> hbase.client.scanner.timeout.period which replaces the
> deprecated hbase.regionserver.lease.period.
>
> Your code that throws LeaseException has been removed by HBASE-16604, maybe
> you can try to upgrade your cluster to 1.1.7? Your client can also upgrade
> to 1.1.7 which will ignore UnknowScannerException and retry when the lease
> is expired at server.
>
> Thanks,
> Phil
>
>
> 2016-12-22 16:51 GMT+08:00 Rajeshkumar J :
>
> > Also there is a solution what i have found from hbase user guide that
> > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.
> > period.
> > How these two properties plays a part in the above exception. Please can
> > anyone explain?
> >
> > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J <
> > rajeshkumarit8...@gmail.com>
> > wrote:
> >
> > > I am using hbase version 1.1.1
> > > Also I didn't understand something here. Whenever a scanner.next() is
> > > called it needs to return rows(based on caching value) within leasing
> > > period or else scanner client will be closed eventually throwing this
> > > exception. Correct me as I didn't get the clear understanding of this
> > issue
> > >
> > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu  wrote:
> > >
> > >> Which hbase release are you using ?
> > >>
> > >> There is heartbeat support when scanning.
> > >> Looks like the version you use doesn't have this support.
> > >>
> > >> Cheers
> > >>
> > >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <
> > rajeshkumarit8...@gmail.com>
> > >> wrote:
> > >> >
> > >> > Hi,
> > >> >
> > >> >   Thanks for the reply. I have properties as below
> > >> >
> > >> > 
> > >> >hbase.regionserver.lease.period
> > >> >90
> > >> >  
> > >> >  
> > >> >hbase.rpc.timeout
> > >> >90>/value>
> > >> >  
> > >> >
> > >> >
> > >> > Correct me If I am wrong.
> > >> >
> > >> > I know hbase.regionserver.lease.period, which says how long a
> scanner
> > >> > lives between calls to scanner.next().
> > >> >
> > >> > As far as I understand when scanner.next() is called it will fetch
> no
> > >> > of rows as in *hbase.client.scanner.caching. *When this fetching
> > >> > process takes more than lease period it will close the scanner
> object.
> > >> > so this exception occuring?
> > >> >
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Rajeshkumar J
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
> > >> richardstar...@outlook.com
> > >> >> wrote:
> > >> >
> > >> >> It means your lease on a region server has expired during a call to
> > >> >> resultscanner.next(). This happens on a slow call to next(). You
> can
> > >> either
> > >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
> > >> >> hbase.regionserver.lease.period.
> > >> >>
> > >> >> https://richardstartin.com
> > >> >>
> > >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J <
> rajeshkumarit8...@gmail.com
> > <
> > >> >> mailto:rajeshkumarit8...@gmail.com>> wrote:
> > >> >>
> > >> >> Hi,
> > >> >>
> > >> >>  I have faced below issue in our production cluster
> > >> >>
> > >> >> org.apache.hadoop.hbase.regionserver.LeaseException:
> > >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> '166881'
> > >> does
> > >> >> not exist
> > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > >> >> removeLease(Leases.java:221)
> > >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> > >> >> cancelLease(Leases.java:206)
> > >> >> at
> > >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> > >> >> scan(RSRpcServices.java:2491)
> > >> >> at
> > >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$
> > >> ClientService$2.
> > >> >> callBlockingMethod(ClientProtos.java:32205)
> > >> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> > >> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> > >> >> at
> > >> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec
> > >> utor.java:130)
> > >> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> > java:107)
> > >> >> at java.lang.Thread.run(Thread.java:744)
> > >> >>
> > >> >>
> > >> >> Can any one explain what is lease exception
> > >> >>
> > >> >> Thanks,
> > >> >> Rajeshkumar J
> > >> >>
> > >>
> > >
> > >
> >
>

Re: Lease exception

2016-12-22 Thread Phil Yang 
In fact at client the rpc timeout of scan request is also
hbase.client.scanner.timeout.period which replaces the
deprecated hbase.regionserver.lease.period.

Your code that throws LeaseException has been removed by HBASE-16604, maybe
you can try to upgrade your cluster to 1.1.7? Your client can also upgrade
to 1.1.7 which will ignore UnknowScannerException and retry when the lease
is expired at server.

Thanks,
Phil


2016-12-22 16:51 GMT+08:00 Rajeshkumar J :

> Also there is a solution what i have found from hbase user guide that
> hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.
> period.
> How these two properties plays a part in the above exception. Please can
> anyone explain?
>
> On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com>
> wrote:
>
> > I am using hbase version 1.1.1
> > Also I didn't understand something here. Whenever a scanner.next() is
> > called it needs to return rows(based on caching value) within leasing
> > period or else scanner client will be closed eventually throwing this
> > exception. Correct me as I didn't get the clear understanding of this
> issue
> >
> > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu  wrote:
> >
> >> Which hbase release are you using ?
> >>
> >> There is heartbeat support when scanning.
> >> Looks like the version you use doesn't have this support.
> >>
> >> Cheers
> >>
> >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com>
> >> wrote:
> >> >
> >> > Hi,
> >> >
> >> >   Thanks for the reply. I have properties as below
> >> >
> >> > 
> >> >hbase.regionserver.lease.period
> >> >90
> >> >  
> >> >  
> >> >hbase.rpc.timeout
> >> >90>/value>
> >> >  
> >> >
> >> >
> >> > Correct me If I am wrong.
> >> >
> >> > I know hbase.regionserver.lease.period, which says how long a scanner
> >> > lives between calls to scanner.next().
> >> >
> >> > As far as I understand when scanner.next() is called it will fetch no
> >> > of rows as in *hbase.client.scanner.caching. *When this fetching
> >> > process takes more than lease period it will close the scanner object.
> >> > so this exception occuring?
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Rajeshkumar J
> >> >
> >> >
> >> >
> >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
> >> richardstar...@outlook.com
> >> >> wrote:
> >> >
> >> >> It means your lease on a region server has expired during a call to
> >> >> resultscanner.next(). This happens on a slow call to next(). You can
> >> either
> >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
> >> >> hbase.regionserver.lease.period.
> >> >>
> >> >> https://richardstartin.com
> >> >>
> >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J  <
> >> >> mailto:rajeshkumarit8...@gmail.com>> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >>  I have faced below issue in our production cluster
> >> >>
> >> >> org.apache.hadoop.hbase.regionserver.LeaseException:
> >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881'
> >> does
> >> >> not exist
> >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> >> >> removeLease(Leases.java:221)
> >> >> at org.apache.hadoop.hbase.regionserver.Leases.
> >> >> cancelLease(Leases.java:206)
> >> >> at
> >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
> >> >> scan(RSRpcServices.java:2491)
> >> >> at
> >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$
> >> ClientService$2.
> >> >> callBlockingMethod(ClientProtos.java:32205)
> >> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> >> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> >> >> at
> >> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec
> >> utor.java:130)
> >> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.
> java:107)
> >> >> at java.lang.Thread.run(Thread.java:744)
> >> >>
> >> >>
> >> >> Can any one explain what is lease exception
> >> >>
> >> >> Thanks,
> >> >> Rajeshkumar J
> >> >>
> >>
> >
> >
>

Re: Lease exception

2016-12-22 Thread Rajeshkumar J 
Also there is a solution what i have found from hbase user guide that
hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.period.
How these two properties plays a part in the above exception. Please can
anyone explain?

On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J 
wrote:

> I am using hbase version 1.1.1
> Also I didn't understand something here. Whenever a scanner.next() is
> called it needs to return rows(based on caching value) within leasing
> period or else scanner client will be closed eventually throwing this
> exception. Correct me as I didn't get the clear understanding of this issue
>
> On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu  wrote:
>
>> Which hbase release are you using ?
>>
>> There is heartbeat support when scanning.
>> Looks like the version you use doesn't have this support.
>>
>> Cheers
>>
>> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J 
>> wrote:
>> >
>> > Hi,
>> >
>> >   Thanks for the reply. I have properties as below
>> >
>> > 
>> >hbase.regionserver.lease.period
>> >90
>> >  
>> >  
>> >hbase.rpc.timeout
>> >90>/value>
>> >  
>> >
>> >
>> > Correct me If I am wrong.
>> >
>> > I know hbase.regionserver.lease.period, which says how long a scanner
>> > lives between calls to scanner.next().
>> >
>> > As far as I understand when scanner.next() is called it will fetch no
>> > of rows as in *hbase.client.scanner.caching. *When this fetching
>> > process takes more than lease period it will close the scanner object.
>> > so this exception occuring?
>> >
>> >
>> > Thanks,
>> >
>> > Rajeshkumar J
>> >
>> >
>> >
>> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin <
>> richardstar...@outlook.com
>> >> wrote:
>> >
>> >> It means your lease on a region server has expired during a call to
>> >> resultscanner.next(). This happens on a slow call to next(). You can
>> either
>> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds
>> >> hbase.regionserver.lease.period.
>> >>
>> >> https://richardstartin.com
>> >>
>> >> On 21 Dec 2016, at 11:30, Rajeshkumar J > >> mailto:rajeshkumarit8...@gmail.com>> wrote:
>> >>
>> >> Hi,
>> >>
>> >>  I have faced below issue in our production cluster
>> >>
>> >> org.apache.hadoop.hbase.regionserver.LeaseException:
>> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881'
>> does
>> >> not exist
>> >> at org.apache.hadoop.hbase.regionserver.Leases.
>> >> removeLease(Leases.java:221)
>> >> at org.apache.hadoop.hbase.regionserver.Leases.
>> >> cancelLease(Leases.java:206)
>> >> at
>> >> org.apache.hadoop.hbase.regionserver.RSRpcServices.
>> >> scan(RSRpcServices.java:2491)
>> >> at
>> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$
>> ClientService$2.
>> >> callBlockingMethod(ClientProtos.java:32205)
>> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
>> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>> >> at
>> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec
>> utor.java:130)
>> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
>> >> at java.lang.Thread.run(Thread.java:744)
>> >>
>> >>
>> >> Can any one explain what is lease exception
>> >>
>> >> Thanks,
>> >> Rajeshkumar J
>> >>
>>
>
>