Re: getting start and stop key
I am getting start rowkey as empty and stop rowkey as empty. Is this possible? On Mon, Jun 5, 2017 at 11:19 PM, Ankit Singhal wrote: > Hi RajeshKumar, > > you can try this.(InputSplit class may vary depending upon the InputFormat > you are using, below example is for TableInputFormat) > TableSplit tableSplit = (TableSplit)context.getInputSplit(); > byte[] startRow=tableSplit.getStartRow() > byte[] endRow=tableSplit.getEndRow() > > Regards, > Ankit Singhal > > On Mon, Jun 5, 2017 at 5:38 AM, Rajeshkumar J > > wrote: > > > Hi, > > > > I am running mapreduce over hbase table using java and I want to know > which > > is the startkey and stopkey for every region. As every mapper takes a > > region as input I need to know the startkey and stopkey of a region in > the > > map function. Is there any function to get that or is there any way to > > achieve this > > > > Thanks > > >
getting start and stop key
Hi, I am running mapreduce over hbase table using java and I want to know which is the startkey and stopkey for every region. As every mapper takes a region as input I need to know the startkey and stopkey of a region in the map function. Is there any function to get that or is there any way to achieve this Thanks
Re: input split process
So we will know only start and end rowkey values of a region. We can't know the other rowkey values within that region. On Wed, May 31, 2017 at 2:59 PM, Ted Yu wrote: > If you look at TableInputFormatBase#getSplits(), you would see that the > following information can be retrieved: > > Table > RegionLocator > Admin > StartEndKeys (region boundaries) > > You can also take a look at calculateRebalancedSplits() to see how it > rebalances the InputSplit's. > > FYI > > On Tue, May 30, 2017 at 11:53 PM, Rajeshkumar J < > rajeshkumarit8...@gmail.com > > wrote: > > > Hi, > > > >I want to custom input split my hbase data. can any one tell me what > are > > the values I have known during this split process like only rowkey values > > or any others > > > > Thanks > > >
input split process
Hi, I want to custom input split my hbase data. can any one tell me what are the values I have known during this split process like only rowkey values or any others Thanks
custom input split
Hi, Can anyone point me the custom input split for hbase mapreduce example? Thanks
Re: region files
I have seen the code in that while creating input split they are also sending region info with that splits. Is there any reason for that as all the hfiles are not going to be in that server On Fri, May 26, 2017 at 7:06 PM, Ted Yu wrote: > Consider running major compaction which restores data locality. > > Thanks > > > On May 26, 2017, at 6:08 AM, Rajeshkumar J > wrote: > > > > Thanks Ted. If data blocks of the hfile may not be on the same node as > the > > region server then how data locality is achieved when mapreduce is run > over > > hbase tables > > > > > > > >> On Fri, May 26, 2017 at 6:15 PM, Ted Yu wrote: > >> > >> The hfiles of a region are stored on hdfs. By default, hdfs has > replication > >> factor of 3. > >> If you're not using read replica feature, any single region is served by > >> one region server (however the data blocks of the hfile may not be on > the > >> same node as the region server). > >> > >> Cheers > >> > >> On Thu, May 25, 2017 at 11:45 PM, Rajeshkumar J < > >> rajeshkumarit8...@gmail.com > >>> wrote: > >> > >>> Hi, > >>> > >>> we have region max file size as 10 GB. Whether the hfiles of a region > >>> exists in same region server or will it be distributed? > >>> > >>> Thanks > >> >
Re: region files
Thanks Ted. If data blocks of the hfile may not be on the same node as the region server then how data locality is achieved when mapreduce is run over hbase tables On Fri, May 26, 2017 at 6:15 PM, Ted Yu wrote: > The hfiles of a region are stored on hdfs. By default, hdfs has replication > factor of 3. > If you're not using read replica feature, any single region is served by > one region server (however the data blocks of the hfile may not be on the > same node as the region server). > > Cheers > > On Thu, May 25, 2017 at 11:45 PM, Rajeshkumar J < > rajeshkumarit8...@gmail.com > > wrote: > > > Hi, > > > >we have region max file size as 10 GB. Whether the hfiles of a region > > exists in same region server or will it be distributed? > > > > Thanks > > >
region files
Hi, we have region max file size as 10 GB. Whether the hfiles of a region exists in same region server or will it be distributed? Thanks
hbase input mapper
I am running mapreduce over hbase table. My row key is a combination of Country and place. For instance Delhi_India. Is there any way i can send each value of matched rowkey to separate mappers. For instance if i have rowkey value such as NewYork_USA,Boston_USA,Delhi_India,Mumbai_India. So all the rows which holds rowkey value as USA goes to 1 mapper then India to another mapper.
MultiTableInputFormat class
Hi, I came across MultiTableInputFormat class which convert HBase tabular data from multiple scanners into a format that is consumable by Map/Reduce. So what will be the input to the mapper class if I have passed a scan object to this class which scans two tables. Whether the input to mapper class is the whole data which the scan object returns ?? can any one explain Thanks
hbase input split
Hi, Is there any way to customize hbase input split based on a column value. For instance if i have a column family 'd' with 'country' column qualifiers. 'country' column holds value such as USA,Australia,India,China. So all the rows which holds country value as USA goes to 1 mapper then Australia to another mapper then India to another mapper and likewise for all the values. Thanks
Re: hbase coprocessor in mapreduce
Hi, I want to groupby hbase data by some columns and again I want to group the resultant data by some columns and check the size of each group if greater than 1 it will be written to hdfs. Initially we are doing this in mapreduce but we got some issue in reducer. So we thought we can move this group by operation to hbase using coprocessor. Hence using this coprocessor we will scan and group data by some columns . Then again I want to group the resultant data how to achieve this in coprocessor?? whether this use case fits in coprocessor?? Thanks On Wed, May 17, 2017 at 1:28 PM, Ted Yu wrote: > Can you clarify your use case ? > > If your mapper calls Table#coprocessorService(), the recipient may not be > on the same region server as the mapper. > > On Tue, May 16, 2017 at 10:58 PM, Rajeshkumar J < > rajeshkumarit8...@gmail.com > > wrote: > > > Hi, > > > >whether I can call hase coprocessor in the map function of mapreduce > > job? > > > > Thanks > > >
hbase coprocessor in mapreduce
Hi, whether I can call hase coprocessor in the map function of mapreduce job? Thanks
Re: hbase coprocessor multinode cluster
Sorry I am getting this error in the command line when I try to run the java hbase client code which makes use of Rowcount endpoint coprocessor Thanks On Tue, May 16, 2017 at 7:21 PM, Ted Yu wrote: > Where did you observe this error ? > > Can you pastebin stack trace ? > > Please pastebin region server log as well. > > Thanks > > On Tue, May 16, 2017 at 6:49 AM, Rajeshkumar J < > rajeshkumarit8...@gmail.com> > wrote: > > > It is throwing error like > > > > org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the > > locations > > > > Thanks > > > > On Tue, May 16, 2017 at 6:32 PM, Ted Yu wrote: > > > > > Did you follow this to deploy RowCountEndpoint ? > > > hbase-client/src/main/java/org/apache/hadoop/hbase/ > > > client/coprocessor/package-info.java > > > > > > If the endpoint hangs, can you select region server(s) and capture > stack > > > trace ? > > > > > > Pastebin the traces you capture. > > > > > > Which hbase release are you using ? > > > > > > Cheers > > > > > > On Mon, May 15, 2017 at 10:48 PM, Rajeshkumar J < > > > rajeshkumarit8...@gmail.com > > > > wrote: > > > > > > > Hi, > > > > > > > > I am currently analyzing hbase coprocessor and I tried run > > > > RowCountEndPoint in single node cluster it is working fine. But whn I > > try > > > > to run in multi node cluster it is not throwing error but it hangs in > > the > > > > sense it is not running continuously. I have loaded coprocessor > > > > dynamically. > > > > > > > > Thanks > > > > > > > > > >
Re: hbase coprocessor multinode cluster
It is throwing error like org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations Thanks On Tue, May 16, 2017 at 6:32 PM, Ted Yu wrote: > Did you follow this to deploy RowCountEndpoint ? > hbase-client/src/main/java/org/apache/hadoop/hbase/ > client/coprocessor/package-info.java > > If the endpoint hangs, can you select region server(s) and capture stack > trace ? > > Pastebin the traces you capture. > > Which hbase release are you using ? > > Cheers > > On Mon, May 15, 2017 at 10:48 PM, Rajeshkumar J < > rajeshkumarit8...@gmail.com > > wrote: > > > Hi, > > > > I am currently analyzing hbase coprocessor and I tried run > > RowCountEndPoint in single node cluster it is working fine. But whn I try > > to run in multi node cluster it is not throwing error but it hangs in the > > sense it is not running continuously. I have loaded coprocessor > > dynamically. > > > > Thanks > > >
group by in hbase coprocessor example
Hi, I went through examples of coprocessors in hbase source. Can any one point me the link where I can find group by coprocessor example. Thanks
hbase coprocessor multinode cluster
Hi, I am currently analyzing hbase coprocessor and I tried run RowCountEndPoint in single node cluster it is working fine. But whn I try to run in multi node cluster it is not throwing error but it hangs in the sense it is not running continuously. I have loaded coprocessor dynamically. Thanks
Re: Call function from a coprocessor which extends BaseRegionObserver from the client side
Hi, Check this example in the below link https://www.3pillarglobal.com/insights/hbase-coprocessors Thanks On Mon, May 15, 2017 at 4:00 PM, Cheyenne Forbes < cheyenne.osanu.for...@gmail.com> wrote: > I have a *Coprocessor* which overrides *postPut* and I also want to call a > custom function in this Coprocessor from the hbase client. In hbase 0.94 I > would use "*public class MyCoprocessor extends BaseRegionObserver > implements MyCoprocessorProtocol*" but *CoprocessorProtocol *no longer > exists in hbase 1.2.5. I've done some goggling and found > *CoprocessorService* but I cant find any way to use *CoprocessorService * > with* BaseRegionObserver* > > Regards, > > Cheyenne O. Forbes >
Re: region size for a mapper
Hi, Thanks ted. we are using default split policy and our flush size is 64 MB. And the size is calculated based on the formula Math.min(getDesiredMaxFileSize(),initialSize * tableRegionsCount * tableRegionsCount * tableRegionsCount); If this size exceeds max region size (10 GB), then max region size will be taken or else calculated value will be taken. Suppose if the calculation returns 11 gb then 10 gb will be considered and it will be sent to mapper. Default mapper mapred.map.child.java.opts is 1 gb. what my doubt what if 10gb of data is sent to this mapper with 1 gb whether there will be any issue?? Thanks On Mon, May 15, 2017 at 2:22 PM, Ted Yu wrote: > Split policy may play a role here. > > Please take a look at: > http://hbase.apache.org/book.html#_custom_split_policies > > On Mon, May 15, 2017 at 1:48 AM, Rajeshkumar J < > rajeshkumarit8...@gmail.com> > wrote: > > > Hi, > > > > As we run mapreduce over hbase it will take each region as input for > each > > mapper. I have given region max size as 10GB. If i have about 5 gb will > it > > take 5 gb of data as input of mappers?? > > > > Thanks > > >
region size for a mapper
Hi, As we run mapreduce over hbase it will take each region as input for each mapper. I have given region max size as 10GB. If i have about 5 gb will it take 5 gb of data as input of mappers?? Thanks
mapreduce over hbase table
Hi, I am running a mapreduce job over hbase table. I went through some blogs and in that it is mentioned that data locality will be achieved for this job other than some cases like region reassignment and others. When I run mapreduce job over hbase table it takes each region as input for each mapper. My question is how this job knows which region exists where? can anyone explain this flow Thanks
Re: hbase datalocality while running mapreduce
Hi, so you are saying whenever map reduce is executed mapper which takes each regions as its input will run on the same machine where the region is hosted, If wring correct me Thanks On Tue, May 9, 2017 at 8:32 PM, Kevin O'Dell wrote: > Hi Rajeshkumar, > > When you execute a MR job over an HBase table, the reads should be > local(assuming you have good locality now). This is occurs because writes > are sent to the RegionServer that owns the particular Region. Once the > writes occur the block are written to the local DataNode first. Even if > locality gets messed up through RegionServer failures Major Compactions > should restore it. > > On Tue, May 9, 2017 at 8:00 AM, Rajeshkumar J > > wrote: > > > Hi, > > > > > >I have a table with its region max file size is 10 gb. If i run any > > mapreduce over this table whether there will be any data locality. If not > > please explain > > > > Thanks > > > > > > -- > Kevin O'Dell > Field Engineer > 850-496-1298 | ke...@rocana.com > @kevinrodell > <http://www.rocana.com> >
hbase datalocality while running mapreduce
Hi, I have a table with its region max file size is 10 gb. If i run any mapreduce over this table whether there will be any data locality. If not please explain Thanks
input split for hbase mapreduce
Hi If I am running mapreduce on hbase tables what will be the input to mapper function Thanks
Re: hbase table creation
Also Ashish while specifying region location is there any option to use regular expression? On Thu, Mar 16, 2017 at 5:55 PM, Rajeshkumar J wrote: > thanks ashish. I got that as that region doesn't contain any data and data > is available in other regions. > > On Thu, Mar 16, 2017 at 5:48 PM, ashish singhi > wrote: > >> Was any data added into this table region ? If not then you can skip this >> region directory from completebulkload. >> >> -----Original Message- >> From: Rajeshkumar J [mailto:rajeshkumarit8...@gmail.com] >> Sent: 16 March 2017 17:44 >> To: user@hbase.apache.org >> Subject: Re: hbase table creation >> >> Ashish, >> >> I have tried as u said but I dont have any data in this folder >> >> /hbase/tmp/t1/region1/d >> >> So in the log >> >> 2017-03-16 13:12:40,120 WARN [main] mapreduce.LoadIncrementalHFiles: >> Bulk load operation did not find any files to load in directory >> /hbase/tmp/t1/region1. Does it contain files in subdirectories that >> correspond to column family names? >> >> So is this data corrupted? >> >> >> >> On Thu, Mar 16, 2017 at 5:14 PM, ashish singhi >> wrote: >> >> > Hi, >> > >> > You can try completebulkload tool to load the data into the table. >> > Below is the command usage, >> > >> > hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles >> > >> > usage: completebulkload /path/to/hfileoutputformat-output tablename >> > -Dcreate.table=no - can be used to avoid creation of table by this tool >> > Note: if you set this to 'no', then the target table must already >> > exist in HBase. >> > >> > >> > For example: >> > Consider tablename as t1 you have copied the data of t1 from cluster1 >> > to >> > /hbase/tmp/t1 directory in cluster2 . >> > Delete the recovered.edits directory or any other directory except >> > column family directory(store dir) from the region directory of that >> > table, Suppose you have two regions in the table t1 and list output of >> > table dir is like below >> > >> > ls /hbase/tmp/t1 >> > >> > drwxr-xr-x/hbase/tmp/t1/.tabledesc >> > -rw-r--r--/hbase/tmp/t1/.tabledesc/.tableinfo.01 >> > drwxr-xr-x/hbase/tmp/t1/.tmp >> > drwxr-xr-x/hbase/tmp/t1/region1 >> > -rw-r--r--/hbase/tmp/t1/region1/.regioninfo >> > drwxr-xr-x/hbase/tmp/t1/region1/d >> > -rwxrwxrwx/hbase/tmp/t1/region1/d/0fcaf624cf124d7cab50ace0a6f0f9 >> > df_SeqId_4_ >> > drwxr-xr-x/hbase/tmp/t1/region1/recovered.edits >> > -rw-r--r--/hbase/tmp/t1/region1/recovered.edits/2.seqid >> > drwxr-xr-x/hbase/tmp/t1/region2 >> > -rw-r--r--/hbase/tmp/t1/region2/.regioninfo >> > drwxr-xr-x/hbase/tmp/t1/region2/d >> > -rwxrwxrwx/hbase/tmp/t1/region2/d/14925680d8a5457e9be1c05087f44d >> > f5_SeqId_4_ >> > drwxr-xr-x/hbase/tmp/t1/region2/recovered.edits >> > -rw-r--r--/hbase/tmp/t1/region2/recovered.edits/2.seqid >> > >> > Delete the /hbase/tmp/t1/region1/recovered.edits and >> > /hbase/tmp/t1/region2/recovered.edits >> > >> > And now run the completebulkload for each region like below, >> > >> > 1) hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles >> > /hbase/tmp/t1/region1 t1 >> > 2) hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles >> > /hbase/tmp/t1/region2 t1 >> > >> > Note: The tool will create the table if doesn't exist with only one >> > region. If you want the same table properties as it is in cluster1 >> > then you will have to create it manually in cluster2. >> > >> > I hope this helps. >> > >> > Regards, >> > Ashish >> > >> > -Original Message- >> > From: Rajeshkumar J [mailto:rajeshkumarit8...@gmail.com] >> > Sent: 16 March 2017 16:46 >> > To: user@hbase.apache.org >> > Subject: Re: hbase table creation >> > >> > Karthi, >> > >> >I have mentioned that as of now I dont have any data in that old >> > cluster. Now only have that copied files in the new cluster. I think i >> > can't use this utility? >> > >> > On Thu, Mar 16, 2017 at 4:10 PM, karthi keyan >> > >> > wrote: >> > >> > > Ted- >> > > >> > > Cool !! Will consider hereafter . &
Re: hbase table creation
thanks ashish. I got that as that region doesn't contain any data and data is available in other regions. On Thu, Mar 16, 2017 at 5:48 PM, ashish singhi wrote: > Was any data added into this table region ? If not then you can skip this > region directory from completebulkload. > > -Original Message- > From: Rajeshkumar J [mailto:rajeshkumarit8...@gmail.com] > Sent: 16 March 2017 17:44 > To: user@hbase.apache.org > Subject: Re: hbase table creation > > Ashish, > > I have tried as u said but I dont have any data in this folder > > /hbase/tmp/t1/region1/d > > So in the log > > 2017-03-16 13:12:40,120 WARN [main] mapreduce.LoadIncrementalHFiles: > Bulk load operation did not find any files to load in directory > /hbase/tmp/t1/region1. Does it contain files in subdirectories that > correspond to column family names? > > So is this data corrupted? > > > > On Thu, Mar 16, 2017 at 5:14 PM, ashish singhi > wrote: > > > Hi, > > > > You can try completebulkload tool to load the data into the table. > > Below is the command usage, > > > > hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles > > > > usage: completebulkload /path/to/hfileoutputformat-output tablename > > -Dcreate.table=no - can be used to avoid creation of table by this tool > > Note: if you set this to 'no', then the target table must already > > exist in HBase. > > > > > > For example: > > Consider tablename as t1 you have copied the data of t1 from cluster1 > > to > > /hbase/tmp/t1 directory in cluster2 . > > Delete the recovered.edits directory or any other directory except > > column family directory(store dir) from the region directory of that > > table, Suppose you have two regions in the table t1 and list output of > > table dir is like below > > > > ls /hbase/tmp/t1 > > > > drwxr-xr-x/hbase/tmp/t1/.tabledesc > > -rw-r--r--/hbase/tmp/t1/.tabledesc/.tableinfo.01 > > drwxr-xr-x/hbase/tmp/t1/.tmp > > drwxr-xr-x/hbase/tmp/t1/region1 > > -rw-r--r--/hbase/tmp/t1/region1/.regioninfo > > drwxr-xr-x/hbase/tmp/t1/region1/d > > -rwxrwxrwx/hbase/tmp/t1/region1/d/0fcaf624cf124d7cab50ace0a6f0f9 > > df_SeqId_4_ > > drwxr-xr-x/hbase/tmp/t1/region1/recovered.edits > > -rw-r--r--/hbase/tmp/t1/region1/recovered.edits/2.seqid > > drwxr-xr-x/hbase/tmp/t1/region2 > > -rw-r--r--/hbase/tmp/t1/region2/.regioninfo > > drwxr-xr-x/hbase/tmp/t1/region2/d > > -rwxrwxrwx/hbase/tmp/t1/region2/d/14925680d8a5457e9be1c05087f44d > > f5_SeqId_4_ > > drwxr-xr-x/hbase/tmp/t1/region2/recovered.edits > > -rw-r--r--/hbase/tmp/t1/region2/recovered.edits/2.seqid > > > > Delete the /hbase/tmp/t1/region1/recovered.edits and > > /hbase/tmp/t1/region2/recovered.edits > > > > And now run the completebulkload for each region like below, > > > > 1) hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles > > /hbase/tmp/t1/region1 t1 > > 2) hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles > > /hbase/tmp/t1/region2 t1 > > > > Note: The tool will create the table if doesn't exist with only one > > region. If you want the same table properties as it is in cluster1 > > then you will have to create it manually in cluster2. > > > > I hope this helps. > > > > Regards, > > Ashish > > > > -Original Message- > > From: Rajeshkumar J [mailto:rajeshkumarit8...@gmail.com] > > Sent: 16 March 2017 16:46 > > To: user@hbase.apache.org > > Subject: Re: hbase table creation > > > > Karthi, > > > >I have mentioned that as of now I dont have any data in that old > > cluster. Now only have that copied files in the new cluster. I think i > > can't use this utility? > > > > On Thu, Mar 16, 2017 at 4:10 PM, karthi keyan > > > > wrote: > > > > > Ted- > > > > > > Cool !! Will consider hereafter . > > > > > > On Thu, Mar 16, 2017 at 4:06 PM, Ted Yu wrote: > > > > > > > karthi: > > > > The link you posted was for 0.94 > > > > > > > > We'd better use up-to-date link from refguide (see my previous > reply). > > > > > > > > Cheers > > > > > > > > On Thu, Mar 16, 2017 at 3:26 AM, karthi keyan > > > > > > > > > > > wrote: > > > > > > > > > Rajesh, > > > > > > > > > > Use HBase snapshots for ba
Re: hbase table creation
Ashish, I have tried as u said but I dont have any data in this folder /hbase/tmp/t1/region1/d So in the log 2017-03-16 13:12:40,120 WARN [main] mapreduce.LoadIncrementalHFiles: Bulk load operation did not find any files to load in directory /hbase/tmp/t1/region1. Does it contain files in subdirectories that correspond to column family names? So is this data corrupted? On Thu, Mar 16, 2017 at 5:14 PM, ashish singhi wrote: > Hi, > > You can try completebulkload tool to load the data into the table. Below > is the command usage, > > hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles > > usage: completebulkload /path/to/hfileoutputformat-output tablename > -Dcreate.table=no - can be used to avoid creation of table by this tool > Note: if you set this to 'no', then the target table must already exist > in HBase. > > > For example: > Consider tablename as t1 you have copied the data of t1 from cluster1 to > /hbase/tmp/t1 directory in cluster2 . > Delete the recovered.edits directory or any other directory except column > family directory(store dir) from the region directory of that table, > Suppose you have two regions in the table t1 and list output of table dir > is like below > > ls /hbase/tmp/t1 > > drwxr-xr-x/hbase/tmp/t1/.tabledesc > -rw-r--r--/hbase/tmp/t1/.tabledesc/.tableinfo.01 > drwxr-xr-x/hbase/tmp/t1/.tmp > drwxr-xr-x/hbase/tmp/t1/region1 > -rw-r--r--/hbase/tmp/t1/region1/.regioninfo > drwxr-xr-x/hbase/tmp/t1/region1/d > -rwxrwxrwx/hbase/tmp/t1/region1/d/0fcaf624cf124d7cab50ace0a6f0f9 > df_SeqId_4_ > drwxr-xr-x/hbase/tmp/t1/region1/recovered.edits > -rw-r--r--/hbase/tmp/t1/region1/recovered.edits/2.seqid > drwxr-xr-x/hbase/tmp/t1/region2 > -rw-r--r--/hbase/tmp/t1/region2/.regioninfo > drwxr-xr-x/hbase/tmp/t1/region2/d > -rwxrwxrwx/hbase/tmp/t1/region2/d/14925680d8a5457e9be1c05087f44d > f5_SeqId_4_ > drwxr-xr-x/hbase/tmp/t1/region2/recovered.edits > -rw-r--r--/hbase/tmp/t1/region2/recovered.edits/2.seqid > > Delete the /hbase/tmp/t1/region1/recovered.edits and > /hbase/tmp/t1/region2/recovered.edits > > And now run the completebulkload for each region like below, > > 1) hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles > /hbase/tmp/t1/region1 t1 > 2) hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles > /hbase/tmp/t1/region2 t1 > > Note: The tool will create the table if doesn't exist with only one > region. If you want the same table properties as it is in cluster1 then you > will have to create it manually in cluster2. > > I hope this helps. > > Regards, > Ashish > > -Original Message- > From: Rajeshkumar J [mailto:rajeshkumarit8...@gmail.com] > Sent: 16 March 2017 16:46 > To: user@hbase.apache.org > Subject: Re: hbase table creation > > Karthi, > >I have mentioned that as of now I dont have any data in that old > cluster. Now only have that copied files in the new cluster. I think i > can't use this utility? > > On Thu, Mar 16, 2017 at 4:10 PM, karthi keyan > wrote: > > > Ted- > > > > Cool !! Will consider hereafter . > > > > On Thu, Mar 16, 2017 at 4:06 PM, Ted Yu wrote: > > > > > karthi: > > > The link you posted was for 0.94 > > > > > > We'd better use up-to-date link from refguide (see my previous reply). > > > > > > Cheers > > > > > > On Thu, Mar 16, 2017 at 3:26 AM, karthi keyan > > > > > > > > wrote: > > > > > > > Rajesh, > > > > > > > > Use HBase snapshots for backup and move the data from your " > > > > /hbase/default/data/testing" with its snapshot and clone them to > > > > your destination cluster. > > > > > > > > Snapshot ref link - http://hbase.apache.org/0.94/ > > > book/ops.snapshots.html > > > > <http://hbase.apache.org/0.94/book/ops.snapshots.html> > > > > > > > > > > > > > > > > On Thu, Mar 16, 2017 at 3:51 PM, sudhakara st > > > > > > > > wrote: > > > > > > > > > You have to use 'copytable', here is more info > > > > > https://hbase.apache.org/book.html#copy.table > > > > > > > > > > On Thu, Mar 16, 2017 at 3:46 PM, Rajeshkumar J < > > > > > rajeshkumarit8...@gmail.com> > > > > > wrote: > > > > > > > > > > > I have copied hbase data of a table from one cluster to another. > > For > > > > > > instance I have a table testing and its data will be in the > > > > > > path /hbase/default/data/testing > > > > > > > > > > > > I have copied these files from existing cluster to new > > > > > > cluster. Is > > > > there > > > > > > any possibilty to create table and load data from these files > > > > > > in > > the > > > > new > > > > > > cluster > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Regards, > > > > > ...sudhakara > > > > > > > > > > > > > > >
Re: hbase table creation
Karthi, I have mentioned that as of now I dont have any data in that old cluster. Now only have that copied files in the new cluster. I think i can't use this utility? On Thu, Mar 16, 2017 at 4:10 PM, karthi keyan wrote: > Ted- > > Cool !! Will consider hereafter . > > On Thu, Mar 16, 2017 at 4:06 PM, Ted Yu wrote: > > > karthi: > > The link you posted was for 0.94 > > > > We'd better use up-to-date link from refguide (see my previous reply). > > > > Cheers > > > > On Thu, Mar 16, 2017 at 3:26 AM, karthi keyan > > > wrote: > > > > > Rajesh, > > > > > > Use HBase snapshots for backup and move the data from your " > > > /hbase/default/data/testing" with its snapshot and clone them to your > > > destination cluster. > > > > > > Snapshot ref link - http://hbase.apache.org/0.94/ > > book/ops.snapshots.html > > > <http://hbase.apache.org/0.94/book/ops.snapshots.html> > > > > > > > > > > > > On Thu, Mar 16, 2017 at 3:51 PM, sudhakara st > > > wrote: > > > > > > > You have to use 'copytable', here is more info > > > > https://hbase.apache.org/book.html#copy.table > > > > > > > > On Thu, Mar 16, 2017 at 3:46 PM, Rajeshkumar J < > > > > rajeshkumarit8...@gmail.com> > > > > wrote: > > > > > > > > > I have copied hbase data of a table from one cluster to another. > For > > > > > instance I have a table testing and its data will be in the path > > > > > /hbase/default/data/testing > > > > > > > > > > I have copied these files from existing cluster to new cluster. Is > > > there > > > > > any possibilty to create table and load data from these files in > the > > > new > > > > > cluster > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Regards, > > > > ...sudhakara > > > > > > > > > >
Re: hbase table creation
but my older cluster as of now lost all the data. As of now I only have that files. On Thu, Mar 16, 2017 at 3:51 PM, sudhakara st wrote: > You have to use 'copytable', here is more info > https://hbase.apache.org/book.html#copy.table > > On Thu, Mar 16, 2017 at 3:46 PM, Rajeshkumar J < > rajeshkumarit8...@gmail.com> > wrote: > > > I have copied hbase data of a table from one cluster to another. For > > instance I have a table testing and its data will be in the path > > /hbase/default/data/testing > > > > I have copied these files from existing cluster to new cluster. Is there > > any possibilty to create table and load data from these files in the new > > cluster > > > > > > -- > > Regards, > ...sudhakara >
hbase table creation
I have copied hbase data of a table from one cluster to another. For instance I have a table testing and its data will be in the path /hbase/default/data/testing I have copied these files from existing cluster to new cluster. Is there any possibilty to create table and load data from these files in the new cluster
Re: client side configurations
No I am using a 3 node cluster and I have added these properties in all nodes. Yes i have added values greater than default i.e.,10. And I am running the java class in hbase master On Thu, Mar 2, 2017 at 3:53 PM, Ted Yu wrote: > bq. I have added this property > > On which machine did you add the property ? > If you added on client machine (normally outside the cluster), region > servers wouldn't see them. > > Did you pass timeout values greater than the default ? > > Cheers > > On Wed, Mar 1, 2017 at 11:00 PM, Rajeshkumar J < > rajeshkumarit8...@gmail.com> > wrote: > > > I am trying to scan a table from a class in java. So this is hbase > client. > > I have created hbase configuration using HBaseCOnfiguration.create() in > the > > class and also I have placed my hbase-site.xml in the classpath. My hbase > > services are up and running. Initially I dont have these properties > > 1.hbase.client.scanner.timeout.period > > 2.hbase.rpc.timeout > > > > I have added this property and i tried to run the java class and it has > > hbase-site.xml in classpath so whether these properties are added to both > > hbase client and server side? or only client side and do server side > needs > > restart? > > >
client side configurations
I am trying to scan a table from a class in java. So this is hbase client. I have created hbase configuration using HBaseCOnfiguration.create() in the class and also I have placed my hbase-site.xml in the classpath. My hbase services are up and running. Initially I dont have these properties 1.hbase.client.scanner.timeout.period 2.hbase.rpc.timeout I have added this property and i tried to run the java class and it has hbase-site.xml in classpath so whether these properties are added to both hbase client and server side? or only client side and do server side needs restart?
Re: hbase dynamic configuration
Thanks Jerry for the reply. Also is there any sample command for this dynamic configuration. As I have tried like update_config 'hbase.hstore.blockingStoreFiles','101' but it is throwing error "ERROR: wrong number of arguments (2 for 1)" On Wed, Mar 1, 2017 at 12:55 PM, Jerry He wrote: > These properties can be used on the client side and the server side. > I assume you are asking about them on the server side. > > Unfortunately, these two are not supported yet for dynamic configuration. > > Thanks. > > Jerry > > On Tue, Feb 28, 2017 at 10:00 PM, Rajeshkumar J < > rajeshkumarit8...@gmail.com > > wrote: > > > Hi, > > > >Can any one tell me whether we can add the below properties > dynamically > > > > 1.hbase.client.scanner.timeout.period > > 2.hbase.rpc.timeout > > > > Thanks, > > Rajeshkumar > > >
hbase dynamic configuration
Hi, Can any one tell me whether we can add the below properties dynamically 1.hbase.client.scanner.timeout.period 2.hbase.rpc.timeout Thanks, Rajeshkumar
hbase.client.scanner.caching
Hi, hbase.client.scanner.caching Description Number of rows that we try to fetch when calling next on a scanner if it is not served from (local, client) memory. This configuration works together with hbase.client.scanner.max.result.size to try and use the network efficiently. The default value is Integer.MAX_VALUE by default so that the network will fill the chunk size defined by hbase.client.scanner.max.result.size rather than be limited by a particular number of rows since the size of rows varies table to table. If you know ahead of time that you will not require more than a certain number of rows from a scan, this configuration should be set to that row limit via Scan#setCaching. Higher caching values will enable faster scanners but will eat up more memory and *some calls of next may take longer and longer times when the cache is empty*. Do not set this value such that the time between invocations is greater than the scanner timeout; i.e. hbase.client.scanner.timeout.period Default 2147483647 Can any one explain below lines *some calls of next may take longer and longer times when the cache is empty* Thanks
hbase.client.scanner.caching
Hi, hbase.client.scanner.caching Description Number of rows that we try to fetch when calling next on a scanner if it is not served from (local, client) memory. This configuration works together with hbase.client.scanner.max.result.size to try and use the network efficiently. The default value is Integer.MAX_VALUE by default so that the network will fill the chunk size defined by hbase.client.scanner.max.result.size rather than be limited by a particular number of rows since the size of rows varies table to table. If you know ahead of time that you will not require more than a certain number of rows from a scan, this configuration should be set to that row limit via Scan#setCaching. Higher caching values will enable faster scanners but will eat up more memory and *some calls of next may take longer and longer times when the cache is empty*. Do not set this value such that the time between invocations is greater than the scanner timeout; i.e. hbase.client.scanner.timeout.period Default 2147483647 Can any one explain the lines which are in bold. Thanks
Re: scan performance
Same one but now I think we have found the cause. we have one column qualifier and five columns in every table. We will add singlecolumnvaluefilter to the scan based on the input parameters. All the time scan is successful except when we add that specific column. We have tested other four columns with single column value filter in a single scan and the scan is successful. But when I test with this specific column I am getting this lease exception. Is there any reason? On Thu, Jan 19, 2017 at 6:01 PM, Yu Li wrote: > So the answer in the previous mail thread > <http://mail-archives.apache.org/mod_mbox/hbase-user/ > 201701.mbox/%3CCAM7-19+6mbP5Cdmtoovh4aN5H_GPaFAzy1xt_ > tdl1mum+t-...@mail.gmail.com%3E> > didn't resolve your problem, or this is a new one? If a new one, mind talk > about more details? Thanks. > > Best Regards, > Yu > > On 19 January 2017 at 20:08, Rajeshkumar J > wrote: > > > I am using SingleColumnValueFilter for filtering based on some values. > > Based on this I am getting lease expired exception during scan. So is > there > > any way to solve this? > > >
scan performance
I am using SingleColumnValueFilter for filtering based on some values. Based on this I am getting lease expired exception during scan. So is there any way to solve this?
org.apache.hadoop.hbase.regionserver.LeaseException
This is the log i got 2017-01-05 11:41:49,629 DEBUG [B.defaultRpcServer.handler=15,queue=0,port=16020] ipc.RpcServer: B.defaultRpcServer.handler=15,queue=0,port=16020: callId: 3 service: ClientService methodName: Scan size: 23 connection: xx.xx.xx.xx:x org.apache.hadoop.hbase.regionserver.LeaseException: lease '706' does not exist at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221) at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206) at org.apache.hadoop.hbase.regionserver.RSRpcServices. scan(RSRpcServices.java:2491) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2. callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:744) 2017-01-05 11:41:49,629 TRACE [B.defaultRpcServer.handler=18,queue=0,port=16020] ipc.RpcServer: callId: 2 service: ClientService methodName: Scan size: 29 connection: xx.xx.xx.xx:x param: scanner_id: 706 number_of_rows: 2147483647 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: true connection: xx.xx.xx.xx:x, response scanner_id: 706 more_results: true stale: false more_results_in_region: false queueTime: 1 processingTime: 60136 totalTime: 60137 I have hbase scanner timeout of 6 but here total time is greater than that so I am getting lease exception. can any one suggest me is there any way to find out why it takes this time.
Re: Lease exception
This is the log i got 2017-01-05 11:41:49,629 DEBUG [B.defaultRpcServer.handler=15,queue=0,port=16020] ipc.RpcServer: B.defaultRpcServer.handler=15,queue=0,port=16020: callId: 3 service: ClientService methodName: Scan size: 23 connection: xx.xx.xx.xx:x org.apache.hadoop.hbase.regionserver.LeaseException: lease '706' does not exist at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221) at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2491) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:744) 2017-01-05 11:41:49,629 TRACE [B.defaultRpcServer.handler=18,queue=0,port=16020] ipc.RpcServer: callId: 2 service: ClientService methodName: Scan size: 29 connection: xx.xx.xx.xx:x param: scanner_id: 706 number_of_rows: 2147483647 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: true connection: xx.xx.xx.xx:x, response scanner_id: 706 more_results: true stale: false more_results_in_region: false queueTime: 1 processingTime: 60136 totalTime: 60137 I have hbase scanner timeout of 6 but here total time is greater than that so I am getting lease exception. can any one suggest me is there any way to find out why it takes this time. Thanks On Thu, Dec 22, 2016 at 3:13 PM, Phil Yang wrote: > https://github.com/apache/hbase/blob/rel/1.1.1/hbase- > server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices. > java#L2491 > > There is a TTL for scanners at server, to prevent client don't close the > scanners and they leak. The TTL is configured by > hbase.client.scanner.timeout.period at server and refreshed when a scan > RPC > request comes . The TTLs of all scanners are managed by Lease. Your error > happens when server closes a scanner but in Lease it is already expired. So > I think you can try to increase hbase.client.scanner.timeout.period at > server or decrease your hbase.client.scanner.timeout.period at client to > prevent the scanner expired before its scanning done. > hbase.client.scanner.timeout.period is used both at client and server, may > be different if you change one of sides. > > BTW, I still suggest that you can upgrade your cluster and client. 1.1.1 > has some data-loss bugs on scanning. > > Thanks, > Phil > > > 2016-12-22 17:26 GMT+08:00 Rajeshkumar J : > > > can you please explain what is the cause of this lease exception and is > > there any solve this in current version > > > > Thanks > > > > On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang wrote: > > > > > In fact at client the rpc timeout of scan request is also > > > hbase.client.scanner.timeout.period which replaces the > > > deprecated hbase.regionserver.lease.period. > > > > > > Your code that throws LeaseException has been removed by HBASE-16604, > > maybe > > > you can try to upgrade your cluster to 1.1.7? Your client can also > > upgrade > > > to 1.1.7 which will ignore UnknowScannerException and retry when the > > lease > > > is expired at server. > > > > > > Thanks, > > > Phil > > > > > > > > > 2016-12-22 16:51 GMT+08:00 Rajeshkumar J >: > > > > > > > Also there is a solution what i have found from hbase user guide that > > > > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout. > > > > period. > > > > How these two properties plays a part in the above exception. Please > > can > > > > anyone explain? > > > > > > > > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J < > > > > rajeshkumarit8...@gmail.com> > > > > wrote: > > > > > > > > > I am using hbase version 1.1.1 > > > > > Also I didn't understand something here. Whenever a scanner.next() > is > > > > > called it needs to return rows(based on caching value) within > leasing > > > > > period or else scanner client will be closed eventually throwing > this > > > > > exception. Correct me as I didn't get the clear understanding of > this > > > > issue > > > > > > > > > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Y
Re: How to Configure hbase.client.scanner.timeout.period and hbase.rpc.timeout
this is the log I am getting while running 2017-01-05 11:41:49,629 DEBUG [B.defaultRpcServer.handler=15,queue=0,port=16020] ipc.RpcServer: B.defaultRpcServer.handler=15,queue=0,port=16020: callId: 3 service: ClientService methodName: Scan size: 23 connection: hostname:port org.apache.hadoop.hbase.regionserver.LeaseException: lease '706' does not exist at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221) at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2491) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:744) 2017-01-05 11:41:49,629 TRACE [B.defaultRpcServer.handler=18,queue=0,port=16020] ipc.RpcServer: callId: 2 service: ClientService methodName: Scan size: 29 connection: hostname:port param: scanner_id: 706 number_of_rows: 2147483647 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: true connection: hostname:port, response scanner_id: 706 more_results: true stale: false more_results_in_region: false queueTime: 1 processingTime: 60136 totalTime: 60137 On Wed, Jan 11, 2017 at 4:45 PM, Rajeshkumar J wrote: > I have tried setting hbase.client.scanner.caching to 10, 100, 1000 and > 1. But it still fails. > > On Wed, Jan 11, 2017 at 2:17 PM, Yu Li wrote: > >> Hi Rajeshkumar, >> >> Simply speaking, in your case I'd suggest to set >> hbase.client.scanner.caching >> to a small value like 10 but not zero, and leave the >> hbase.client.scanner.caching or hbase.rpc.timeout setting as default. >> Below >> are some detailed explanation: >> >> Setting hbase.client.scanner.caching to zero will cause the scan returns >> no >> data (please check whether you are getting a wrong result with this >> setting), so please don't make such settings unless you intend to renew >> the >> scan lease. See RsRpcServices#scan source code for more details if >> interested. >> >> For version 1.1.1 (actually after HBASE-16973), the default value of >> hbase.client.scanner.caching >> is Integer.MAX_VALUE, and it won't return (for a single scanner.next call) >> until the result size reached hbase.client.scanner.max.result.size or >> times >> out. If you are using a sparse filter, then it's strongly suggested to >> set hbase.client.scanner.caching >> to some small value but not zero. See HBASE-16973 for more details if >> interested. >> >> Regarding hbase.rpc.timeout and hbase.client.scanner.timeout.period: >> * For scan it's hbase.client.scanner.timeout.period taking effect >> * For all other kinds of non-admin operations like >> get/put/append/increment/multi hbase.rpc.timeout takes effect, unless you >> specified hbase.client.operation.timeout. >> Check ClientScanner#scannerTimeout, ScannerCallable#call, >> AbstractRpcClient#channelOperationTimeout and >> AbstractRpcClient#callBlockingMethod for more details if interested. >> >> Best Regards, >> Yu >> >> On 11 January 2017 at 15:29, Rajeshkumar J >> wrote: >> >> > Hi, As i got lease expire exception i have increased the values of >> these >> > properties hbase.client.scanner.timeout.period and hbase.rpc.timeout. >> Is >> > there any disadvantage increasing these two values? >> > >> > On Mon, Jan 2, 2017 at 11:01 AM, Rajeshkumar J < >> > rajeshkumarit8...@gmail.com> >> > wrote: >> > >> > > I am using hbase 1.1.1. >> > > >> > > I have following property value in hbase-site.xml and I didn't set >> > value >> > > hbase.client.scanner.caching property(so it must pick default value >> for >> > > this) >> > > >> > > >> > > hbase.client.scanner.timeout.period >> > > 6 >> > > >> > > >> > > >> > > hbase.rpc.timeout >> > > 6 >> > > >> > > When scanning 30 tables in hbase(each table holds 45 million records) >> I >> > > got the following error >> > > >> > > WARN client.ScannerCallable [main]: Ignore, probably already closed
Doubt in Scanner.next()
I have hbase.client.scanner.caching as 100. I am scanning a table. For instance say we have 500 rows matching for the current scan. When I give the statement ResultScanner.next() what will happen? whether it will return 100 rows for each next operation or something else Thanks
Re: How to Configure hbase.client.scanner.timeout.period and hbase.rpc.timeout
I have tried setting hbase.client.scanner.caching to 10, 100, 1000 and 1. But it still fails. On Wed, Jan 11, 2017 at 2:17 PM, Yu Li wrote: > Hi Rajeshkumar, > > Simply speaking, in your case I'd suggest to set > hbase.client.scanner.caching > to a small value like 10 but not zero, and leave the > hbase.client.scanner.caching or hbase.rpc.timeout setting as default. Below > are some detailed explanation: > > Setting hbase.client.scanner.caching to zero will cause the scan returns no > data (please check whether you are getting a wrong result with this > setting), so please don't make such settings unless you intend to renew the > scan lease. See RsRpcServices#scan source code for more details if > interested. > > For version 1.1.1 (actually after HBASE-16973), the default value of > hbase.client.scanner.caching > is Integer.MAX_VALUE, and it won't return (for a single scanner.next call) > until the result size reached hbase.client.scanner.max.result.size or > times > out. If you are using a sparse filter, then it's strongly suggested to > set hbase.client.scanner.caching > to some small value but not zero. See HBASE-16973 for more details if > interested. > > Regarding hbase.rpc.timeout and hbase.client.scanner.timeout.period: > * For scan it's hbase.client.scanner.timeout.period taking effect > * For all other kinds of non-admin operations like > get/put/append/increment/multi hbase.rpc.timeout takes effect, unless you > specified hbase.client.operation.timeout. > Check ClientScanner#scannerTimeout, ScannerCallable#call, > AbstractRpcClient#channelOperationTimeout and > AbstractRpcClient#callBlockingMethod for more details if interested. > > Best Regards, > Yu > > On 11 January 2017 at 15:29, Rajeshkumar J > wrote: > > > Hi, As i got lease expire exception i have increased the values of these > > properties hbase.client.scanner.timeout.period and hbase.rpc.timeout. Is > > there any disadvantage increasing these two values? > > > > On Mon, Jan 2, 2017 at 11:01 AM, Rajeshkumar J < > > rajeshkumarit8...@gmail.com> > > wrote: > > > > > I am using hbase 1.1.1. > > > > > > I have following property value in hbase-site.xml and I didn't set > > value > > > hbase.client.scanner.caching property(so it must pick default value for > > > this) > > > > > > > > > hbase.client.scanner.timeout.period > > > 6 > > > > > > > > > > > > hbase.rpc.timeout > > > 6 > > > > > > When scanning 30 tables in hbase(each table holds 45 million records) I > > > got the following error > > > > > > WARN client.ScannerCallable [main]: Ignore, probably already closed > > > org.apache.hadoop.hbase.regionserver.LeaseException: > > > org.apache.hadoop.hbase.regionserver.LeaseException: lease does not > > exist > > > > > > But when I added hbase.client.scanner.caching property value as zero I > > > didn't get any error. Why this happened? > > > Even though I gave zero it supposed to pick the default value? How can > > > this process succeeds? > > > > > > Thanks > > > > > > On Mon, Jan 2, 2017 at 10:44 AM, Josh Elser wrote: > > > > > >> I am having an extremely difficult time parsing this one, Rajeshkumar. > > >> > > >> If you still have a question/something you don't understand, I'd > > >> appreciate it if you could try to re-state it a little more clearly. > > >> > > >> > > >> Rajeshkumar J wrote: > > >> > > >>> I am using hbase 1.1.1. Yes while setting zero it must pick the > default > > >>> value that is INTEGER.MAX_VALUE in hbase 1.1.1. Already we have set > the > > >>> value as INTEGER.MAX_VALUE only but earlier it had failed but now > > >>> succeeded. So only I am asking this. Also I didn't change any > property > > >>> whatever I used in process which failed in the process which > succeeded > > >>> except assigning Zero value for this property > > >>> > > >>> On Thu, Dec 29, 2016 at 1:33 AM, Josh Elser > wrote: > > >>> > > >>> Most likely, since you gave a nonsensical value, HBase used a default > > >>>> value instead of the one you provided. Since you have not shared the > > >>>> version of HBase which you are using, I would recommend that you > look >
Re: How to Configure hbase.client.scanner.timeout.period and hbase.rpc.timeout
Hi, As i got lease expire exception i have increased the values of these properties hbase.client.scanner.timeout.period and hbase.rpc.timeout. Is there any disadvantage increasing these two values? On Mon, Jan 2, 2017 at 11:01 AM, Rajeshkumar J wrote: > I am using hbase 1.1.1. > > I have following property value in hbase-site.xml and I didn't set value > hbase.client.scanner.caching property(so it must pick default value for > this) > > > hbase.client.scanner.timeout.period > 6 > > > > hbase.rpc.timeout > 6 > > When scanning 30 tables in hbase(each table holds 45 million records) I > got the following error > > WARN client.ScannerCallable [main]: Ignore, probably already closed > org.apache.hadoop.hbase.regionserver.LeaseException: > org.apache.hadoop.hbase.regionserver.LeaseException: lease does not exist > > But when I added hbase.client.scanner.caching property value as zero I > didn't get any error. Why this happened? > Even though I gave zero it supposed to pick the default value? How can > this process succeeds? > > Thanks > > On Mon, Jan 2, 2017 at 10:44 AM, Josh Elser wrote: > >> I am having an extremely difficult time parsing this one, Rajeshkumar. >> >> If you still have a question/something you don't understand, I'd >> appreciate it if you could try to re-state it a little more clearly. >> >> >> Rajeshkumar J wrote: >> >>> I am using hbase 1.1.1. Yes while setting zero it must pick the default >>> value that is INTEGER.MAX_VALUE in hbase 1.1.1. Already we have set the >>> value as INTEGER.MAX_VALUE only but earlier it had failed but now >>> succeeded. So only I am asking this. Also I didn't change any property >>> whatever I used in process which failed in the process which succeeded >>> except assigning Zero value for this property >>> >>> On Thu, Dec 29, 2016 at 1:33 AM, Josh Elser wrote: >>> >>> Most likely, since you gave a nonsensical value, HBase used a default >>>> value instead of the one you provided. Since you have not shared the >>>> version of HBase which you are using, I would recommend that you look at >>>> the code. It should be very obvious what value is being used instead of >>>> the >>>> bogus value you provided. >>>> >>>> >>>> Rajeshkumar J wrote: >>>> >>>> I have tried setting hbase.client.scanner.caching property to 0 with >>>>> above >>>>> two values as 6. It runs fine. But hbase.client.scanner.caching >>>>> property tells no of rows to be fetched for every scanner but here I >>>>> have >>>>> given zero. How this worked? >>>>> >>>>> On Tue, Dec 27, 2016 at 8:05 PM, Josh Elser >>>>> wrote: >>>>> >>>>> hbase.client.scanner.timeout.period is a timeout specifically for RPCs >>>>> >>>>>> that come from the HBase Scanner classes (e.g. ClientScanner) while >>>>>> hbase.rpc.timeout is the default timeout for any RPC. I believe that >>>>>> the >>>>>> hbase.client.scanner.timeout.period is also used by the >>>>>> RegionServers to >>>>>> define the lifetime of the Lease (the cause of the LeaseException >>>>>> you're >>>>>> seeing). >>>>>> >>>>>> Generally, when you see these kinds of exceptions while scanning data >>>>>> in >>>>>> HBase, it is just a factor of your hardware and current performance >>>>>> (in >>>>>> other words, how long it takes to read your data). I can't really >>>>>> give a >>>>>> firm answer because it is dependent on your system's performance. You >>>>>> should be able to approximate the performance with some >>>>>> back-of-the-envelope math. >>>>>> >>>>>> Some changes like https://issues.apache.org/jira/browse/HBASE-13090 >>>>>> and >>>>>> https://issues.apache.org/jira/browse/HBASE-1 should help reduce >>>>>> the >>>>>> need for you to tweak configuration properties in the future. >>>>>> >>>>>> >>>>>> Rajeshkumar J wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>>> I have following property value as below >>>>>>> >>>>>>> >>>>>>>hbase.client.scanner.timeout.period >>>>>>>6 >>>>>>> >>>>>>> >>>>>>> >>>>>>>hbase.rpc.timeout >>>>>>>6 >>>>>>> >>>>>>> When scanning 30 tables in hbase(each table holds 45 million >>>>>>> records) I >>>>>>> got >>>>>>> >>>>>>> WARN client.ScannerCallable [main]: Ignore, probably already closed >>>>>>> org.apache.hadoop.hbase.regionserver.LeaseException: >>>>>>> org.apache.hadoop.hbase.regionserver.LeaseException: lease does not >>>>>>> exist >>>>>>> >>>>>>> But when I change the value to >>>>>>> >>>>>>> >>>>>>>hbase.client.scanner.timeout.period >>>>>>>7 >>>>>>> >>>>>>> >>>>>>> >>>>>>>hbase.rpc.timeout >>>>>>>7 >>>>>>> >>>>>>> It runs successfully. >>>>>>> Can any one tell me the reason for this failure and also is there any >>>>>>> factor to calculate these property values for any data. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>> >
Re: RPC calls in hbase
we have 7000 regions in which we only have data in 20 regions. Whether this may be the reason for this? On Thu, Jan 5, 2017 at 10:58 PM, Stack wrote: > On Thu, Jan 5, 2017 at 6:09 AM, Rajeshkumar J > > wrote: > > > Adding this > > Here Clauster have three machines each have 125 GB as RAM in which 70 GB > is > > free, One Machine acts as HMaster and also Region Server. Another machine > > acts as Secondary Hmaster and also Region Server. Third Machine is a > > dedicated Hregion Server. We have about 7000 Regions. > > Maybe is this the problem?? Containing more regions > > > > > Hard to diagnose with so little info. Given the above, you have only a few > (if beefy) machines with more than the usual allotment of regions. > > Suggest you do some background reading and study your cluster while under > load looking at resource usage, metrics, logs and the usual suspects. > > St.Ack > > > > > On Thu, Jan 5, 2017 at 3:27 AM, Stack wrote: > > > > > (Thanks for jumping in Billy) > > > > > > Smaller Scans? (This is good on Scans in 1.1.+ which you seem to be on: > > > https://blogs.apache.org/hbase/entry/scan_improvements_in_hbase_1) > > > > > > Otherwise look at other factors. Are the servers loaded? GC on > > > client/server or resources being taken up by adjacent processes; etc. > > > > > > St.Ack > > > > > > > > > > > > On Wed, Jan 4, 2017 at 11:00 AM, Billy Watson < > williamrwat...@gmail.com> > > > wrote: > > > > > > > Disclaimer: this is what I think I know and I'm not an HBase > > > contributor... > > > > > > > > Anytime we get scanner timeouts like that, we first expand the > timeout > > > > itself like you're doing to fix the short-term problem. But then we > > look > > > at > > > > our rowkey design and what we're doing. Timeouts like that are > usually > > > > symptomatic of bad rowkey utilization, too few servers or too many > > > regions > > > > per server (and therefore too little RAM to serve the regions). > > > > > > > > Make sure you're using good rowkey design and actually taking > advantage > > > of > > > > your rowkey. HBase wasn't really intended to do random scans of > columns > > > > without rowkey filters which is why I think that timeout is set so > low > > by > > > > default. > > > > > > > > This page helps: > > > > > > > > http://hbase.apache.org/0.94/book/rowkey.design.html > > > > > > > > William Watson > > > > Lead Software Engineer > > > > > > > > On Wed, Jan 4, 2017 at 1:41 AM, Rajeshkumar J < > > > rajeshkumarit8...@gmail.com > > > > > > > > > wrote: > > > > > > > > > Hi Stack, > > > > > > > > > >Yes it worked. I had a serious doubt regarding lease does not > > exist > > > > > exception. We have 30 tables which have 45 million records each. We > > are > > > > > scanning the table initially process is successful but after it > gets > > > > failed > > > > > with lease does not exist exception. By changing hbase.rpc.timeout > > and > > > > > hbase.client.scanner.timeout.period it got succeeded. It runs for > > > > sometime > > > > > but again it fails. Then again increasing the value it gets > > succeeded. > > > > > Initially I have default values for these two property 6 and > then > > > > > changed to 7 and then changed to 8. > > > > > > > > > > Can you tell me how to resolve this? > > > > > > > > > > On Wed, Jan 4, 2017 at 11:42 AM, Stack wrote: > > > > > > > > > > > Yeah. Try it. > > > > > > S > > > > > > > > > > > > On Tue, Jan 3, 2017 at 9:18 PM, Rajeshkumar J < > > > > > rajeshkumarit8...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > So I need to add property in log4j.properties as below > > > > > > > > > > > > > > log4j.logger.org.apache.hadoop.hbase.ipc.RpcServer = TRACE > > > > > > > > > > > > > > On Wed, Jan 4, 2017 at 10:23 AM, Stack > wrote: > > > >
Re: RPC calls in hbase
Adding this Here Clauster have three machines each have 125 GB as RAM in which 70 GB is free, One Machine acts as HMaster and also Region Server. Another machine acts as Secondary Hmaster and also Region Server. Third Machine is a dedicated Hregion Server. We have about 7000 Regions. Maybe is this the problem?? Containing more regions On Thu, Jan 5, 2017 at 3:27 AM, Stack wrote: > (Thanks for jumping in Billy) > > Smaller Scans? (This is good on Scans in 1.1.+ which you seem to be on: > https://blogs.apache.org/hbase/entry/scan_improvements_in_hbase_1) > > Otherwise look at other factors. Are the servers loaded? GC on > client/server or resources being taken up by adjacent processes; etc. > > St.Ack > > > > On Wed, Jan 4, 2017 at 11:00 AM, Billy Watson > wrote: > > > Disclaimer: this is what I think I know and I'm not an HBase > contributor... > > > > Anytime we get scanner timeouts like that, we first expand the timeout > > itself like you're doing to fix the short-term problem. But then we look > at > > our rowkey design and what we're doing. Timeouts like that are usually > > symptomatic of bad rowkey utilization, too few servers or too many > regions > > per server (and therefore too little RAM to serve the regions). > > > > Make sure you're using good rowkey design and actually taking advantage > of > > your rowkey. HBase wasn't really intended to do random scans of columns > > without rowkey filters which is why I think that timeout is set so low by > > default. > > > > This page helps: > > > > http://hbase.apache.org/0.94/book/rowkey.design.html > > > > William Watson > > Lead Software Engineer > > > > On Wed, Jan 4, 2017 at 1:41 AM, Rajeshkumar J < > rajeshkumarit8...@gmail.com > > > > > wrote: > > > > > Hi Stack, > > > > > >Yes it worked. I had a serious doubt regarding lease does not exist > > > exception. We have 30 tables which have 45 million records each. We are > > > scanning the table initially process is successful but after it gets > > failed > > > with lease does not exist exception. By changing hbase.rpc.timeout and > > > hbase.client.scanner.timeout.period it got succeeded. It runs for > > sometime > > > but again it fails. Then again increasing the value it gets succeeded. > > > Initially I have default values for these two property 6 and then > > > changed to 7 and then changed to 8. > > > > > > Can you tell me how to resolve this? > > > > > > On Wed, Jan 4, 2017 at 11:42 AM, Stack wrote: > > > > > > > Yeah. Try it. > > > > S > > > > > > > > On Tue, Jan 3, 2017 at 9:18 PM, Rajeshkumar J < > > > rajeshkumarit8...@gmail.com > > > > > > > > > wrote: > > > > > > > > > So I need to add property in log4j.properties as below > > > > > > > > > > log4j.logger.org.apache.hadoop.hbase.ipc.RpcServer = TRACE > > > > > > > > > > On Wed, Jan 4, 2017 at 10:23 AM, Stack wrote: > > > > > > > > > > > See http://hbase.apache.org/book.html#log4j > > > > > > > > > > > > Or to set it temporarily via UI, see > > > > > > http://hbase.apache.org/book.html#trouble.log.levels > > > > > > > > > > > > St.Ack > > > > > > > > > > > > On Tue, Jan 3, 2017 at 6:58 PM, Rajeshkumar J < > > > > > rajeshkumarit8...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Do we have to include this property in hbase-site.xml? > > > > > > > > > > > > > > On Wed, Jan 4, 2017 at 2:12 AM, Stack > wrote: > > > > > > > > > > > > > > > On Mon, Jan 2, 2017 at 10:32 PM, Rajeshkumar J < > > > > > > > > rajeshkumarit8...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Do all rpc calls are logged in hbase region server log? > Also > > I > > > > need > > > > > > to > > > > > > > > find > > > > > > > > > the time taken for each scan calls is this possible > > > > > > > > > > > > > > > > > > If you enable trace level logging for the rpcserver class, > > > > > > > > org.apache.hadoop.hbase.ipc.RpcServer, then you can see a > > bunch > > > of > > > > > > > detail > > > > > > > > on each rpc invocation (including timings, size, etc): > > > > > > > > > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: RPC calls in hbase
Hi Stack, Yes it worked. I had a serious doubt regarding lease does not exist exception. We have 30 tables which have 45 million records each. We are scanning the table initially process is successful but after it gets failed with lease does not exist exception. By changing hbase.rpc.timeout and hbase.client.scanner.timeout.period it got succeeded. It runs for sometime but again it fails. Then again increasing the value it gets succeeded. Initially I have default values for these two property 6 and then changed to 7 and then changed to 8. Can you tell me how to resolve this? On Wed, Jan 4, 2017 at 11:42 AM, Stack wrote: > Yeah. Try it. > S > > On Tue, Jan 3, 2017 at 9:18 PM, Rajeshkumar J > > wrote: > > > So I need to add property in log4j.properties as below > > > > log4j.logger.org.apache.hadoop.hbase.ipc.RpcServer = TRACE > > > > On Wed, Jan 4, 2017 at 10:23 AM, Stack wrote: > > > > > See http://hbase.apache.org/book.html#log4j > > > > > > Or to set it temporarily via UI, see > > > http://hbase.apache.org/book.html#trouble.log.levels > > > > > > St.Ack > > > > > > On Tue, Jan 3, 2017 at 6:58 PM, Rajeshkumar J < > > rajeshkumarit8...@gmail.com > > > > > > > wrote: > > > > > > > Do we have to include this property in hbase-site.xml? > > > > > > > > On Wed, Jan 4, 2017 at 2:12 AM, Stack wrote: > > > > > > > > > On Mon, Jan 2, 2017 at 10:32 PM, Rajeshkumar J < > > > > > rajeshkumarit8...@gmail.com> > > > > > wrote: > > > > > > > > > > > Do all rpc calls are logged in hbase region server log? Also I > need > > > to > > > > > find > > > > > > the time taken for each scan calls is this possible > > > > > > > > > > > > If you enable trace level logging for the rpcserver class, > > > > > org.apache.hadoop.hbase.ipc.RpcServer, then you can see a bunch of > > > > detail > > > > > on each rpc invocation (including timings, size, etc): > > > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > >
Re: RPC calls in hbase
So I need to add property in log4j.properties as below log4j.logger.org.apache.hadoop.hbase.ipc.RpcServer = TRACE On Wed, Jan 4, 2017 at 10:23 AM, Stack wrote: > See http://hbase.apache.org/book.html#log4j > > Or to set it temporarily via UI, see > http://hbase.apache.org/book.html#trouble.log.levels > > St.Ack > > On Tue, Jan 3, 2017 at 6:58 PM, Rajeshkumar J > > wrote: > > > Do we have to include this property in hbase-site.xml? > > > > On Wed, Jan 4, 2017 at 2:12 AM, Stack wrote: > > > > > On Mon, Jan 2, 2017 at 10:32 PM, Rajeshkumar J < > > > rajeshkumarit8...@gmail.com> > > > wrote: > > > > > > > Do all rpc calls are logged in hbase region server log? Also I need > to > > > find > > > > the time taken for each scan calls is this possible > > > > > > > > If you enable trace level logging for the rpcserver class, > > > org.apache.hadoop.hbase.ipc.RpcServer, then you can see a bunch of > > detail > > > on each rpc invocation (including timings, size, etc): > > > > > > St.Ack > > > > > > > > > > > > > > > > Thanks > > > > > > > > > >
Re: RPC calls in hbase
Do we have to include this property in hbase-site.xml? On Wed, Jan 4, 2017 at 2:12 AM, Stack wrote: > On Mon, Jan 2, 2017 at 10:32 PM, Rajeshkumar J < > rajeshkumarit8...@gmail.com> > wrote: > > > Do all rpc calls are logged in hbase region server log? Also I need to > find > > the time taken for each scan calls is this possible > > > > If you enable trace level logging for the rpcserver class, > org.apache.hadoop.hbase.ipc.RpcServer, then you can see a bunch of detail > on each rpc invocation (including timings, size, etc): > > St.Ack > > > > > > Thanks > > >
RPC calls in hbase
Do all rpc calls are logged in hbase region server log? Also I need to find the time taken for each scan calls is this possible Thanks
Re: How to Configure hbase.client.scanner.timeout.period and hbase.rpc.timeout
I am using hbase 1.1.1. I have following property value in hbase-site.xml and I didn't set value hbase.client.scanner.caching property(so it must pick default value for this) hbase.client.scanner.timeout.period 6 hbase.rpc.timeout 6 When scanning 30 tables in hbase(each table holds 45 million records) I got the following error WARN client.ScannerCallable [main]: Ignore, probably already closed org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease does not exist But when I added hbase.client.scanner.caching property value as zero I didn't get any error. Why this happened? Even though I gave zero it supposed to pick the default value? How can this process succeeds? Thanks On Mon, Jan 2, 2017 at 10:44 AM, Josh Elser wrote: > I am having an extremely difficult time parsing this one, Rajeshkumar. > > If you still have a question/something you don't understand, I'd > appreciate it if you could try to re-state it a little more clearly. > > > Rajeshkumar J wrote: > >> I am using hbase 1.1.1. Yes while setting zero it must pick the default >> value that is INTEGER.MAX_VALUE in hbase 1.1.1. Already we have set the >> value as INTEGER.MAX_VALUE only but earlier it had failed but now >> succeeded. So only I am asking this. Also I didn't change any property >> whatever I used in process which failed in the process which succeeded >> except assigning Zero value for this property >> >> On Thu, Dec 29, 2016 at 1:33 AM, Josh Elser wrote: >> >> Most likely, since you gave a nonsensical value, HBase used a default >>> value instead of the one you provided. Since you have not shared the >>> version of HBase which you are using, I would recommend that you look at >>> the code. It should be very obvious what value is being used instead of >>> the >>> bogus value you provided. >>> >>> >>> Rajeshkumar J wrote: >>> >>> I have tried setting hbase.client.scanner.caching property to 0 with >>>> above >>>> two values as 6. It runs fine. But hbase.client.scanner.caching >>>> property tells no of rows to be fetched for every scanner but here I >>>> have >>>> given zero. How this worked? >>>> >>>> On Tue, Dec 27, 2016 at 8:05 PM, Josh Elser wrote: >>>> >>>> hbase.client.scanner.timeout.period is a timeout specifically for RPCs >>>> >>>>> that come from the HBase Scanner classes (e.g. ClientScanner) while >>>>> hbase.rpc.timeout is the default timeout for any RPC. I believe that >>>>> the >>>>> hbase.client.scanner.timeout.period is also used by the RegionServers >>>>> to >>>>> define the lifetime of the Lease (the cause of the LeaseException >>>>> you're >>>>> seeing). >>>>> >>>>> Generally, when you see these kinds of exceptions while scanning data >>>>> in >>>>> HBase, it is just a factor of your hardware and current performance (in >>>>> other words, how long it takes to read your data). I can't really give >>>>> a >>>>> firm answer because it is dependent on your system's performance. You >>>>> should be able to approximate the performance with some >>>>> back-of-the-envelope math. >>>>> >>>>> Some changes like https://issues.apache.org/jira/browse/HBASE-13090 >>>>> and >>>>> https://issues.apache.org/jira/browse/HBASE-1 should help reduce >>>>> the >>>>> need for you to tweak configuration properties in the future. >>>>> >>>>> >>>>> Rajeshkumar J wrote: >>>>> >>>>> Hi, >>>>> >>>>>> I have following property value as below >>>>>> >>>>>> >>>>>>hbase.client.scanner.timeout.period >>>>>>6 >>>>>> >>>>>> >>>>>> >>>>>>hbase.rpc.timeout >>>>>>6 >>>>>> >>>>>> When scanning 30 tables in hbase(each table holds 45 million records) >>>>>> I >>>>>> got >>>>>> >>>>>> WARN client.ScannerCallable [main]: Ignore, probably already closed >>>>>> org.apache.hadoop.hbase.regionserver.LeaseException: >>>>>> org.apache.hadoop.hbase.regionserver.LeaseException: lease does not >>>>>> exist >>>>>> >>>>>> But when I change the value to >>>>>> >>>>>> >>>>>>hbase.client.scanner.timeout.period >>>>>>7 >>>>>> >>>>>> >>>>>> >>>>>>hbase.rpc.timeout >>>>>>7 >>>>>> >>>>>> It runs successfully. >>>>>> Can any one tell me the reason for this failure and also is there any >>>>>> factor to calculate these property values for any data. >>>>>> >>>>>> >>>>>> >>>>>> >>
Re: How to Configure hbase.client.scanner.timeout.period and hbase.rpc.timeout
I am using hbase 1.1.1. Yes while setting zero it must pick the default value that is INTEGER.MAX_VALUE in hbase 1.1.1. Already we have set the value as INTEGER.MAX_VALUE only but earlier it had failed but now succeeded. So only I am asking this. Also I didn't change any property whatever I used in process which failed in the process which succeeded except assigning Zero value for this property On Thu, Dec 29, 2016 at 1:33 AM, Josh Elser wrote: > Most likely, since you gave a nonsensical value, HBase used a default > value instead of the one you provided. Since you have not shared the > version of HBase which you are using, I would recommend that you look at > the code. It should be very obvious what value is being used instead of the > bogus value you provided. > > > Rajeshkumar J wrote: > >> I have tried setting hbase.client.scanner.caching property to 0 with above >> two values as 6. It runs fine. But hbase.client.scanner.caching >> property tells no of rows to be fetched for every scanner but here I have >> given zero. How this worked? >> >> On Tue, Dec 27, 2016 at 8:05 PM, Josh Elser wrote: >> >> hbase.client.scanner.timeout.period is a timeout specifically for RPCs >>> that come from the HBase Scanner classes (e.g. ClientScanner) while >>> hbase.rpc.timeout is the default timeout for any RPC. I believe that the >>> hbase.client.scanner.timeout.period is also used by the RegionServers to >>> define the lifetime of the Lease (the cause of the LeaseException you're >>> seeing). >>> >>> Generally, when you see these kinds of exceptions while scanning data in >>> HBase, it is just a factor of your hardware and current performance (in >>> other words, how long it takes to read your data). I can't really give a >>> firm answer because it is dependent on your system's performance. You >>> should be able to approximate the performance with some >>> back-of-the-envelope math. >>> >>> Some changes like https://issues.apache.org/jira/browse/HBASE-13090 and >>> https://issues.apache.org/jira/browse/HBASE-1 should help reduce the >>> need for you to tweak configuration properties in the future. >>> >>> >>> Rajeshkumar J wrote: >>> >>> Hi, >>>>I have following property value as below >>>> >>>> >>>> hbase.client.scanner.timeout.period >>>> 6 >>>> >>>> >>>> >>>> hbase.rpc.timeout >>>> 6 >>>> >>>> When scanning 30 tables in hbase(each table holds 45 million records) I >>>> got >>>> >>>> WARN client.ScannerCallable [main]: Ignore, probably already closed >>>> org.apache.hadoop.hbase.regionserver.LeaseException: >>>> org.apache.hadoop.hbase.regionserver.LeaseException: lease does not >>>> exist >>>> >>>> But when I change the value to >>>> >>>> >>>> hbase.client.scanner.timeout.period >>>> 7 >>>> >>>> >>>> >>>> hbase.rpc.timeout >>>> 7 >>>> >>>> It runs successfully. >>>> Can any one tell me the reason for this failure and also is there any >>>> factor to calculate these property values for any data. >>>> >>>> >>>> >>
Re: Internals of hbase scan
Thanks for ur reply. Whenever I issue ResultScanner.next() can you point me the class where will be the control will switch to? On Wed, Dec 28, 2016 at 6:04 PM, Ted Yu wrote: > You can start from http://hbase.apache.org/book.html#hregion.scans > > To get to know internals, you should look at the code - in IDE such as > Eclipse. > Start from StoreScanner and read the classes which reference it. > > Cheers > > On Wed, Dec 28, 2016 at 12:59 AM, Rajeshkumar J < > rajeshkumarit8...@gmail.com > > wrote: > > > Can anyone point me where I can learn internals of hbase such as scan in > > depth. > > >
Internals of hbase scan
Can anyone point me where I can learn internals of hbase such as scan in depth.
Re: How to Configure hbase.client.scanner.timeout.period and hbase.rpc.timeout
I have tried setting hbase.client.scanner.caching property to 0 with above two values as 6. It runs fine. But hbase.client.scanner.caching property tells no of rows to be fetched for every scanner but here I have given zero. How this worked? On Tue, Dec 27, 2016 at 8:05 PM, Josh Elser wrote: > hbase.client.scanner.timeout.period is a timeout specifically for RPCs > that come from the HBase Scanner classes (e.g. ClientScanner) while > hbase.rpc.timeout is the default timeout for any RPC. I believe that the > hbase.client.scanner.timeout.period is also used by the RegionServers to > define the lifetime of the Lease (the cause of the LeaseException you're > seeing). > > Generally, when you see these kinds of exceptions while scanning data in > HBase, it is just a factor of your hardware and current performance (in > other words, how long it takes to read your data). I can't really give a > firm answer because it is dependent on your system's performance. You > should be able to approximate the performance with some > back-of-the-envelope math. > > Some changes like https://issues.apache.org/jira/browse/HBASE-13090 and > https://issues.apache.org/jira/browse/HBASE-1 should help reduce the > need for you to tweak configuration properties in the future. > > > Rajeshkumar J wrote: > >> Hi, >> I have following property value as below >> >> >> hbase.client.scanner.timeout.period >> 6 >> >> >> >> hbase.rpc.timeout >> 6 >> >> When scanning 30 tables in hbase(each table holds 45 million records) I >> got >> >> WARN client.ScannerCallable [main]: Ignore, probably already closed >> org.apache.hadoop.hbase.regionserver.LeaseException: >> org.apache.hadoop.hbase.regionserver.LeaseException: lease does not exist >> >> But when I change the value to >> >> >> hbase.client.scanner.timeout.period >> 7 >> >> >> >> hbase.rpc.timeout >> 7 >> >> It runs successfully. >> Can any one tell me the reason for this failure and also is there any >> factor to calculate these property values for any data. >> >>
How to Configure hbase.client.scanner.timeout.period and hbase.rpc.timeout
Hi, I have following property value as below hbase.client.scanner.timeout.period 6 hbase.rpc.timeout 6 When scanning 30 tables in hbase(each table holds 45 million records) I got WARN client.ScannerCallable [main]: Ignore, probably already closed org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease does not exist But when I change the value to hbase.client.scanner.timeout.period 7 hbase.rpc.timeout 7 It runs successfully. Can any one tell me the reason for this failure and also is there any factor to calculate these property values for any data.
Re: Lease exception
Also how to change this property hbase.client.scanner.timeout.period in client side as I only know that to change this property in hbase-site.xml On Thu, Dec 22, 2016 at 3:13 PM, Phil Yang wrote: > https://github.com/apache/hbase/blob/rel/1.1.1/hbase- > server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices. > java#L2491 > > There is a TTL for scanners at server, to prevent client don't close the > scanners and they leak. The TTL is configured by > hbase.client.scanner.timeout.period at server and refreshed when a scan > RPC > request comes . The TTLs of all scanners are managed by Lease. Your error > happens when server closes a scanner but in Lease it is already expired. So > I think you can try to increase hbase.client.scanner.timeout.period at > server or decrease your hbase.client.scanner.timeout.period at client to > prevent the scanner expired before its scanning done. > hbase.client.scanner.timeout.period is used both at client and server, may > be different if you change one of sides. > > BTW, I still suggest that you can upgrade your cluster and client. 1.1.1 > has some data-loss bugs on scanning. > > Thanks, > Phil > > > 2016-12-22 17:26 GMT+08:00 Rajeshkumar J : > > > can you please explain what is the cause of this lease exception and is > > there any solve this in current version > > > > Thanks > > > > On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang wrote: > > > > > In fact at client the rpc timeout of scan request is also > > > hbase.client.scanner.timeout.period which replaces the > > > deprecated hbase.regionserver.lease.period. > > > > > > Your code that throws LeaseException has been removed by HBASE-16604, > > maybe > > > you can try to upgrade your cluster to 1.1.7? Your client can also > > upgrade > > > to 1.1.7 which will ignore UnknowScannerException and retry when the > > lease > > > is expired at server. > > > > > > Thanks, > > > Phil > > > > > > > > > 2016-12-22 16:51 GMT+08:00 Rajeshkumar J >: > > > > > > > Also there is a solution what i have found from hbase user guide that > > > > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout. > > > > period. > > > > How these two properties plays a part in the above exception. Please > > can > > > > anyone explain? > > > > > > > > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J < > > > > rajeshkumarit8...@gmail.com> > > > > wrote: > > > > > > > > > I am using hbase version 1.1.1 > > > > > Also I didn't understand something here. Whenever a scanner.next() > is > > > > > called it needs to return rows(based on caching value) within > leasing > > > > > period or else scanner client will be closed eventually throwing > this > > > > > exception. Correct me as I didn't get the clear understanding of > this > > > > issue > > > > > > > > > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu > wrote: > > > > > > > > > >> Which hbase release are you using ? > > > > >> > > > > >> There is heartbeat support when scanning. > > > > >> Looks like the version you use doesn't have this support. > > > > >> > > > > >> Cheers > > > > >> > > > > >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J < > > > > rajeshkumarit8...@gmail.com> > > > > >> wrote: > > > > >> > > > > > >> > Hi, > > > > >> > > > > > >> > Thanks for the reply. I have properties as below > > > > >> > > > > > >> > > > > > >> >hbase.regionserver.lease.period > > > > >> >90 > > > > >> > > > > > >> > > > > > >> >hbase.rpc.timeout > > > > >> >90>/value> > > > > >> > > > > > >> > > > > > >> > > > > > >> > Correct me If I am wrong. > > > > >> > > > > > >> > I know hbase.regionserver.lease.period, which says how long a > > > scanner > > > > >> > lives between calls to scanner.next(). > > > > >> > > > > > >> >
Re: Lease exception
sorry for the delay. I didn't get the lease concept here whether it is specific to hbase or like lease in hadoop? On Thu, Dec 22, 2016 at 3:13 PM, Phil Yang wrote: > https://github.com/apache/hbase/blob/rel/1.1.1/hbase- > server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices. > java#L2491 > > There is a TTL for scanners at server, to prevent client don't close the > scanners and they leak. The TTL is configured by > hbase.client.scanner.timeout.period at server and refreshed when a scan > RPC > request comes . The TTLs of all scanners are managed by Lease. Your error > happens when server closes a scanner but in Lease it is already expired. So > I think you can try to increase hbase.client.scanner.timeout.period at > server or decrease your hbase.client.scanner.timeout.period at client to > prevent the scanner expired before its scanning done. > hbase.client.scanner.timeout.period is used both at client and server, may > be different if you change one of sides. > > BTW, I still suggest that you can upgrade your cluster and client. 1.1.1 > has some data-loss bugs on scanning. > > Thanks, > Phil > > > 2016-12-22 17:26 GMT+08:00 Rajeshkumar J : > > > can you please explain what is the cause of this lease exception and is > > there any solve this in current version > > > > Thanks > > > > On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang wrote: > > > > > In fact at client the rpc timeout of scan request is also > > > hbase.client.scanner.timeout.period which replaces the > > > deprecated hbase.regionserver.lease.period. > > > > > > Your code that throws LeaseException has been removed by HBASE-16604, > > maybe > > > you can try to upgrade your cluster to 1.1.7? Your client can also > > upgrade > > > to 1.1.7 which will ignore UnknowScannerException and retry when the > > lease > > > is expired at server. > > > > > > Thanks, > > > Phil > > > > > > > > > 2016-12-22 16:51 GMT+08:00 Rajeshkumar J >: > > > > > > > Also there is a solution what i have found from hbase user guide that > > > > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout. > > > > period. > > > > How these two properties plays a part in the above exception. Please > > can > > > > anyone explain? > > > > > > > > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J < > > > > rajeshkumarit8...@gmail.com> > > > > wrote: > > > > > > > > > I am using hbase version 1.1.1 > > > > > Also I didn't understand something here. Whenever a scanner.next() > is > > > > > called it needs to return rows(based on caching value) within > leasing > > > > > period or else scanner client will be closed eventually throwing > this > > > > > exception. Correct me as I didn't get the clear understanding of > this > > > > issue > > > > > > > > > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu > wrote: > > > > > > > > > >> Which hbase release are you using ? > > > > >> > > > > >> There is heartbeat support when scanning. > > > > >> Looks like the version you use doesn't have this support. > > > > >> > > > > >> Cheers > > > > >> > > > > >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J < > > > > rajeshkumarit8...@gmail.com> > > > > >> wrote: > > > > >> > > > > > >> > Hi, > > > > >> > > > > > >> > Thanks for the reply. I have properties as below > > > > >> > > > > > >> > > > > > >> >hbase.regionserver.lease.period > > > > >> >90 > > > > >> > > > > > >> > > > > > >> >hbase.rpc.timeout > > > > >> >90>/value> > > > > >> > > > > > >> > > > > > >> > > > > > >> > Correct me If I am wrong. > > > > >> > > > > > >> > I know hbase.regionserver.lease.period, which says how long a > > > scanner > > > > >> > lives between calls to scanner.next(). > > > > >> > > > > > >> > As far as I understand when scanner.next() is called it will > fetch &
Re: Lease exception
can you please explain what is the cause of this lease exception and is there any solve this in current version Thanks On Thu, Dec 22, 2016 at 2:54 PM, Phil Yang wrote: > In fact at client the rpc timeout of scan request is also > hbase.client.scanner.timeout.period which replaces the > deprecated hbase.regionserver.lease.period. > > Your code that throws LeaseException has been removed by HBASE-16604, maybe > you can try to upgrade your cluster to 1.1.7? Your client can also upgrade > to 1.1.7 which will ignore UnknowScannerException and retry when the lease > is expired at server. > > Thanks, > Phil > > > 2016-12-22 16:51 GMT+08:00 Rajeshkumar J : > > > Also there is a solution what i have found from hbase user guide that > > hbase.rpc.timeout must be greater than hbase.client.scanner.timeout. > > period. > > How these two properties plays a part in the above exception. Please can > > anyone explain? > > > > On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J < > > rajeshkumarit8...@gmail.com> > > wrote: > > > > > I am using hbase version 1.1.1 > > > Also I didn't understand something here. Whenever a scanner.next() is > > > called it needs to return rows(based on caching value) within leasing > > > period or else scanner client will be closed eventually throwing this > > > exception. Correct me as I didn't get the clear understanding of this > > issue > > > > > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu wrote: > > > > > >> Which hbase release are you using ? > > >> > > >> There is heartbeat support when scanning. > > >> Looks like the version you use doesn't have this support. > > >> > > >> Cheers > > >> > > >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J < > > rajeshkumarit8...@gmail.com> > > >> wrote: > > >> > > > >> > Hi, > > >> > > > >> > Thanks for the reply. I have properties as below > > >> > > > >> > > > >> >hbase.regionserver.lease.period > > >> >90 > > >> > > > >> > > > >> >hbase.rpc.timeout > > >> >90>/value> > > >> > > > >> > > > >> > > > >> > Correct me If I am wrong. > > >> > > > >> > I know hbase.regionserver.lease.period, which says how long a > scanner > > >> > lives between calls to scanner.next(). > > >> > > > >> > As far as I understand when scanner.next() is called it will fetch > no > > >> > of rows as in *hbase.client.scanner.caching. *When this fetching > > >> > process takes more than lease period it will close the scanner > object. > > >> > so this exception occuring? > > >> > > > >> > > > >> > Thanks, > > >> > > > >> > Rajeshkumar J > > >> > > > >> > > > >> > > > >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin < > > >> richardstar...@outlook.com > > >> >> wrote: > > >> > > > >> >> It means your lease on a region server has expired during a call to > > >> >> resultscanner.next(). This happens on a slow call to next(). You > can > > >> either > > >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds > > >> >> hbase.regionserver.lease.period. > > >> >> > > >> >> https://richardstartin.com > > >> >> > > >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J < > rajeshkumarit8...@gmail.com > > < > > >> >> mailto:rajeshkumarit8...@gmail.com>> wrote: > > >> >> > > >> >> Hi, > > >> >> > > >> >> I have faced below issue in our production cluster > > >> >> > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException: > > >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease > '166881' > > >> does > > >> >> not exist > > >> >> at org.apache.hadoop.hbase.regionserver.Leases. > > >> >> removeLease(Leases.java:221) > > >> >> at org.apache.hadoop.hbase.regionserver.Leases. > > >> >> cancelLease(Leases.java:206) > > >> >> at > > >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices. > > >> >> scan(RSRpcServices.java:2491) > > >> >> at > > >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ > > >> ClientService$2. > > >> >> callBlockingMethod(ClientProtos.java:32205) > > >> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > > >> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > > >> >> at > > >> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec > > >> utor.java:130) > > >> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor. > > java:107) > > >> >> at java.lang.Thread.run(Thread.java:744) > > >> >> > > >> >> > > >> >> Can any one explain what is lease exception > > >> >> > > >> >> Thanks, > > >> >> Rajeshkumar J > > >> >> > > >> > > > > > > > > >
Re: Lease exception
Also there is a solution what i have found from hbase user guide that hbase.rpc.timeout must be greater than hbase.client.scanner.timeout.period. How these two properties plays a part in the above exception. Please can anyone explain? On Wed, Dec 21, 2016 at 9:39 PM, Rajeshkumar J wrote: > I am using hbase version 1.1.1 > Also I didn't understand something here. Whenever a scanner.next() is > called it needs to return rows(based on caching value) within leasing > period or else scanner client will be closed eventually throwing this > exception. Correct me as I didn't get the clear understanding of this issue > > On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu wrote: > >> Which hbase release are you using ? >> >> There is heartbeat support when scanning. >> Looks like the version you use doesn't have this support. >> >> Cheers >> >> > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J >> wrote: >> > >> > Hi, >> > >> > Thanks for the reply. I have properties as below >> > >> > >> >hbase.regionserver.lease.period >> >90 >> > >> > >> >hbase.rpc.timeout >> >90>/value> >> > >> > >> > >> > Correct me If I am wrong. >> > >> > I know hbase.regionserver.lease.period, which says how long a scanner >> > lives between calls to scanner.next(). >> > >> > As far as I understand when scanner.next() is called it will fetch no >> > of rows as in *hbase.client.scanner.caching. *When this fetching >> > process takes more than lease period it will close the scanner object. >> > so this exception occuring? >> > >> > >> > Thanks, >> > >> > Rajeshkumar J >> > >> > >> > >> > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin < >> richardstar...@outlook.com >> >> wrote: >> > >> >> It means your lease on a region server has expired during a call to >> >> resultscanner.next(). This happens on a slow call to next(). You can >> either >> >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds >> >> hbase.regionserver.lease.period. >> >> >> >> https://richardstartin.com >> >> >> >> On 21 Dec 2016, at 11:30, Rajeshkumar J > >> mailto:rajeshkumarit8...@gmail.com>> wrote: >> >> >> >> Hi, >> >> >> >> I have faced below issue in our production cluster >> >> >> >> org.apache.hadoop.hbase.regionserver.LeaseException: >> >> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' >> does >> >> not exist >> >> at org.apache.hadoop.hbase.regionserver.Leases. >> >> removeLease(Leases.java:221) >> >> at org.apache.hadoop.hbase.regionserver.Leases. >> >> cancelLease(Leases.java:206) >> >> at >> >> org.apache.hadoop.hbase.regionserver.RSRpcServices. >> >> scan(RSRpcServices.java:2491) >> >> at >> >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ >> ClientService$2. >> >> callBlockingMethod(ClientProtos.java:32205) >> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) >> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) >> >> at >> >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExec >> utor.java:130) >> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) >> >> at java.lang.Thread.run(Thread.java:744) >> >> >> >> >> >> Can any one explain what is lease exception >> >> >> >> Thanks, >> >> Rajeshkumar J >> >> >> > >
Re: Lease exception
I am using hbase version 1.1.1 Also I didn't understand something here. Whenever a scanner.next() is called it needs to return rows(based on caching value) within leasing period or else scanner client will be closed eventually throwing this exception. Correct me as I didn't get the clear understanding of this issue On Wed, Dec 21, 2016 at 7:31 PM, Ted Yu wrote: > Which hbase release are you using ? > > There is heartbeat support when scanning. > Looks like the version you use doesn't have this support. > > Cheers > > > On Dec 21, 2016, at 4:02 AM, Rajeshkumar J > wrote: > > > > Hi, > > > > Thanks for the reply. I have properties as below > > > > > >hbase.regionserver.lease.period > >90 > > > > > >hbase.rpc.timeout > >90>/value> > > > > > > > > Correct me If I am wrong. > > > > I know hbase.regionserver.lease.period, which says how long a scanner > > lives between calls to scanner.next(). > > > > As far as I understand when scanner.next() is called it will fetch no > > of rows as in *hbase.client.scanner.caching. *When this fetching > > process takes more than lease period it will close the scanner object. > > so this exception occuring? > > > > > > Thanks, > > > > Rajeshkumar J > > > > > > > > On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin < > richardstar...@outlook.com > >> wrote: > > > >> It means your lease on a region server has expired during a call to > >> resultscanner.next(). This happens on a slow call to next(). You can > either > >> embrace it or "fix" it by making sure hbase.rpc.timeout exceeds > >> hbase.regionserver.lease.period. > >> > >> https://richardstartin.com > >> > >> On 21 Dec 2016, at 11:30, Rajeshkumar J >> mailto:rajeshkumarit8...@gmail.com>> wrote: > >> > >> Hi, > >> > >> I have faced below issue in our production cluster > >> > >> org.apache.hadoop.hbase.regionserver.LeaseException: > >> org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' > does > >> not exist > >> at org.apache.hadoop.hbase.regionserver.Leases. > >> removeLease(Leases.java:221) > >> at org.apache.hadoop.hbase.regionserver.Leases. > >> cancelLease(Leases.java:206) > >> at > >> org.apache.hadoop.hbase.regionserver.RSRpcServices. > >> scan(RSRpcServices.java:2491) > >> at > >> org.apache.hadoop.hbase.protobuf.generated. > ClientProtos$ClientService$2. > >> callBlockingMethod(ClientProtos.java:32205) > >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > >> at > >> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop( > RpcExecutor.java:130) > >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > >> at java.lang.Thread.run(Thread.java:744) > >> > >> > >> Can any one explain what is lease exception > >> > >> Thanks, > >> Rajeshkumar J > >> >
Re: Lease exception
Hi, Thanks for the reply. I have properties as below hbase.regionserver.lease.period 90 hbase.rpc.timeout 90>/value> Correct me If I am wrong. I know hbase.regionserver.lease.period, which says how long a scanner lives between calls to scanner.next(). As far as I understand when scanner.next() is called it will fetch no of rows as in *hbase.client.scanner.caching. *When this fetching process takes more than lease period it will close the scanner object. so this exception occuring? Thanks, Rajeshkumar J On Wed, Dec 21, 2016 at 5:07 PM, Richard Startin wrote: > It means your lease on a region server has expired during a call to > resultscanner.next(). This happens on a slow call to next(). You can either > embrace it or "fix" it by making sure hbase.rpc.timeout exceeds > hbase.regionserver.lease.period. > > https://richardstartin.com > > On 21 Dec 2016, at 11:30, Rajeshkumar J mailto:rajeshkumarit8...@gmail.com>> wrote: > > Hi, > > I have faced below issue in our production cluster > > org.apache.hadoop.hbase.regionserver.LeaseException: > org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does > not exist > at org.apache.hadoop.hbase.regionserver.Leases. > removeLease(Leases.java:221) > at org.apache.hadoop.hbase.regionserver.Leases. > cancelLease(Leases.java:206) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices. > scan(RSRpcServices.java:2491) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2. > callBlockingMethod(ClientProtos.java:32205) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > at java.lang.Thread.run(Thread.java:744) > > > Can any one explain what is lease exception > > Thanks, > Rajeshkumar J >
Lease exception
Hi, I have faced below issue in our production cluster org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does not exist at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221) at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2491) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:744) Can any one explain what is lease exception Thanks, Rajeshkumar J
Hbase 1.1.1 API documentation link
Hi, We are moving from hbase 0.94 to hbase 1.1.1. Some methods of hbase 0.94 are deprecated for hbase 1.1.1. So can anyone post me the link for Api documentation for Hbase version 1.1.1 as I can't find that. Thanks
Re: Exceptions in Thrift
Hi Ted, Is the difference in the thrift version is the root cause of this problem? As I am using this in web service when multiple (more than 10 requests per sec) requests (clients) are given for this service only it throws error otherwise it works fine. Thanks On Tue, Feb 16, 2016 at 6:12 PM, Ted Yu wrote: > 0.98 uses thrift 0.9.0 > > 0.9.0 > > Can you pastebin the complete stack trace ? > > Were all the client connections encountering this problem ? > > Thanks > > On Tue, Feb 16, 2016 at 12:47 AM, Rajeshkumar J < > rajeshkumarit8...@gmail.com > > wrote: > > > Hi, > > > > Please ignore my previous mail. > > > > My C# code was compiled against thrift 0.92 and I am trying to connect > to > > hbase 0.98.17 > > > > Thanks > > > > On Tue, Feb 16, 2016 at 2:02 PM, Rajeshkumar J < > > rajeshkumarit8...@gmail.com> > > wrote: > > > > > Hi, > > > > > > For both I have used hbase 0.98.17 only > > > > > > Thnaks > > > > > > On Tue, Feb 16, 2016 at 1:14 PM, Ted Yu wrote: > > > > > >> What version of thrift was your C# code compiled against ? > > >> > > >> Which release of hbase do you try to connect to ? > > >> > > >> Cheers > > >> > > >> On Mon, Feb 15, 2016 at 11:04 PM, Rajeshkumar J < > > >> rajeshkumarit8...@gmail.com > > >> > wrote: > > >> > > >> > Hi, > > >> > > > >> > > > >> >I am connecting to thrift server via C# code and I am getting the > > >> > following exception as shown below > > >> > > > >> >Thrift.Protocol.TProtocolException :Missing version in > > >> readMessageBegin, > > >> > old client? > > >> > > > >> > Can any one explain me why above occurs > > >> > > > >> > Thanks > > >> > > > >> > > > > > > > > >
Re: Exceptions in Thrift
Hi, Please ignore my previous mail. My C# code was compiled against thrift 0.92 and I am trying to connect to hbase 0.98.17 Thanks On Tue, Feb 16, 2016 at 2:02 PM, Rajeshkumar J wrote: > Hi, > > For both I have used hbase 0.98.17 only > > Thnaks > > On Tue, Feb 16, 2016 at 1:14 PM, Ted Yu wrote: > >> What version of thrift was your C# code compiled against ? >> >> Which release of hbase do you try to connect to ? >> >> Cheers >> >> On Mon, Feb 15, 2016 at 11:04 PM, Rajeshkumar J < >> rajeshkumarit8...@gmail.com >> > wrote: >> >> > Hi, >> > >> > >> >I am connecting to thrift server via C# code and I am getting the >> > following exception as shown below >> > >> >Thrift.Protocol.TProtocolException :Missing version in >> readMessageBegin, >> > old client? >> > >> > Can any one explain me why above occurs >> > >> > Thanks >> > >> > >
Re: Exceptions in Thrift
Hi, For both I have used hbase 0.98.17 only Thnaks On Tue, Feb 16, 2016 at 1:14 PM, Ted Yu wrote: > What version of thrift was your C# code compiled against ? > > Which release of hbase do you try to connect to ? > > Cheers > > On Mon, Feb 15, 2016 at 11:04 PM, Rajeshkumar J < > rajeshkumarit8...@gmail.com > > wrote: > > > Hi, > > > > > >I am connecting to thrift server via C# code and I am getting the > > following exception as shown below > > > >Thrift.Protocol.TProtocolException :Missing version in > readMessageBegin, > > old client? > > > > Can any one explain me why above occurs > > > > Thanks > > >
Exceptions in Thrift
Hi, I am connecting to thrift server via C# code and I am getting the following exception as shown below Thrift.Protocol.TProtocolException :Missing version in readMessageBegin, old client? Can any one explain me why above occurs Thanks
Issue while trying to store hbase data using pig
Hi,
I am trying to store output from the pig relation into hbase using the
following code
store hbasedata into 'hbase://evallist' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('fa:ID,fa:date,fa:email,fa:customername');
But it throws
Caused by: java.lang.IllegalArgumentException: Must specify table name
can any one help me in this?
Re: Thrift server 2 in Hbase
Hi, Sorry. This issue is from my side as thrift server 2 didn't started correctly. Thanks On Tue, Feb 2, 2016 at 1:35 AM, Stack wrote: > On Tue, Jan 26, 2016 at 9:02 PM, Rajeshkumar J < > rajeshkumarit8...@gmail.com> > wrote: > > > Hi, > > > > > > I have table named temp in hbase. First I have started the thrift > server > > 1 and queried the table and also it returns the result. > > Then I have stopped the thrift server 1 and now I have started thrift > > server 2 and queried the table with some queries which I have used > earlier > > but it didn't return any results. Is there any reasons? > > > > > No. It would be a bug if they did not return result. If you can reproduce, > suggest you file a JIRA. > Thanks, > St.Ack > > > > > Thanks > > >
Fuzzy Row filter with ranges
Hi, Can any one guide me details regarding how to implement fuzzy row filter with ranges in hbase using java api Thanks
Re: Thrift server 2 in Hbase
I am using HBase 0.98.17 Thanks On Wed, Jan 27, 2016 at 11:08 AM, Ted Yu wrote: > Which hbase release are you using ? > > There were some relatively recent fixes going into thrift2 server. > > Cheers > > On Tue, Jan 26, 2016 at 9:12 PM, Rajeshkumar J < > rajeshkumarit8...@gmail.com> > wrote: > > > There is no update happened in the table I have just created sample table > > with 10 rows and tested this. And I have used below command to start > hbase > > thrift server 1 > > > > hbase org.apache.hadoop.hbase.thrift.ThriftServer start > > > > And I have used below command to start hbase thrift server 2 > > > > hbase org.apache.hadoop.hbase.thrift2.ThriftServer start > > > > Thanks > > > > On Wed, Jan 27, 2016 at 10:35 AM, Ted Yu wrote: > > > > > Can you provide a bit more information ? > > > > > > Which hbase release are you using ? > > > In between the two queries, was there any concurrent update to the > > > underlying table ? > > > > > > BTW I assume thrift servers 1 and 2 use the same binary - they just > > resided > > > on different machines. > > > > > > Cheers > > > > > > On Tue, Jan 26, 2016 at 9:02 PM, Rajeshkumar J < > > > rajeshkumarit8...@gmail.com> > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > I have table named temp in hbase. First I have started the thrift > > > server > > > > 1 and queried the table and also it returns the result. > > > > Then I have stopped the thrift server 1 and now I have started > thrift > > > > server 2 and queried the table with some queries which I have used > > > earlier > > > > but it didn't return any results. Is there any reasons? > > > > > > > > Thanks > > > > > > > > > >
Re: Thrift server 2 in Hbase
There is no update happened in the table I have just created sample table with 10 rows and tested this. And I have used below command to start hbase thrift server 1 hbase org.apache.hadoop.hbase.thrift.ThriftServer start And I have used below command to start hbase thrift server 2 hbase org.apache.hadoop.hbase.thrift2.ThriftServer start Thanks On Wed, Jan 27, 2016 at 10:35 AM, Ted Yu wrote: > Can you provide a bit more information ? > > Which hbase release are you using ? > In between the two queries, was there any concurrent update to the > underlying table ? > > BTW I assume thrift servers 1 and 2 use the same binary - they just resided > on different machines. > > Cheers > > On Tue, Jan 26, 2016 at 9:02 PM, Rajeshkumar J < > rajeshkumarit8...@gmail.com> > wrote: > > > Hi, > > > > > > I have table named temp in hbase. First I have started the thrift > server > > 1 and queried the table and also it returns the result. > > Then I have stopped the thrift server 1 and now I have started thrift > > server 2 and queried the table with some queries which I have used > earlier > > but it didn't return any results. Is there any reasons? > > > > Thanks > > >
Thrift server 2 in Hbase
Hi, I have table named temp in hbase. First I have started the thrift server 1 and queried the table and also it returns the result. Then I have stopped the thrift server 1 and now I have started thrift server 2 and queried the table with some queries which I have used earlier but it didn't return any results. Is there any reasons? Thanks
Re: Getting all the columns of row-key at once
yes In Rdms suppose I am concatenating my result in one string
for(;index wrote:
> No it is not possible. Even in RDBMS if you want to do select * from table
> where row=100, you need to do rs.getXXX(i) with suitable index to get the
> expected row right?
>
> If you want specific rows you specify that in the select query - similar
> case here. When you say you don't want to iterate you mean that the Result
> object should just have all the values of all the columns appended to it
> and currently there is no such way possible.
>
> Regards
> Ram
>
> On Fri, Jan 22, 2016 at 11:11 AM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com
> > wrote:
>
> > If that is the case if I do maintain only one versions of my data is this
> > retrieval is possible?
> >
> > Thanks
> >
> > On Fri, Jan 22, 2016 at 11:01 AM, ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com> wrote:
> >
> > > Currently that is not possible. The reason being that the columns are
> not
> > > fixed in HBase.
> > > There could be another row or may another version of this row 100 where
> > > there are only col2 and col4 populated and there is no col1 and col3.
> > > So as per your schema you should be knowing with which column the value
> > is
> > > associated.
> > > In other words
> > >
> > > Row-key col1col2 col3col4
> > >
> > > 100 xxxyyy zzzaaa
> > > 100 xxx
> > yyy
> > >
> > > Now how do you know xxx is associated with col2 or col1 when you try to
> > > retrieve the latest version of row key 100?
> > >
> > > Regards
> > > Ram
> > >
> > >
> > > On Fri, Jan 22, 2016 at 10:55 AM, Rajeshkumar J <
> > > rajeshkumarit8...@gmail.com
> > > > wrote:
> > >
> > > > Hi,
> > > >
> > > > For instance
> > > >
> > > > Row-key col1col2 col3col4
> > > >
> > > > 100 xxxyyy zzz
> aaa
> > > >
> > > > I am scanning this row-key(100) and I want to get the value as
> > > > xxx,yyy,zzz,aaa from Result instance. Not using iterator to get
> > then
> > > > then then .
> > > >
> > > > Thanks
> > > >
> > > > On Fri, Jan 22, 2016 at 10:47 AM, ramkrishna vasudevan <
> > > > ramkrishna.s.vasude...@gmail.com> wrote:
> > > >
> > > > > Once you retrieve a result it will have all the columns that were
> > > > scanned.
> > > > > If suppose you had 5 columns and you specifically wanted only 2
> > columns
> > > > out
> > > > > of it you can add the required columns using scan.addColumn() API
> > then
> > > > the
> > > > > result will have only those 2 columns.
> > > > > If nothing is specified your result will have entire set of columns
> > > that
> > > > > comprises that row (including multiple Column families).
> > > > >
> > > > > But every column's result is an individual KeyValue which you may
> > have
> > > to
> > > > > iterate and get it.
> > > > > >> So is there any option to get all the column
> > > > > values of row-key at once.
> > > > > So this is already happening for you. Am I missing something here?
> > > > >
> > > > > On Fri, Jan 22, 2016 at 10:31 AM, Rajeshkumar J <
> > > > > rajeshkumarit8...@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have already posted this in mailing list but with changes in
> my
> > > use
> > > > > > case. Is there any options to retrieve all the columns of
> row-key
> > at
> > > > > once.
> > > > > >
> > > > > > ResultScanner resultScanner = table.getScanner(scan);
> > > > > > Iterator iterator = resultScanner.iterator();
> > > > > > while (iterator.hasNext()) {
> > > > > > Result next = iterator.next();
> > > > > > for (KeyValue key : next.list()) {
> > > > > >
> > > > > > System.out.println(Bytes.toString(key.getValue()));
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > > > This is how I am doing scan using java api. Using this I can get
> > > only
> > > > > one
> > > > > > columns in each iteration. So is there any option to get all the
> > > column
> > > > > > values of row-key at once.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Getting all the columns of row-key at once
If that is the case if I do maintain only one versions of my data is this
retrieval is possible?
Thanks
On Fri, Jan 22, 2016 at 11:01 AM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:
> Currently that is not possible. The reason being that the columns are not
> fixed in HBase.
> There could be another row or may another version of this row 100 where
> there are only col2 and col4 populated and there is no col1 and col3.
> So as per your schema you should be knowing with which column the value is
> associated.
> In other words
>
> Row-key col1col2 col3col4
>
> 100 xxxyyy zzzaaa
> 100 xxx yyy
>
> Now how do you know xxx is associated with col2 or col1 when you try to
> retrieve the latest version of row key 100?
>
> Regards
> Ram
>
>
> On Fri, Jan 22, 2016 at 10:55 AM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com
> > wrote:
>
> > Hi,
> >
> > For instance
> >
> > Row-key col1col2 col3col4
> >
> > 100 xxxyyy zzzaaa
> >
> > I am scanning this row-key(100) and I want to get the value as
> > xxx,yyy,zzz,aaa from Result instance. Not using iterator to get then
> > then then .
> >
> > Thanks
> >
> > On Fri, Jan 22, 2016 at 10:47 AM, ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com> wrote:
> >
> > > Once you retrieve a result it will have all the columns that were
> > scanned.
> > > If suppose you had 5 columns and you specifically wanted only 2 columns
> > out
> > > of it you can add the required columns using scan.addColumn() API then
> > the
> > > result will have only those 2 columns.
> > > If nothing is specified your result will have entire set of columns
> that
> > > comprises that row (including multiple Column families).
> > >
> > > But every column's result is an individual KeyValue which you may have
> to
> > > iterate and get it.
> > > >> So is there any option to get all the column
> > > values of row-key at once.
> > > So this is already happening for you. Am I missing something here?
> > >
> > > On Fri, Jan 22, 2016 at 10:31 AM, Rajeshkumar J <
> > > rajeshkumarit8...@gmail.com
> > > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I have already posted this in mailing list but with changes in my
> use
> > > > case. Is there any options to retrieve all the columns of row-key at
> > > once.
> > > >
> > > > ResultScanner resultScanner = table.getScanner(scan);
> > > > Iterator iterator = resultScanner.iterator();
> > > > while (iterator.hasNext()) {
> > > > Result next = iterator.next();
> > > > for (KeyValue key : next.list()) {
> > > >
> > > > System.out.println(Bytes.toString(key.getValue()));
> > > > }
> > > >
> > > >
> > > >
> > > > This is how I am doing scan using java api. Using this I can get
> only
> > > one
> > > > columns in each iteration. So is there any option to get all the
> column
> > > > values of row-key at once.
> > > >
> > > > Thanks
> > > >
> > >
> >
>
Re: Getting all the columns of row-key at once
Hi,
For instance
Row-key col1col2 col3col4
100 xxxyyy zzzaaa
I am scanning this row-key(100) and I want to get the value as
xxx,yyy,zzz,aaa from Result instance. Not using iterator to get then
then then .
Thanks
On Fri, Jan 22, 2016 at 10:47 AM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:
> Once you retrieve a result it will have all the columns that were scanned.
> If suppose you had 5 columns and you specifically wanted only 2 columns out
> of it you can add the required columns using scan.addColumn() API then the
> result will have only those 2 columns.
> If nothing is specified your result will have entire set of columns that
> comprises that row (including multiple Column families).
>
> But every column's result is an individual KeyValue which you may have to
> iterate and get it.
> >> So is there any option to get all the column
> values of row-key at once.
> So this is already happening for you. Am I missing something here?
>
> On Fri, Jan 22, 2016 at 10:31 AM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com
> > wrote:
>
> > Hi,
> >
> > I have already posted this in mailing list but with changes in my use
> > case. Is there any options to retrieve all the columns of row-key at
> once.
> >
> > ResultScanner resultScanner = table.getScanner(scan);
> > Iterator iterator = resultScanner.iterator();
> > while (iterator.hasNext()) {
> > Result next = iterator.next();
> > for (KeyValue key : next.list()) {
> >
> > System.out.println(Bytes.toString(key.getValue()));
> > }
> >
> >
> >
> > This is how I am doing scan using java api. Using this I can get only
> one
> > columns in each iteration. So is there any option to get all the column
> > values of row-key at once.
> >
> > Thanks
> >
>
Getting all the columns of row-key at once
Hi,
I have already posted this in mailing list but with changes in my use
case. Is there any options to retrieve all the columns of row-key at once.
ResultScanner resultScanner = table.getScanner(scan);
Iterator iterator = resultScanner.iterator();
while (iterator.hasNext()) {
Result next = iterator.next();
for (KeyValue key : next.list()) {
System.out.println(Bytes.toString(key.getValue()));
}
This is how I am doing scan using java api. Using this I can get only one
columns in each iteration. So is there any option to get all the column
values of row-key at once.
Thanks
Re: Iteration in ResultScanner
Hi gokul,
As you said I have used result.rawCells() but there is no functions to
get all the columns of the given row-key.
Thanks
On Wed, Jan 20, 2016 at 12:23 PM, Gokul Balakrishnan
wrote:
> Yes. You can use result.rawCells() to get the array of Cell instances
> backing your query, if you need to retrieve all columns in your row at
> once. You can then retrieve column values in the same way as above from
> each Cell instance.
>
> Alternatively, if your requirement is to process each column one by one,
> you can use result.cellScanner() to get a CellScanner instance (say
> "cellScanner"), and then iterate over each cell through
> cellScanner.current().
>
> Please see [1] for all operations supported by
> org.apache.hadoop.hbase.client.Result.
>
> [1]
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Result.html
>
> On 20 January 2016 at 12:02, Rajeshkumar J
> wrote:
>
> > Hi,
> >
> > I want to get all the column values for a given row-key.
> >
> > Row-key col1 col2 col3 col4 col5
> >
> > id100 XXXYYZZZ AAA BBB
> >
> > is there any option available to get all the column values for given
> > row-key. Here for id100 I want to have XX,YY, ZZZ ,AAA,BBB
> >
> > Thanks
> >
> >
> >
> > On Wed, Jan 20, 2016 at 11:30 AM, Gokul Balakrishnan >
> > wrote:
> >
> > > There are various ways, one of which is as follows:
> > >
> > > Result result = table.get(new Get("id100".**getBytes()));
> > > Cell dataCell
> > > = result.getColumnLatestCell("some_column_family","some_qualifier");
> > > byte[] data = CellUtil.cloneValue(dataCell);
> > >
> > > The byte array "data" will represent the value of the column you've
> > > retrieved. Note that this code returns the latest version of the cell
> > > you've specified, and you'll have to do the necessary null checks for
> > > "dataCell".
> > >
> > > On 20 January 2016 at 11:09, Rajeshkumar J <
> rajeshkumarit8...@gmail.com>
> > > wrote:
> > >
> > > > Hi solomon,
> > > >
> > > >The statement you have given is of
> > > org.apache.hadoop.hbase.client.Result
> > > > type. How to get all the column values from this??
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Jan 19, 2016 at 6:45 PM, Solomon Duskis
> > > wrote:
> > > >
> > > > > It sounds like you want a Get rathe than a scan. *table.get(new
> > > > > Get("id100".**getBytes()));* should do the trick.
> > > > >
> > > > > On Tue, Jan 19, 2016 at 7:17 AM, Rajeshkumar J <
> > > > > rajeshkumarit8...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have implemented Range scan using Java API as shown below
> > > > > >
> > > > > > Scan scan = new Scan();
> > > > > > Configuration config = HBaseConfiguration.create();
> > > > > > HTable table = new HTable(config, "tablename");
> > > > > > scan.setStartRow("id100".getBytes());
> > > > > > scan.setStopRow("id100".getBytes());
> > > > > > ResultScanner scanner = table.getScanner(scan);
> > > > > >
> > > > > > Is there any option like so that i can get all the column values
> > only
> > > > > for a
> > > > > > particular row-key without iteration.
> > > > > >
> > > > > > For ex : I have used id100 which is a row-key. After the scan
> > > completed
> > > > > how
> > > > > > to get all the column values of id100 row key without iteration.
> > > > > >
> > > > > > Thanks,
> > > > > > Rajeshkumar J
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Java API vs Hbase Thrift
Hi, As you all said I have tried Rest web service using Hbase Java API to get data from Hbase table but it seems to be slower than that of one using Hbase thrift server. can any one tell how ? Thanks On Sat, Jan 16, 2016 at 5:41 PM, Zheng Shen wrote: > Java API is at least 10 times faster than thrift on Hbase write > operations based on my experience in production environment (cloudera > 5.4.7, hbase 1.0.0) > > Zheng > > ---Original--- > From: "Vladimir Rodionov " > Date: 2016/1/15 06:31:34 > To: "user@hbase.apache.org"; > Subject: Re: Java API vs Hbase Thrift > > > >> I have to access hbase using Java API will it be fast like thrift. > > Bear in mind that when you use Thrift Gateway/Thrift API you access HBase > RegionServer through the single gateway server, > when you use Java API - you access Region Server directly. > Java API is much more scalable. > > -Vlad > > On Tue, Jan 12, 2016 at 7:36 AM, Anil Gupta wrote: > > > Java api should be same or better in performance as compared to Thrift > api. > > With Thrift api there is an extra hop. So, most of the time java api > would > > be better for performance. > > > > Sent from my iPhone > > > > > On Jan 12, 2016, at 4:29 AM, Rajeshkumar J < > rajeshkumarit8...@gmail.com> > > wrote: > > > > > > Hi, > > > > > > I am currently accessing records via Hbase thrift server and it is > fast. > > > If I have to access hbase using Java API will it be fast like thrift. > > > > > > Thanks > > > >
Re: Iteration in ResultScanner
Hi,
I want to get all the column values for a given row-key.
Row-key col1 col2 col3 col4 col5
id100 XXXYYZZZ AAA BBB
is there any option available to get all the column values for given
row-key. Here for id100 I want to have XX,YY, ZZZ ,AAA,BBB
Thanks
On Wed, Jan 20, 2016 at 11:30 AM, Gokul Balakrishnan
wrote:
> There are various ways, one of which is as follows:
>
> Result result = table.get(new Get("id100".**getBytes()));
> Cell dataCell
> = result.getColumnLatestCell("some_column_family","some_qualifier");
> byte[] data = CellUtil.cloneValue(dataCell);
>
> The byte array "data" will represent the value of the column you've
> retrieved. Note that this code returns the latest version of the cell
> you've specified, and you'll have to do the necessary null checks for
> "dataCell".
>
> On 20 January 2016 at 11:09, Rajeshkumar J
> wrote:
>
> > Hi solomon,
> >
> >The statement you have given is of
> org.apache.hadoop.hbase.client.Result
> > type. How to get all the column values from this??
> >
> > Thanks
> >
> > On Tue, Jan 19, 2016 at 6:45 PM, Solomon Duskis
> wrote:
> >
> > > It sounds like you want a Get rathe than a scan. *table.get(new
> > > Get("id100".**getBytes()));* should do the trick.
> > >
> > > On Tue, Jan 19, 2016 at 7:17 AM, Rajeshkumar J <
> > > rajeshkumarit8...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I have implemented Range scan using Java API as shown below
> > > >
> > > > Scan scan = new Scan();
> > > > Configuration config = HBaseConfiguration.create();
> > > > HTable table = new HTable(config, "tablename");
> > > > scan.setStartRow("id100".getBytes());
> > > > scan.setStopRow("id100".getBytes());
> > > > ResultScanner scanner = table.getScanner(scan);
> > > >
> > > > Is there any option like so that i can get all the column values only
> > > for a
> > > > particular row-key without iteration.
> > > >
> > > > For ex : I have used id100 which is a row-key. After the scan
> completed
> > > how
> > > > to get all the column values of id100 row key without iteration.
> > > >
> > > > Thanks,
> > > > Rajeshkumar J
> > > >
> > >
> >
>
Re: Iteration in ResultScanner
Hi solomon,
The statement you have given is of org.apache.hadoop.hbase.client.Result
type. How to get all the column values from this??
Thanks
On Tue, Jan 19, 2016 at 6:45 PM, Solomon Duskis wrote:
> It sounds like you want a Get rathe than a scan. *table.get(new
> Get("id100".**getBytes()));* should do the trick.
>
> On Tue, Jan 19, 2016 at 7:17 AM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have implemented Range scan using Java API as shown below
> >
> > Scan scan = new Scan();
> > Configuration config = HBaseConfiguration.create();
> > HTable table = new HTable(config, "tablename");
> > scan.setStartRow("id100".getBytes());
> > scan.setStopRow("id100".getBytes());
> > ResultScanner scanner = table.getScanner(scan);
> >
> > Is there any option like so that i can get all the column values only
> for a
> > particular row-key without iteration.
> >
> > For ex : I have used id100 which is a row-key. After the scan completed
> how
> > to get all the column values of id100 row key without iteration.
> >
> > Thanks,
> > Rajeshkumar J
> >
>
Iteration in ResultScanner
Hi,
I have implemented Range scan using Java API as shown below
Scan scan = new Scan();
Configuration config = HBaseConfiguration.create();
HTable table = new HTable(config, "tablename");
scan.setStartRow("id100".getBytes());
scan.setStopRow("id100".getBytes());
ResultScanner scanner = table.getScanner(scan);
Is there any option like so that i can get all the column values only for a
particular row-key without iteration.
For ex : I have used id100 which is a row-key. After the scan completed how
to get all the column values of id100 row key without iteration.
Thanks,
Rajeshkumar J
Java API vs Hbase Thrift
Hi, I am currently accessing records via Hbase thrift server and it is fast. If I have to access hbase using Java API will it be fast like thrift. Thanks
Re: Hbase thrift server issues
Hi Ted, As mentioned in that HBASE-14926 it is fixed in HBase 0.98.17. I have googled it for both its source code and binary but I didn't find one. can you guide me where I can find Hbase 0.98.17 source code or binary package Thanks On Tue, Jan 12, 2016 at 12:57 AM, Ted Yu wrote: > I took a look at the stack trace you posted. > > Some thrift worker was in the state shown in HBASE-14926 > > I wonder if the fix for HBASE-14926 would help in your case. > > Cheers > > On Mon, Jan 11, 2016 at 5:11 AM, Rajeshkumar J < > rajeshkumarit8...@gmail.com> > wrote: > > > Hi Ted, > > > >Plz ignore my previous mail as the log is taken when hbase is working > > fine. Please find the logs below when Hbase didn't return any records > > > > http://pastebin.com/APYjiGSP > > > > Thanks > > > > On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu wrote: > > > > > When you search for 'java jstack tutorial' using google, you would > find a > > > lot of tutorials. > > > > > > You can use 'ps aux' command to search for the following class: > > > org.apache.hadoop.hbase.thrift.ThriftServer > > > > > > Apply jstack on the corresponding process id. > > > > > > Cheers > > > > > > On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J < > > > rajeshkumarit8...@gmail.com> > > > wrote: > > > > > > > Hi Ted, > > > > > > > >I don't know how how to take stacktrace using jstack. But i tried > > > using > > > > some commands but it fails. can you help me in this? > > > > > > > > Thanks > > > > > > > > On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu > wrote: > > > > > > > > > What I meant was to capture stack trace of the thrift server (using > > > > jstack > > > > > command) when your query is running. > > > > > > > > > > This would give us some more clue as to the state thrift server was > > in > > > > when > > > > > the problem happened. > > > > > > > > > > Cheers > > > > > > > > > > On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J < > > > > > rajeshkumarit8...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Ted, > > > > > > > > > > > >This is what I found in the Hbase thrift server logs > > > > > > > > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:java.library.path=;C:\Hadoop\bin > > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:java.io.tmpdir=C:\Windows\TEMP\ > > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:java.compiler= > > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:os.name=Windows Server 2012 > > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:os.arch=amd64 > > > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:os.version=6.2 > > > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:user.name= > > > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:user.home=C:\ > > > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:user.dir=C:\HBase\bin > > > > > > 2016-01-09 06:27:43,142 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > > Initiating client connection, > > > > > > connectString=hostname:2181,hostname:2181,hostname:2181 > > > > > > sessionTimeout=9 watcher=hconnection-0x45e35460x0, > > > >
Re: Hbase thrift server issues
Hi Ted,
Plz ignore my previous mail as the log is taken when hbase is working
fine. Please find the logs below when Hbase didn't return any records
http://pastebin.com/APYjiGSP
Thanks
On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu wrote:
> When you search for 'java jstack tutorial' using google, you would find a
> lot of tutorials.
>
> You can use 'ps aux' command to search for the following class:
> org.apache.hadoop.hbase.thrift.ThriftServer
>
> Apply jstack on the corresponding process id.
>
> Cheers
>
> On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> >I don't know how how to take stacktrace using jstack. But i tried
> using
> > some commands but it fails. can you help me in this?
> >
> > Thanks
> >
> > On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu wrote:
> >
> > > What I meant was to capture stack trace of the thrift server (using
> > jstack
> > > command) when your query is running.
> > >
> > > This would give us some more clue as to the state thrift server was in
> > when
> > > the problem happened.
> > >
> > > Cheers
> > >
> > > On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
> > > rajeshkumarit8...@gmail.com>
> > > wrote:
> > >
> > > > Hi Ted,
> > > >
> > > >This is what I found in the Hbase thrift server logs
> > > >
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.library.path=;C:\Hadoop\bin
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.io.tmpdir=C:\Windows\TEMP\
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.compiler=
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.name=Windows Server 2012
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.arch=amd64
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.version=6.2
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.name=
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.home=C:\
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.dir=C:\HBase\bin
> > > > 2016-01-09 06:27:43,142 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > > Initiating client connection,
> > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > sessionTimeout=9 watcher=hconnection-0x45e35460x0,
> > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > 2016-01-09 06:27:43,339 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > > (unknown error)
> > > > 2016-01-09 06:27:43,342 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to
> > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > 2016-01-09 06:27:43,365 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated
> > > timeout
> > > > = 4
> > > > 2016-01-09 07:14:10,607 INFO
> > > > [hostname,6,1452337815006_splitLogManager__ChoreService_1]
> > > > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
> > > > Monitor missed its start time
> > > > 2016-01-09 07:16:43,511 INFO
> > > > [B.defaultRpcServer.handler=8,queue=2,port=6] master.HMaster:
> > > > Client=Administrator/null create 'chk2', {NAME => 'fa',
> > > DATA_BLOCK_ENCODING
> > > > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE =>
Re: Hbase thrift server issues
Hi Ted,
I have taken stacktrace of Hbase thrift server. Plz find it in below
location
http://pastebin.com/QQjZhi36
Thanks
On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu wrote:
> When you search for 'java jstack tutorial' using google, you would find a
> lot of tutorials.
>
> You can use 'ps aux' command to search for the following class:
> org.apache.hadoop.hbase.thrift.ThriftServer
>
> Apply jstack on the corresponding process id.
>
> Cheers
>
> On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> >I don't know how how to take stacktrace using jstack. But i tried
> using
> > some commands but it fails. can you help me in this?
> >
> > Thanks
> >
> > On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu wrote:
> >
> > > What I meant was to capture stack trace of the thrift server (using
> > jstack
> > > command) when your query is running.
> > >
> > > This would give us some more clue as to the state thrift server was in
> > when
> > > the problem happened.
> > >
> > > Cheers
> > >
> > > On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
> > > rajeshkumarit8...@gmail.com>
> > > wrote:
> > >
> > > > Hi Ted,
> > > >
> > > >This is what I found in the Hbase thrift server logs
> > > >
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.library.path=;C:\Hadoop\bin
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.io.tmpdir=C:\Windows\TEMP\
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.compiler=
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.name=Windows Server 2012
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.arch=amd64
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.version=6.2
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.name=
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.home=C:\
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.dir=C:\HBase\bin
> > > > 2016-01-09 06:27:43,142 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > > Initiating client connection,
> > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > sessionTimeout=9 watcher=hconnection-0x45e35460x0,
> > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > 2016-01-09 06:27:43,339 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > > (unknown error)
> > > > 2016-01-09 06:27:43,342 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to
> > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > 2016-01-09 06:27:43,365 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated
> > > timeout
> > > > = 4
> > > > 2016-01-09 07:14:10,607 INFO
> > > > [hostname,6,1452337815006_splitLogManager__ChoreService_1]
> > > > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
> > > > Monitor missed its start time
> > > > 2016-01-09 07:16:43,511 INFO
> > > > [B.defaultRpcServer.handler=8,queue=2,port=6] master.HMaster:
> > > > Client=Administrator/null create 'chk2', {NAME => 'fa',
> > > DATA_BLOCK_ENCODING
> > > > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS
> =>
> > > '1&
Re: Hbase thrift server issues
Hi Ted,
I don't know how how to take stacktrace using jstack. But i tried using
some commands but it fails. can you help me in this?
Thanks
On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu wrote:
> What I meant was to capture stack trace of the thrift server (using jstack
> command) when your query is running.
>
> This would give us some more clue as to the state thrift server was in when
> the problem happened.
>
> Cheers
>
> On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> >This is what I found in the Hbase thrift server logs
> >
> > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:java.library.path=;C:\Hadoop\bin
> > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:java.io.tmpdir=C:\Windows\TEMP\
> > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:java.compiler=
> > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:os.name=Windows Server 2012
> > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:os.arch=amd64
> > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:os.version=6.2
> > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:user.name=
> > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:user.home=C:\
> > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:user.dir=C:\HBase\bin
> > 2016-01-09 06:27:43,142 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > Initiating client connection,
> > connectString=hostname:2181,hostname:2181,hostname:2181
> > sessionTimeout=9 watcher=hconnection-0x45e35460x0,
> > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > 2016-01-09 06:27:43,339 INFO [thrift-worker-0-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server
> > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > (unknown error)
> > 2016-01-09 06:27:43,342 INFO [thrift-worker-0-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Socket connection established to
> > hostname/xx.xx.x.xx:2181, initiating session
> > 2016-01-09 06:27:43,365 INFO [thrift-worker-0-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server
> > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated
> timeout
> > = 4
> > 2016-01-09 07:14:10,607 INFO
> > [hostname,6,1452337815006_splitLogManager__ChoreService_1]
> > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
> > Monitor missed its start time
> > 2016-01-09 07:16:43,511 INFO
> > [B.defaultRpcServer.handler=8,queue=2,port=6] master.HMaster:
> > Client=Administrator/null create 'chk2', {NAME => 'fa',
> DATA_BLOCK_ENCODING
> > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS =>
> '1',
> > COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> > KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY =>
> 'false',
> > BLOCKCACHE => 'true'}
> > 2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1]
> > procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure
> > (table=chk2) user=Administrator (auth:SIMPLE) id=2 state=ROLLEDBACK
> > exec-time=462msec exception=org.apache.hadoop.hbase.TableExistsException:
> > chk2
> > 2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1]
> > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x1522613893e0004
> > 2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1]
> > zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
> > 2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread]
> > zookeeper.ClientCnxn: EventThread shut down
> > 2016-01-09 18:40:29,469 INFO [thrift-worker-1]
> > zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x591287f8
> > connecting to ZooKeeper
> ensemble=hostname:2181,hostname:2181,hostname:2181
> > 2016-01-09 18:40:29,482 INFO [thrift-worker-1] zookeeper.ZooKeeper:
> > Initiating client connection,
> > connectString=hostname:21
Re: Hbase thrift server issues
tname:2181,hostname:2181,hostname:2181 2016-01-09 20:05:42,476 INFO [thrift-worker-13] zookeeper.ZooKeeper: Initiating client connection, connectString=hostname:2181,hostname:2181,hostname:2181 sessionTimeout=9 watcher=hconnection-0x6305dbeb0x0, quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase 2016-01-09 20:05:42,507 INFO [thrift-worker-13-SendThread(hostname:2181)] zookeeper.ClientCnxn: Opening socket connection to server hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL (unknown error) 2016-01-09 20:05:42,510 INFO [thrift-worker-13-SendThread(hostname:2181)] zookeeper.ClientCnxn: Socket connection established to hostname/xx.xx.x.xx:2181, initiating session 2016-01-09 20:05:42,708 INFO [thrift-worker-13-SendThread(hostname:2181)] zookeeper.ClientCnxn: Session establishment complete on server hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated timeout = 4 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1522613893e0005 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1] zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed Thanks, Rajeshkumar J On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu wrote: > Can you take a stack trace of the thrift server and pastebin the trace ? > > Thanks > > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J < > rajeshkumarit8...@gmail.com> > wrote: > > > Hi, > > > > > > I am connecting via Hbase thrift server to access records in Hbase and I > am > > doing this in C# and i am using range scan to get records from the table. > > This seems to be working for few hours but after few hours it didn't > > returned any records and then i searched hbase logs i found this > > > > 2016-01-04 00:44:54,880 INFO [ConnectionCleaner] > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper > > sessionid=0x3520b1721650006 > > 2016-01-04 00:44:54,891 INFO [ConnectionCleaner] zookeeper.ZooKeeper: > > Session: 0x3520b1721650006 closed > > 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread] > > zookeeper.ClientCnxn: EventThread shut down > > > > And I am using hbase 0.98.13 > > > > I have asked this question before and some user suggested me to upgrade > to > > new version so i have upgraded HBase to 1.1.2. But now it also it didn't > > return any records but different log generated as shown below > > > > 2016-01-09 20:05:42,708 INFO [thrift-worker-13-SendThread(:2181)] > > zookeeper.ClientCnxn: Session establishment complete on server > > /xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated timeout = > > 4 > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1] > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper > > sessionid=0x1522613893e0005 > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1] > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed > > 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread] > > zookeeper.ClientCnxn: EventThread shut down > > > > Can any one help me in this? > > > > Regards, > > Rajeshkumar J > > >
Hbase thrift server issues
Hi, I am connecting via Hbase thrift server to access records in Hbase and I am doing this in C# and i am using range scan to get records from the table. This seems to be working for few hours but after few hours it didn't returned any records and then i searched hbase logs i found this 2016-01-04 00:44:54,880 INFO [ConnectionCleaner] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x3520b1721650006 2016-01-04 00:44:54,891 INFO [ConnectionCleaner] zookeeper.ZooKeeper: Session: 0x3520b1721650006 closed 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread] zookeeper.ClientCnxn: EventThread shut down And I am using hbase 0.98.13 I have asked this question before and some user suggested me to upgrade to new version so i have upgraded HBase to 1.1.2. But now it also it didn't return any records but different log generated as shown below 2016-01-09 20:05:42,708 INFO [thrift-worker-13-SendThread(:2181)] zookeeper.ClientCnxn: Session establishment complete on server /xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated timeout = 4 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1522613893e0005 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1] zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread] zookeeper.ClientCnxn: EventThread shut down Can any one help me in this? Regards, Rajeshkumar J
Re: Hbase thrift server issue
Hi Talat, I am using Hbase 0.98.13 Thanks On Thu, Jan 7, 2016 at 2:00 PM, Talat Uyarer wrote: > Hi Rajeshkumar, > > Which version of Hbase do you use ? Look at this issue: > https://issues.apache.org/jira/browse/HBASE-14196 > May be you are effected by the issue. > > Thanks > > 2016-01-06 23:29 GMT-08:00 Rajeshkumar J : > > Hi, > > > > I am connecting via Hbase thrift server to access records in Hbase and > I > > am doing this in C# and i am using range scan to get records from the > > table. This seems to be working for few hours but after few hours it > didn't > > returned any records and then i searched hbase logs i found this > > > > 2016-01-04 00:44:54,880 INFO [ConnectionCleaner] > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper > > sessionid=0x3520b1721650006 > > 2016-01-04 00:44:54,891 INFO [ConnectionCleaner] zookeeper.ZooKeeper: > > Session: 0x3520b1721650006 closed > > 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread] > > zookeeper.ClientCnxn: EventThread shut down > > > > Can any one help me in resolving this? > > > > Thanks > > > > -- > Talat UYARER > Websitesi: http://talat.uyarer.com > Twitter: http://twitter.com/talatuyarer > Linkedin: http://tr.linkedin.com/pub/talat-uyarer/10/142/304 >
Re: issue in fuzzy filter
Hi Ted,
I have solved this issue as this is due to illegal class name.
Thanks
On Mon, Jan 4, 2016 at 7:35 PM, Ted Yu wrote:
> Can you pastebin the complete error you encountered ?
> What dependencies have you added ?
>
> Thanks
>
> > On Jan 4, 2016, at 3:44 AM, Rajeshkumar J
> wrote:
> >
> > Hi,
> >
> > We can use fuzzy row filter when rowkey has fixed length.I decided to
> > design row key as -mm-dd| platform id | control id where platform id
> > and control id are a two digit nos.
> > we have combo box through which we can select differen values of date,
> > platform and control for search. For this we decided to use fuzzy filter.
> > So now actions can be performed by -mm-dd|??|?? and so on.
> >
> > Now i am trying to use fuzzy row filter in java api and i tried below
> > code but it throws error.
> >
> >
> >Configuration config = HBaseConfiguration.create();
> >
> >
> >HTable hTable = new HTable(config, "test");
> >
> >Scan scan = new Scan();
> >
> >scan.setFilter(new FuzzyRowFilter(
> >Arrays.asList(
> >new Pair(
> >
> > Bytes.toBytesBinary("\\x00\\x00\\x00\\x00_login_"),
> >new byte[]{1, 1, 1, 1, 0, 0, 0, 0, 0, 0,
> > 0};
> >}
> >
> >
> > I have added all the dependency jars but still it throws Create
> constructor
> > " FuzzyRowFilter(java.util.list) " and i am using hbase version 0.98.13
> >
> > Can any one help me in solving this?
> >
> >
> > Thanks
>
issue in fuzzy filter
Hi,
We can use fuzzy row filter when rowkey has fixed length.I decided to
design row key as -mm-dd| platform id | control id where platform id
and control id are a two digit nos.
we have combo box through which we can select differen values of date,
platform and control for search. For this we decided to use fuzzy filter.
So now actions can be performed by -mm-dd|??|?? and so on.
Now i am trying to use fuzzy row filter in java api and i tried below
code but it throws error.
Configuration config = HBaseConfiguration.create();
HTable hTable = new HTable(config, "test");
Scan scan = new Scan();
scan.setFilter(new FuzzyRowFilter(
Arrays.asList(
new Pair(
Bytes.toBytesBinary("\\x00\\x00\\x00\\x00_login_"),
new byte[]{1, 1, 1, 1, 0, 0, 0, 0, 0, 0,
0};
}
I have added all the dependency jars but still it throws Create constructor
" FuzzyRowFilter(java.util.list) " and i am using hbase version 0.98.13
Can any one help me in solving this?
Thanks
Re: Type of filter to use
Hi Tariq,
I have tried rowfilter both with substring and regex filter it is very
slow but i need data retrieval to be less than 3 seconds.
Thanks
On Mon, Jan 4, 2016 at 4:33 PM, Mohammad Tariq wrote:
> Hi Rajesh,
>
> To begin with I would suggest you to rethink about your key design. If you
> are getting a lot of data(which I assume you do) then this type of keys
> could lead you to region hotspotting
> <https://issues.apache.org/jira/browse/HBASE-11682>.
>
> Coming back to your original question you could also use row filter with
> substring comparator.
>
>
> [image: http://]
> Tariq, Mohammad
> about.me/mti
> [image: http://]
> <http://about.me/mti>
>
>
> On Mon, Jan 4, 2016 at 4:08 PM, Rajeshkumar J >
> wrote:
>
> > Hi,
> >
> > For the above query i decided to use fuzzy rowfilter as of now. we can
> > use fuzzy row filter when rowkey has fixed length. For the above query i
> > decided to design row key as -mm-dd| platform id | control id where
> > platform id and control id are a two digit nos.
> > So now actions can be performed by -mm-dd|??|?? and so on.
> >
> > Now i am trying to use fuzzy row filter in java api and i tried below
> > code but it throws error.
> >
> >
> > Configuration config = HBaseConfiguration.create();
> >
> >
> > HTable hTable = new HTable(config, "test");
> >
> > Scan scan = new Scan();
> >
> > scan.setFilter(new FuzzyRowFilter(
> > Arrays.asList(
> > new Pair(
> >
> > Bytes.toBytesBinary("\\x00\\x00\\x00\\x00_login_"),
> > new byte[]{1, 1, 1, 1, 0, 0, 0, 0, 0, 0,
> > 0};
> > }
> >
> >
> > I have added all the dependency jars but still it throws Create
> constructor
> > " FuzzyRowFilter(java.util.list) " and i am using hbase version 0.98.13
> >
> > Can any one help me in solving this?
> >
> >
> > Thanks
> >
> > On Mon, Jan 4, 2016 at 12:16 PM, Rajeshkumar J <
> > rajeshkumarit8...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I have designed my row key as -mm-dd| platform | control and the
> > > column values are the ids of the customers who visited the respective
> > > platform and control on a particular day.
> > > For instance
> > > 2016-01-01 | Bigdata | Bigdata 1002,20005,3009
> > >
> > > Now we have a combo box through which we can choose any date, platform
> > and
> > > control values.
> > >
> > > if they choose these three values we can match the row key easily and
> > > retrieve records. what if they chose only platform or control or date
> > > values what type of filter do i have to use for this scenario
> > >
> > > Thanks
> > >
> >
>
Re: Type of filter to use
Hi,
For the above query i decided to use fuzzy rowfilter as of now. we can
use fuzzy row filter when rowkey has fixed length. For the above query i
decided to design row key as -mm-dd| platform id | control id where
platform id and control id are a two digit nos.
So now actions can be performed by -mm-dd|??|?? and so on.
Now i am trying to use fuzzy row filter in java api and i tried below
code but it throws error.
Configuration config = HBaseConfiguration.create();
HTable hTable = new HTable(config, "test");
Scan scan = new Scan();
scan.setFilter(new FuzzyRowFilter(
Arrays.asList(
new Pair(
Bytes.toBytesBinary("\\x00\\x00\\x00\\x00_login_"),
new byte[]{1, 1, 1, 1, 0, 0, 0, 0, 0, 0,
0};
}
I have added all the dependency jars but still it throws Create constructor
" FuzzyRowFilter(java.util.list) " and i am using hbase version 0.98.13
Can any one help me in solving this?
Thanks
On Mon, Jan 4, 2016 at 12:16 PM, Rajeshkumar J
wrote:
> Hi,
>
> I have designed my row key as -mm-dd| platform | control and the
> column values are the ids of the customers who visited the respective
> platform and control on a particular day.
> For instance
> 2016-01-01 | Bigdata | Bigdata 1002,20005,3009
>
> Now we have a combo box through which we can choose any date, platform and
> control values.
>
> if they choose these three values we can match the row key easily and
> retrieve records. what if they chose only platform or control or date
> values what type of filter do i have to use for this scenario
>
> Thanks
>
Type of filter to use
Hi, I have designed my row key as -mm-dd| platform | control and the column values are the ids of the customers who visited the respective platform and control on a particular day. For instance 2016-01-01 | Bigdata | Bigdata 1002,20005,3009 Now we have a combo box through which we can choose any date, platform and control values. if they choose these three values we can match the row key easily and retrieve records. what if they chose only platform or control or date values what type of filter do i have to use for this scenario Thanks
Re: MultiRowRangeFilter in hbase shell
Hi Ted,
Also I do want to scan multiple row key ranges and I came across this
jira
https://issues.apache.org/jira/browse/HBASE-11144
whether this is implemented if so guide me the command to make use of it
Thanks
On Thu, Dec 31, 2015 at 7:43 PM, Ted Yu wrote:
> MultiRowRangeFilter involves List of RowRange's whose definition depends
> on:
>
> public RowRange(String startRow, boolean startRowInclusive, String
> stopRow,
>
> boolean stopRowInclusive) {
>
> It would be tedious to construct MultiRowRangeFilter in shell.
>
> Can you use Java API ?
>
>
> Cheers
>
> On Thu, Dec 31, 2015 at 3:58 AM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com>
> wrote:
>
> > Hi
> >
> >
> > Can any one tell me hbase shell command to use how
> > MultiRowRangeFilter ?
> >
> > Thanks
> >
>
MultiRowRangeFilter in hbase shell
Hi Can any one tell me hbase shell command to use how MultiRowRangeFilter ? Thanks
Re: regular expression in range scan
Hi Ted,
This is my usecase we have 3 combo boxes to choose date and platform and
control. Based on the options chosen we are retrieving data from hbase. We
have designed our row key as date|platform|control. So if the user chooses
single value for each combo box (date : 2015-12-12, platform : bigdata,
control : bigdata and this will be concatenated as
2015-12-12|bigdata|bigdata and passed to hbase) it works and retrieves
data. But if the user chooses multiple options for either platform or
control it will fail. So help me in resolving this problem?
Thanks
On Thu, Dec 31, 2015 at 11:43 AM, Ted Yu wrote:
> For the example given below, you can specify PrefixFilter for the scan.
>
> Please see also for examples of filter involving regex:
> https://issues.apache.org/jira/browse/HBASE-9428
>
> > On Dec 30, 2015, at 9:57 PM, Rajeshkumar J
> wrote:
> >
> > Hi,
> >
> > Currently i am using range scan for querying data in my hbase table. Is
> > there any option available to use regular expression in range scan. For
> ex
> > this is my range scan query
> >
> > scan 'tmp', {STARTROW => '539428', ENDROW => '539430'}
> >
> > Is there any option available so that we can use
> >
> >
> > scan 'tmp', {STARTROW => '5394.*', ENDROW => '539430'}
> >
> > Thanks
>
regular expression in range scan
Hi,
Currently i am using range scan for querying data in my hbase table. Is
there any option available to use regular expression in range scan. For ex
this is my range scan query
scan 'tmp', {STARTROW => '539428', ENDROW => '539430'}
Is there any option available so that we can use
scan 'tmp', {STARTROW => '5394.*', ENDROW => '539430'}
Thanks
