Re: parallel processing of regions in coprocessor
In production, I assume you have more than one server hosting the regions
you want to process.
Assuming load balancer does its job of spreading regions across servers,
you should consider using batchCoprocessorService.
There're only so many handlers on each region server, parallel invocation
of coprocessors of all relevant regions on individual server places burden
on the server.
FYI
On Thu, Mar 2, 2017 at 8:22 AM, Daniel Połaczański
wrote:
> thx, this is what I needed
>
> 2017-03-02 11:07 GMT+01:00 Ted Yu :
>
> > Daniel:
> > If you don't pass your ExecutorService to HTable ctor, the following
> would
> > be called:
> >
> > public static ThreadPoolExecutor getDefaultExecutor(Configuration
> conf)
> > {
> >
> > int maxThreads = conf.getInt("hbase.htable.threads.max", Integer.
> > MAX_VALUE);
> >
> > Note the default value.
> >
> > When the number of regions involved is high, consider adjusting the above
> > config.
> >
> > On Wed, Mar 1, 2017 at 11:04 PM, Daniel Połaczański <
> > dpolaczan...@gmail.com>
> > wrote:
> >
> > > Yes. I am using batch variant. I will try your variant
> > >
> > > 02.03.2017 7:53 AM "Jingcheng Du" napisał(a):
> > >
> > > What APIs are you using to execute the coprocessor?
> > > HTable#batchCoprocessorService? It executes the coprocessors regions
> by
> > > regions in each region server.
> > > How about using coprocessorService? This API should run coprocessors in
> > > parallel for different regions.
> > >
> > > 2017-03-02 14:32 GMT+08:00 Daniel Połaczański >:
> > >
> > > > I invoke my business logic which is similar to map reduce paradigm.
> > > > Everything works. Only performance is problem that regions are not
> > > > processed parallely.
> > > > I'm thinking about some hack like invoking 4 times rpc from client
> > when i
> > > > have 4 regions
> > > >
> > > > 02.03.2017 00:43 "Ted Yu" napisał(a):
> > > >
> > > > > To my knowledge, there is no support for this type of combination
> of
> > > map
> > > > > reduce and coprocessor.
> > > > >
> > > > > On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański <
> > > > dpolaczan...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > It is something like map reduce processing.
> > > > > > I want to run map and combine phase inside in a coprocessor for
> > every
> > > > > > region separately, return partial results to the client. On the
> > > client
> > > > > > invoke reduce phase and store result in another hbase table.
> > > > > >
> > > > > > 2017-03-01 23:26 GMT+01:00 Ted Yu :
> > > > > >
> > > > > > > Can you describe your use case in more detail ?
> > > > > > >
> > > > > > > What type of custom coprocessor are you loading to the region
> > > server
> > > > ?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański <
> > > > > > dpolaczan...@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > > Let's assume that we have cluster consisting from one
> > > RegionServer
> > > > > and
> > > > > > > the
> > > > > > > > RegionsServer contains one table consisting from 3 regions.
> > > > > > > >
> > > > > > > > I would like to process regions in coprocessor parallely. Is
> > it
> > > > > > > possible?
> > > > > > > >
> > > > > > > > I observed that currenlty it invokes coproprocessor with all
> > the
> > > > > > regions
> > > > > > > > one by one. So it process at the beginig region1, when
> finished
> > > > > process
> > > > > > > > regions2 and ...
> > > > > > > >
> > > > > > > > Is it possible to change this behaviour?
> > > > > > > >
> > > > > > > > Regards
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: parallel processing of regions in coprocessor
thx, this is what I needed
2017-03-02 11:07 GMT+01:00 Ted Yu :
> Daniel:
> If you don't pass your ExecutorService to HTable ctor, the following would
> be called:
>
> public static ThreadPoolExecutor getDefaultExecutor(Configuration conf)
> {
>
> int maxThreads = conf.getInt("hbase.htable.threads.max", Integer.
> MAX_VALUE);
>
> Note the default value.
>
> When the number of regions involved is high, consider adjusting the above
> config.
>
> On Wed, Mar 1, 2017 at 11:04 PM, Daniel Połaczański <
> dpolaczan...@gmail.com>
> wrote:
>
> > Yes. I am using batch variant. I will try your variant
> >
> > 02.03.2017 7:53 AM "Jingcheng Du" napisał(a):
> >
> > What APIs are you using to execute the coprocessor?
> > HTable#batchCoprocessorService? It executes the coprocessors regions by
> > regions in each region server.
> > How about using coprocessorService? This API should run coprocessors in
> > parallel for different regions.
> >
> > 2017-03-02 14:32 GMT+08:00 Daniel Połaczański :
> >
> > > I invoke my business logic which is similar to map reduce paradigm.
> > > Everything works. Only performance is problem that regions are not
> > > processed parallely.
> > > I'm thinking about some hack like invoking 4 times rpc from client
> when i
> > > have 4 regions
> > >
> > > 02.03.2017 00:43 "Ted Yu" napisał(a):
> > >
> > > > To my knowledge, there is no support for this type of combination of
> > map
> > > > reduce and coprocessor.
> > > >
> > > > On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański <
> > > dpolaczan...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > It is something like map reduce processing.
> > > > > I want to run map and combine phase inside in a coprocessor for
> every
> > > > > region separately, return partial results to the client. On the
> > client
> > > > > invoke reduce phase and store result in another hbase table.
> > > > >
> > > > > 2017-03-01 23:26 GMT+01:00 Ted Yu :
> > > > >
> > > > > > Can you describe your use case in more detail ?
> > > > > >
> > > > > > What type of custom coprocessor are you loading to the region
> > server
> > > ?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański <
> > > > > dpolaczan...@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > Let's assume that we have cluster consisting from one
> > RegionServer
> > > > and
> > > > > > the
> > > > > > > RegionsServer contains one table consisting from 3 regions.
> > > > > > >
> > > > > > > I would like to process regions in coprocessor parallely. Is
> it
> > > > > > possible?
> > > > > > >
> > > > > > > I observed that currenlty it invokes coproprocessor with all
> the
> > > > > regions
> > > > > > > one by one. So it process at the beginig region1, when finished
> > > > process
> > > > > > > regions2 and ...
> > > > > > >
> > > > > > > Is it possible to change this behaviour?
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: parallel processing of regions in coprocessor
Sent from my BlackBerry 10 smartphone.
Original Message
From: Ted Yu
Sent: Thursday 2 March 2017 15:37
To: user@hbase.apache.org
Reply To: user@hbase.apache.org
Subject: Re: parallel processing of regions in coprocessor
Daniel:
If you don't pass your ExecutorService to HTable ctor, the following would
be called:
public static ThreadPoolExecutor getDefaultExecutor(Configuration conf) {
int maxThreads = conf.getInt("hbase.htable.threads.max", Integer.
MAX_VALUE);
Note the default value.
When the number of regions involved is high, consider adjusting the above
config.
On Wed, Mar 1, 2017 at 11:04 PM, Daniel Połaczański
wrote:
> Yes. I am using batch variant. I will try your variant
>
> 02.03.2017 7:53 AM "Jingcheng Du" napisał(a):
>
> What APIs are you using to execute the coprocessor?
> HTable#batchCoprocessorService? It executes the coprocessors regions by
> regions in each region server.
> How about using coprocessorService? This API should run coprocessors in
> parallel for different regions.
>
> 2017-03-02 14:32 GMT+08:00 Daniel Połaczański :
>
> > I invoke my business logic which is similar to map reduce paradigm.
> > Everything works. Only performance is problem that regions are not
> > processed parallely.
> > I'm thinking about some hack like invoking 4 times rpc from client when i
> > have 4 regions
> >
> > 02.03.2017 00:43 "Ted Yu" napisał(a):
> >
> > > To my knowledge, there is no support for this type of combination of
> map
> > > reduce and coprocessor.
> > >
> > > On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański <
> > dpolaczan...@gmail.com
> > > >
> > > wrote:
> > >
> > > > It is something like map reduce processing.
> > > > I want to run map and combine phase inside in a coprocessor for every
> > > > region separately, return partial results to the client. On the
> client
> > > > invoke reduce phase and store result in another hbase table.
> > > >
> > > > 2017-03-01 23:26 GMT+01:00 Ted Yu :
> > > >
> > > > > Can you describe your use case in more detail ?
> > > > >
> > > > > What type of custom coprocessor are you loading to the region
> server
> > ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański <
> > > > dpolaczan...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > Let's assume that we have cluster consisting from one
> RegionServer
> > > and
> > > > > the
> > > > > > RegionsServer contains one table consisting from 3 regions.
> > > > > >
> > > > > > I would like to process regions in coprocessor parallely. Is it
> > > > > possible?
> > > > > >
> > > > > > I observed that currenlty it invokes coproprocessor with all the
> > > > regions
> > > > > > one by one. So it process at the beginig region1, when finished
> > > process
> > > > > > regions2 and ...
> > > > > >
> > > > > > Is it possible to change this behaviour?
> > > > > >
> > > > > > Regards
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: parallel processing of regions in coprocessor
Daniel:
If you don't pass your ExecutorService to HTable ctor, the following would
be called:
public static ThreadPoolExecutor getDefaultExecutor(Configuration conf) {
int maxThreads = conf.getInt("hbase.htable.threads.max", Integer.
MAX_VALUE);
Note the default value.
When the number of regions involved is high, consider adjusting the above
config.
On Wed, Mar 1, 2017 at 11:04 PM, Daniel Połaczański
wrote:
> Yes. I am using batch variant. I will try your variant
>
> 02.03.2017 7:53 AM "Jingcheng Du" napisał(a):
>
> What APIs are you using to execute the coprocessor?
> HTable#batchCoprocessorService? It executes the coprocessors regions by
> regions in each region server.
> How about using coprocessorService? This API should run coprocessors in
> parallel for different regions.
>
> 2017-03-02 14:32 GMT+08:00 Daniel Połaczański :
>
> > I invoke my business logic which is similar to map reduce paradigm.
> > Everything works. Only performance is problem that regions are not
> > processed parallely.
> > I'm thinking about some hack like invoking 4 times rpc from client when i
> > have 4 regions
> >
> > 02.03.2017 00:43 "Ted Yu" napisał(a):
> >
> > > To my knowledge, there is no support for this type of combination of
> map
> > > reduce and coprocessor.
> > >
> > > On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański <
> > dpolaczan...@gmail.com
> > > >
> > > wrote:
> > >
> > > > It is something like map reduce processing.
> > > > I want to run map and combine phase inside in a coprocessor for every
> > > > region separately, return partial results to the client. On the
> client
> > > > invoke reduce phase and store result in another hbase table.
> > > >
> > > > 2017-03-01 23:26 GMT+01:00 Ted Yu :
> > > >
> > > > > Can you describe your use case in more detail ?
> > > > >
> > > > > What type of custom coprocessor are you loading to the region
> server
> > ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański <
> > > > dpolaczan...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > Let's assume that we have cluster consisting from one
> RegionServer
> > > and
> > > > > the
> > > > > > RegionsServer contains one table consisting from 3 regions.
> > > > > >
> > > > > > I would like to process regions in coprocessor parallely. Is it
> > > > > possible?
> > > > > >
> > > > > > I observed that currenlty it invokes coproprocessor with all the
> > > > regions
> > > > > > one by one. So it process at the beginig region1, when finished
> > > process
> > > > > > regions2 and ...
> > > > > >
> > > > > > Is it possible to change this behaviour?
> > > > > >
> > > > > > Regards
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: parallel processing of regions in coprocessor
Yes. I am using batch variant. I will try your variant 02.03.2017 7:53 AM "Jingcheng Du" napisał(a): What APIs are you using to execute the coprocessor? HTable#batchCoprocessorService? It executes the coprocessors regions by regions in each region server. How about using coprocessorService? This API should run coprocessors in parallel for different regions. 2017-03-02 14:32 GMT+08:00 Daniel Połaczański : > I invoke my business logic which is similar to map reduce paradigm. > Everything works. Only performance is problem that regions are not > processed parallely. > I'm thinking about some hack like invoking 4 times rpc from client when i > have 4 regions > > 02.03.2017 00:43 "Ted Yu" napisał(a): > > > To my knowledge, there is no support for this type of combination of map > > reduce and coprocessor. > > > > On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański < > dpolaczan...@gmail.com > > > > > wrote: > > > > > It is something like map reduce processing. > > > I want to run map and combine phase inside in a coprocessor for every > > > region separately, return partial results to the client. On the client > > > invoke reduce phase and store result in another hbase table. > > > > > > 2017-03-01 23:26 GMT+01:00 Ted Yu : > > > > > > > Can you describe your use case in more detail ? > > > > > > > > What type of custom coprocessor are you loading to the region server > ? > > > > > > > > Thanks > > > > > > > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański < > > > dpolaczan...@gmail.com > > > > > > > > > wrote: > > > > > > > > > Hi, > > > > > Let's assume that we have cluster consisting from one RegionServer > > and > > > > the > > > > > RegionsServer contains one table consisting from 3 regions. > > > > > > > > > > I would like to process regions in coprocessor parallely. Is it > > > > possible? > > > > > > > > > > I observed that currenlty it invokes coproprocessor with all the > > > regions > > > > > one by one. So it process at the beginig region1, when finished > > process > > > > > regions2 and ... > > > > > > > > > > Is it possible to change this behaviour? > > > > > > > > > > Regards > > > > > > > > > > > > > > >
Re: parallel processing of regions in coprocessor
What APIs are you using to execute the coprocessor? HTable#batchCoprocessorService? It executes the coprocessors regions by regions in each region server. How about using coprocessorService? This API should run coprocessors in parallel for different regions. 2017-03-02 14:32 GMT+08:00 Daniel Połaczański : > I invoke my business logic which is similar to map reduce paradigm. > Everything works. Only performance is problem that regions are not > processed parallely. > I'm thinking about some hack like invoking 4 times rpc from client when i > have 4 regions > > 02.03.2017 00:43 "Ted Yu" napisał(a): > > > To my knowledge, there is no support for this type of combination of map > > reduce and coprocessor. > > > > On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański < > dpolaczan...@gmail.com > > > > > wrote: > > > > > It is something like map reduce processing. > > > I want to run map and combine phase inside in a coprocessor for every > > > region separately, return partial results to the client. On the client > > > invoke reduce phase and store result in another hbase table. > > > > > > 2017-03-01 23:26 GMT+01:00 Ted Yu : > > > > > > > Can you describe your use case in more detail ? > > > > > > > > What type of custom coprocessor are you loading to the region server > ? > > > > > > > > Thanks > > > > > > > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański < > > > dpolaczan...@gmail.com > > > > > > > > > wrote: > > > > > > > > > Hi, > > > > > Let's assume that we have cluster consisting from one RegionServer > > and > > > > the > > > > > RegionsServer contains one table consisting from 3 regions. > > > > > > > > > > I would like to process regions in coprocessor parallely. Is it > > > > possible? > > > > > > > > > > I observed that currenlty it invokes coproprocessor with all the > > > regions > > > > > one by one. So it process at the beginig region1, when finished > > process > > > > > regions2 and ... > > > > > > > > > > Is it possible to change this behaviour? > > > > > > > > > > Regards > > > > > > > > > > > > > > >
Re: parallel processing of regions in coprocessor
I invoke my business logic which is similar to map reduce paradigm. Everything works. Only performance is problem that regions are not processed parallely. I'm thinking about some hack like invoking 4 times rpc from client when i have 4 regions 02.03.2017 00:43 "Ted Yu" napisał(a): > To my knowledge, there is no support for this type of combination of map > reduce and coprocessor. > > On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański > > wrote: > > > It is something like map reduce processing. > > I want to run map and combine phase inside in a coprocessor for every > > region separately, return partial results to the client. On the client > > invoke reduce phase and store result in another hbase table. > > > > 2017-03-01 23:26 GMT+01:00 Ted Yu : > > > > > Can you describe your use case in more detail ? > > > > > > What type of custom coprocessor are you loading to the region server ? > > > > > > Thanks > > > > > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański < > > dpolaczan...@gmail.com > > > > > > > wrote: > > > > > > > Hi, > > > > Let's assume that we have cluster consisting from one RegionServer > and > > > the > > > > RegionsServer contains one table consisting from 3 regions. > > > > > > > > I would like to process regions in coprocessor parallely. Is it > > > possible? > > > > > > > > I observed that currenlty it invokes coproprocessor with all the > > regions > > > > one by one. So it process at the beginig region1, when finished > process > > > > regions2 and ... > > > > > > > > Is it possible to change this behaviour? > > > > > > > > Regards > > > > > > > > > >
Re: parallel processing of regions in coprocessor
To my knowledge, there is no support for this type of combination of map reduce and coprocessor. On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański wrote: > It is something like map reduce processing. > I want to run map and combine phase inside in a coprocessor for every > region separately, return partial results to the client. On the client > invoke reduce phase and store result in another hbase table. > > 2017-03-01 23:26 GMT+01:00 Ted Yu : > > > Can you describe your use case in more detail ? > > > > What type of custom coprocessor are you loading to the region server ? > > > > Thanks > > > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański < > dpolaczan...@gmail.com > > > > > wrote: > > > > > Hi, > > > Let's assume that we have cluster consisting from one RegionServer and > > the > > > RegionsServer contains one table consisting from 3 regions. > > > > > > I would like to process regions in coprocessor parallely. Is it > > possible? > > > > > > I observed that currenlty it invokes coproprocessor with all the > regions > > > one by one. So it process at the beginig region1, when finished process > > > regions2 and ... > > > > > > Is it possible to change this behaviour? > > > > > > Regards > > > > > >
Re: parallel processing of regions in coprocessor
It is something like map reduce processing. I want to run map and combine phase inside in a coprocessor for every region separately, return partial results to the client. On the client invoke reduce phase and store result in another hbase table. 2017-03-01 23:26 GMT+01:00 Ted Yu : > Can you describe your use case in more detail ? > > What type of custom coprocessor are you loading to the region server ? > > Thanks > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański > > wrote: > > > Hi, > > Let's assume that we have cluster consisting from one RegionServer and > the > > RegionsServer contains one table consisting from 3 regions. > > > > I would like to process regions in coprocessor parallely. Is it > possible? > > > > I observed that currenlty it invokes coproprocessor with all the regions > > one by one. So it process at the beginig region1, when finished process > > regions2 and ... > > > > Is it possible to change this behaviour? > > > > Regards > > >
Re: parallel processing of regions in coprocessor
Can you describe your use case in more detail ? What type of custom coprocessor are you loading to the region server ? Thanks On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański wrote: > Hi, > Let's assume that we have cluster consisting from one RegionServer and the > RegionsServer contains one table consisting from 3 regions. > > I would like to process regions in coprocessor parallely. Is it possible? > > I observed that currenlty it invokes coproprocessor with all the regions > one by one. So it process at the beginig region1, when finished process > regions2 and ... > > Is it possible to change this behaviour? > > Regards >
parallel processing of regions in coprocessor
Hi, Let's assume that we have cluster consisting from one RegionServer and the RegionsServer contains one table consisting from 3 regions. I would like to process regions in coprocessor parallely. Is it possible? I observed that currenlty it invokes coproprocessor with all the regions one by one. So it process at the beginig region1, when finished process regions2 and ... Is it possible to change this behaviour? Regards
