You may set giraph.userPartitionCount=<number of workers> and giraph.maxPartitionsInMemory=1. Like Avery said though, since parallelism occurs on a partition level (each thread processes a different partition) if you only have one partition per worker you cannot take advantage of multithreading.
Best, Matthew On Fri, Oct 24, 2014 at 3:53 AM, Zhang, David (Paypal Risk) < pengzh...@ebay.com> wrote: > I think no good solution. You can try to run a java application by using > FileInputFormat.getSplits to get the size of the array, which number you > can set to giraph workers. > > Or run a simple map-reduce job by using IdentityMapper to see how many > mappers there. > > > > Thanks, > > Zhang, David (Paypal Risk) > > *From:* Charith Wickramarachchi [mailto:charith.dhanus...@gmail.com] > *Sent:* 2014年10月24日 5:37 > *To:* user > *Subject:* Re: Resource Allocation Model Of Apache Giraph > > > > Thanks Claudio and Avery, > > > > I find a way way to configure hadoop to have desired number of mappers > per machine as Claudio mentioned. > > > > Avery, > > Could you please tell me how I can configure giraph to make each worker > handle only a single partition? > > > > Thanks, > Charith > > > > On Thu, Oct 23, 2014 at 2:26 PM, Avery Ching <ach...@apache.org> wrote: > > Regarding your second point, partitions are decoupled from workers. A > worker can handle zero or more partitions. You can make each worker handle > one partition, but we typically like multiple partitions since we can use > multi-threading per machine. > > > > On 10/23/14, 9:04 AM, Claudio Martella wrote: > > the way mappers (or containers) and hence workers are assigned to > machines is not under the control of giraph, but of the underlying hadoop > environment (with different responsibilities that depend on the hadoop > version, e.g. YARN). You'll have to tweak your hadoop configuration to > control the maximum number of workers assigned to one machine (optimally > one with multiple threads). > > > > On Thu, Oct 23, 2014 at 5:53 PM, Charith Wickramarachchi < > charith.dhanus...@gmail.com> wrote: > > Hi Folks, > > > > I'm wondering what is the resource allocation model for Apache Giraph. > > > > As I understand each worker is one to one Mapped with a Mapper and a > worker can process multiple partitions with a user defined number of > threads. > > > > Is it possible to make sure that one worker, only process a single > partition? Also is it possible to control the worker assignment in the > cluster nodes? (Ex: Make sure only N workers runs on a single machine, > assuming we have enough resources) > > > > Thanks, > > Charith > > > > > > > > > > > > > > > > > > -- > > Charith Dhanushka Wickramaarachchi > > > > Tel +1 213 447 4253 > > Web http://apache.org/~charith <http://www-scf.usc.edu/%7Ecwickram/> > > Blog http://charith.wickramaarachchi.org/ > <http://charithwiki.blogspot.com/> > > Twitter @charithwiki <https://twitter.com/charithwiki> > > > > This communication may contain privileged or other > confidential information and is intended exclusively for the addressee/s. > If you are not the intended recipient/s, or believe that you may have > received this communication in error, please reply to the > sender indicating that fact and delete the copy you received and in > addition, you should not print, copy, retransmit, disseminate, or otherwise > use the information contained in this communication. > Internet communications cannot be guaranteed to be timely, secure, error > or virus-free. The sender does not accept liability for any errors > or omissions > > > > > > -- > > Claudio Martella > > > > > > > > > -- > > Charith Dhanushka Wickramaarachchi > > > > Tel +1 213 447 4253 > > Web http://apache.org/~charith <http://www-scf.usc.edu/~cwickram/> > > Blog http://charith.wickramaarachchi.org/ > <http://charithwiki.blogspot.com/> > > Twitter @charithwiki <https://twitter.com/charithwiki> > > > > This communication may contain privileged or other > confidential information and is intended exclusively for the addressee/s. > If you are not the intended recipient/s, or believe that you may have > received this communication in error, please reply to the > sender indicating that fact and delete the copy you received and in > addition, you should not print, copy, retransmit, disseminate, or otherwise > use the information contained in this communication. > Internet communications cannot be guaranteed to be timely, secure, error > or virus-free. The sender does not accept liability for any errors > or omissions >
