[google-appengine] MapReduce as Cron jobs: How to specify the number of shards?
Hey there Today I created a library to run MapReduces as cron jobs in python. See here: http://devblog.miumeet.com/2011/02/schedule-mapreduce-daily-on-appengine.html However, I didn't figure out how to I'm able to set the shard_count programmatically. In mapreduce/control.py there is a function I call: def start_map(name, handler_spec, reader_spec, reader_parameters, shard_count=_DEFAULT_SHARD_COUNT, [...]) However, no matter what o I pass as the *shard_count argument, it is ignored.* * * Any ideas? Cheers, -Andrin -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] MapReduce as Cron jobs: How to specify the number of shards?
In your cron_mapreduce.py add these two lines
shard_count=int(self.request.get("shard_count",
mr_control._DEFAULT_SHARD_COUNT))
mr_control.start_map(
self.request.get("name"),
self.request.get("reader_spec", "your_mapreduce.map"),
self.request.get("reader_parameters",
"mapreduce.input_readers.DatastoreInputReader"),
{ "entity_kind": self.request.get("entity_kind", "models.YourModel"),
"processing_rate": int(self.request.get("processing_rate", 100)) },
shard_count = shard_count,
mapreduce_parameters={"done_callback": self.request.get("done_callback",
None) } )
2011/2/10 Andrin von Rechenberg :
> Hey there
> Today I created a library to run MapReduces as cron jobs in python.
> See
> here: http://devblog.miumeet.com/2011/02/schedule-mapreduce-daily-on-appengine.html
> However, I didn't figure out how to I'm able to set the
> shard_count programmatically.
> In mapreduce/control.py there is a function I call:
>
> def start_map(name,
>
> handler_spec,
>
> reader_spec,
>
> reader_parameters,
>
> shard_count=_DEFAULT_SHARD_COUNT,
>
> [...])
>
> However, no matter what o I pass as the shard_count argument, it is ignored.
>
> Any ideas?
>
> Cheers,
>
> -Andrin
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] MapReduce as Cron jobs: How to specify the number of shards?
Today I updated the mapreduce library.
I see the same "only 1 shard" when I use the dev_server.
The dev_server does not have the __scatter__ property of objects.
The mapreduce library then falls back to a single shard.
And on the production it depends on how many objects have a
__scatter__ property.
If less then shard_count have __scatter__ you get less shards.
GAE Team: What determines if an object gets a __scatter__ property?
2011/2/11 djidjadji :
> In your cron_mapreduce.py add these two lines
>
> shard_count=int(self.request.get("shard_count",
> mr_control._DEFAULT_SHARD_COUNT))
>
> mr_control.start_map(
> self.request.get("name"),
> self.request.get("reader_spec", "your_mapreduce.map"),
> self.request.get("reader_parameters",
> "mapreduce.input_readers.DatastoreInputReader"),
> { "entity_kind": self.request.get("entity_kind", "models.YourModel"),
> "processing_rate": int(self.request.get("processing_rate", 100)) },
> shard_count = shard_count,
> mapreduce_parameters={"done_callback": self.request.get("done_callback",
> None) } )
>
>
> 2011/2/10 Andrin von Rechenberg :
>> Hey there
>> Today I created a library to run MapReduces as cron jobs in python.
>> See
>> here: http://devblog.miumeet.com/2011/02/schedule-mapreduce-daily-on-appengine.html
>> However, I didn't figure out how to I'm able to set the
>> shard_count programmatically.
>> In mapreduce/control.py there is a function I call:
>>
>> def start_map(name,
>>
>> handler_spec,
>>
>> reader_spec,
>>
>> reader_parameters,
>>
>> shard_count=_DEFAULT_SHARD_COUNT,
>>
>> [...])
>>
>> However, no matter what o I pass as the shard_count argument, it is ignored.
>>
>> Any ideas?
>>
>> Cheers,
>>
>> -Andrin
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] MapReduce as Cron jobs: How to specify the number of shards?
This document explains the strategy:
http://code.google.com/p/appengine-mapreduce/wiki/ScatterPropertyImplementation
It says tht there is a .8% chance of an entity getting this property. That
seems really low. I wonder if they meant 8% not .8%?
Stephen
On Mon, Feb 14, 2011 at 7:12 AM, djidjadji wrote:
> Today I updated the mapreduce library.
> I see the same "only 1 shard" when I use the dev_server.
> The dev_server does not have the __scatter__ property of objects.
> The mapreduce library then falls back to a single shard.
>
> And on the production it depends on how many objects have a
> __scatter__ property.
> If less then shard_count have __scatter__ you get less shards.
>
> GAE Team: What determines if an object gets a __scatter__ property?
>
> 2011/2/11 djidjadji :
> > In your cron_mapreduce.py add these two lines
> >
> > shard_count=int(self.request.get("shard_count",
> > mr_control._DEFAULT_SHARD_COUNT))
> >
> > mr_control.start_map(
> > self.request.get("name"),
> > self.request.get("reader_spec", "your_mapreduce.map"),
> > self.request.get("reader_parameters",
> > "mapreduce.input_readers.DatastoreInputReader"),
> > { "entity_kind": self.request.get("entity_kind", "models.YourModel"),
> > "processing_rate": int(self.request.get("processing_rate", 100)) },
> >shard_count = shard_count,
> > mapreduce_parameters={"done_callback":
> self.request.get("done_callback",
> > None) } )
> >
> >
> > 2011/2/10 Andrin von Rechenberg :
> >> Hey there
> >> Today I created a library to run MapReduces as cron jobs in python.
> >> See
> >> here:
> http://devblog.miumeet.com/2011/02/schedule-mapreduce-daily-on-appengine.html
> >> However, I didn't figure out how to I'm able to set the
> >> shard_count programmatically.
> >> In mapreduce/control.py there is a function I call:
> >>
> >> def start_map(name,
> >>
> >> handler_spec,
> >>
> >> reader_spec,
> >>
> >> reader_parameters,
> >>
> >> shard_count=_DEFAULT_SHARD_COUNT,
> >>
> >> [...])
> >>
> >> However, no matter what o I pass as the shard_count argument, it is
> ignored.
> >>
> >> Any ideas?
> >>
> >> Cheers,
> >>
> >> -Andrin
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
