Re: [google-appengine] mapreduce and cron job
You must use the control module from the mapreduce module
One line in the doc tells you to chain MR tasks use a done callback
and start a new one with the control submodule.
Use this method to start a task from inside a cron handler
Have a look at mapreduce/control.py and the test code for control.py
from mapreduce import control as mr_control
class MRCronHandler(webapp.RequestHandler):
def get(self):
mr_control.start_map(
"Name of the task",
"mymr.sometask",
"mapreduce.input_readers.DatastoreInputReader",
{"entity_kind": "models.SomeModel",
},
mapreduce_parameters={"done_callback": "/mr-end-task"},
)
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write('MR Cron Started')
2010/10/24 slash :
> Hi, everybody
>
> I learned a lot from this group and Nick's blog. Thanks.
>
> I want to combine mapreduce and cron job in order to run mapping
> automatically. I did lots of search in this group and from other
> resource but I got nothing.
>
> I'm able to run cron job and mapreduce separately. But I really
> don't know how to combine them. Maybe I missed something really basic.
>
> Could anyone tell me how to run mapreduce with cron job? I didn't
> find the corresponding url to trigger mapreduce. Thank you.
>
> PS: I know that it's possible to do this and I subscribe issue 41
> in appengine-mapreduce project.
>
> Here is the link:
> http://code.google.com/p/appengine-mapreduce/issues/detail?id=41.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appeng...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] mapreduce and cron job
Oops! My bad, this won't work. It must be Monday. Sorry, Steve On Mon, Oct 25, 2010 at 11:56 AM, Stephen Johnson wrote: > Hello, > Java: Put something like this in your cron.xml (for Python modify > accordingly). > > > > > /cron/map-reduce-start?map-reduce-path=/mapreduce/&mapper-class=YOUR.PACKAGE.YOURMAPPERCLASSHERE&entity-kind=YOURENTITYKINDHERE&num-shards=YOURNUMBERSHARDSHERE&processing-rate=YOURPROCESSINGRATEHERE&controller-queue=YOURCONTROLLERQUEUEHERE&worker-queue=YOURWORKERQUEUEHERE&callback-queue=YOURCALLBACKQUEUEHERE&callback-url=YOURCALLBACKURLHERE > > Some description > > every 1 hours > > > > Stephen > > > On Sun, Oct 24, 2010 at 3:14 AM, slash wrote: > >> Hi, everybody >> >> I learned a lot from this group and Nick's blog. Thanks. >> >> I want to combine mapreduce and cron job in order to run mapping >> automatically. I did lots of search in this group and from other >> resource but I got nothing. >> >>I'm able to run cron job and mapreduce separately. But I really >> don't know how to combine them. Maybe I missed something really basic. >> >>Could anyone tell me how to run mapreduce with cron job? I didn't >> find the corresponding url to trigger mapreduce. Thank you. >> >>PS: I know that it's possible to do this and I subscribe issue 41 >> in appengine-mapreduce project. >> >>Here is the link: >> http://code.google.com/p/appengine-mapreduce/issues/detail?id=41. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] mapreduce and cron job
Hello, Java: Put something like this in your cron.xml (for Python modify accordingly). /cron/map-reduce-start?map-reduce-path=/mapreduce/&mapper-class=YOUR.PACKAGE.YOURMAPPERCLASSHERE&entity-kind=YOURENTITYKINDHERE&num-shards=YOURNUMBERSHARDSHERE&processing-rate=YOURPROCESSINGRATEHERE&controller-queue=YOURCONTROLLERQUEUEHERE&worker-queue=YOURWORKERQUEUEHERE&callback-queue=YOURCALLBACKQUEUEHERE&callback-url=YOURCALLBACKURLHERE Some description every 1 hours Stephen On Sun, Oct 24, 2010 at 3:14 AM, slash wrote: > Hi, everybody > > I learned a lot from this group and Nick's blog. Thanks. > > I want to combine mapreduce and cron job in order to run mapping > automatically. I did lots of search in this group and from other > resource but I got nothing. > >I'm able to run cron job and mapreduce separately. But I really > don't know how to combine them. Maybe I missed something really basic. > >Could anyone tell me how to run mapreduce with cron job? I didn't > find the corresponding url to trigger mapreduce. Thank you. > >PS: I know that it's possible to do this and I subscribe issue 41 > in appengine-mapreduce project. > >Here is the link: > http://code.google.com/p/appengine-mapreduce/issues/detail?id=41. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] mapreduce and cron job
Hi, everybody I learned a lot from this group and Nick's blog. Thanks. I want to combine mapreduce and cron job in order to run mapping automatically. I did lots of search in this group and from other resource but I got nothing. I'm able to run cron job and mapreduce separately. But I really don't know how to combine them. Maybe I missed something really basic. Could anyone tell me how to run mapreduce with cron job? I didn't find the corresponding url to trigger mapreduce. Thank you. PS: I know that it's possible to do this and I subscribe issue 41 in appengine-mapreduce project. Here is the link: http://code.google.com/p/appengine-mapreduce/issues/detail?id=41. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
