[google-appengine] MapReduce Failures
I'm seeing shards abruptly fail in my MR jobs for no apparent reason and without retrying: task_name=appengine-mrshard-1581047187783C3601732-14-2-retry-0 app_engine_release=1.8.0 instance=00c61b117c53a40e120ac864168a3fe51c2ce Shard 1581047187783C3601732-14 failed permanently. Is there some adjustment I can make to my queue parameters to avoid or reduce these issues? Recently I had been having problems with MR jobs throwing "UnknownErrors" and "ApplicationError followed by "RetrySliceErrors", and setting the min_backoff_seconds to 1 seemed to help with reducing the retry errors. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] app rocket and connecting to external databases
It was my understanding that it isn't possible to connect directly to external databases from app engine because you're limited to making http requests, but I came across App Rocket recently and some how it seems that it is connecting to an external mysql instance. How is App Rocket actually working and is there a way to connect to external databases after all? https://github.com/k7d/approcket/tree/master/rocket -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] export JSON to Cloud Storage
I'm trying to create JSON representations of my models in the datastore and
then write them to a file in Cloud Storage in a MapperPipeline.
The map method looks something like this:
def map_method(entity):
json_message = make_json_message(entity)
yield(json_message)
and the params for the output_writer look something like this:
"output_writer":{
"filesystem": "gs",
"gs_bucket_name": "json_export",
"mime_type": "text/utf-8"
}
The pipeline completes successfully, but the file I'm getting in Cloud
Storage seems to be in a binary format.
How do I get a JSON file out that I can read?
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Run Map Reduce on non-default versions?
Other than being able to customize the bundle, are there any other differences between it and the SDK version? What's the "official" way of using map reduces according to Google? So when you run your mapreduces in a separate queue do you pass in the queue_name to the pipeline.start() command, just like with the normal Pipeline API? On Friday, March 1, 2013 9:48:11 AM UTC-8, Jim Morrison wrote: > > As long as you start the mapreduce on a non-default version it will > continue to run in the non-default version. I also run my mapreduces > through a separate taskqueue. One advantage of using the separate > mapreduce bundle is that you can put in your own changes. E.g. I > pickle the mapper key value pairs so that I can send full objects > between the mapper and reducer. > > On Fri, Mar 1, 2013 at 7:48 AM, Ranjit Chacko > > > wrote: > > > > I have a couple of questions about the App Engine Map Reduce API. First > of > > all there's a mapreduce package in the SDK, and there's a separate > mapreduce > > bundle here: > > > > https://developers.google.com/appengine/downloads > > > > Which one should I be using? Should I be using the bundle, or is the > > documentation out of date and I should actually use the SDK version? > > > > Second I'd like to be able to run mapreduce's on a non-default version > to > > make sure that the requests from the mapreduce don't interfere with user > > requests. > > > > What's the best way to do this? Can I start the pipeline with a task > queue, > > and set the target version of that queue to be my non-default version? > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Google App Engine" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to google-appengi...@googlegroups.com . > > To post to this group, send email to > > google-a...@googlegroups.com. > > > Visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > > > -- > Jim > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Run Map Reduce on non-default versions?
I have a couple of questions about the App Engine Map Reduce API. First of all there's a mapreduce package in the SDK, and there's a separate mapreduce bundle here: https://developers.google.com/appengine/downloads Which one should I be using? Should I be using the bundle, or is the documentation out of date and I should actually use the SDK version? Second I'd like to be able to run mapreduce's on a non-default version to make sure that the requests from the mapreduce don't interfere with user requests. What's the best way to do this? Can I start the pipeline with a task queue, and set the target version of that queue to be my non-default version? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Task Queue parameters for MapReduce
What are some best practices for setting Task Queue parameters for MapReduce jobs? We are currently running MapReduces on a separate queue which targets a non-default version in order to avoid interfering with user serving requests. Is there a good way to keep the MapReduce jobs on the default version, and be sure it won't block user serving requests? Also we recently had a problem with MapReduce jobs failing, and I think the reason was that the task_retry_limit was set too low which seems to have been the problem in this issue as well: http://code.google.com/p/googleappengine/issues/detail?id=8621 Are there any recommendations regarding retry parameters that came out of the resolution of that ticket? Thanks, -Ranjit -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/uTNubqUSxWoJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
