Re: [google-appengine] Will cloud Dataflows replace the MapReduce/Pipeline API?
That is correct. It support one namespace. If there are 1000 namespaces, then we have to setup 1000 mapreduce jobs, and monitor them. It is more work and not convenient, if you are doing data processing more often. Please star this issue, if you are using namespace and mapreduce https://code.google.com/p/appengine-mapreduce/issues/detail?id=108 On Fri, Jul 25, 2014 at 12:13 AM, 'Tom Kaitchuck' via Google App Engine google-appengine@googlegroups.com wrote: Mapreduce does support namespaces: https://code.google.com/p/appengine-mapreduce/source/browse/trunk/java/src/main/java/com/google/appengine/tools/mapreduce/inputs/DatastoreInput.java#28 On Thu, Jul 24, 2014 at 12:28 AM, Aswath Satrasala aswath.satras...@gmail.com wrote: Currently, mapreduce does not support namespaces well and takes takes longer time to setup and code. Any idea, on the the dataflow when it will be for public? Will it support for processing namespaces data in appengine? -Aswath On Fri, Jun 27, 2014 at 12:18 AM, 'Tom Kaitchuck' via Google App Engine google-appengine@googlegroups.com wrote: No -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Will cloud Dataflows replace the MapReduce/Pipeline API?
Yes. Multiple namespaces are not directly supported. Given a finite known set, the simplest thing to do would be to construct an InMemoryInput and have the mapper operate on a namespace. You could also, for instance create a new input class that iterated over namespaces. Both of the above are predicated on the idea that the mapper can operate on a namespace as a single item. If this is not practical, there is not much that can be done. It is not possible to issue / shard a datastore query that crosses namespaces. So there is no good way to shard the data. On Tue, Jul 29, 2014 at 4:26 AM, Aswath Satrasala aswath.satras...@gmail.com wrote: That is correct. It support one namespace. If there are 1000 namespaces, then we have to setup 1000 mapreduce jobs, and monitor them. It is more work and not convenient, if you are doing data processing more often. Please star this issue, if you are using namespace and mapreduce https://code.google.com/p/appengine-mapreduce/issues/detail?id=108 On Fri, Jul 25, 2014 at 12:13 AM, 'Tom Kaitchuck' via Google App Engine google-appengine@googlegroups.com wrote: Mapreduce does support namespaces: https://code.google.com/p/appengine-mapreduce/source/browse/trunk/java/src/main/java/com/google/appengine/tools/mapreduce/inputs/DatastoreInput.java#28 On Thu, Jul 24, 2014 at 12:28 AM, Aswath Satrasala aswath.satras...@gmail.com wrote: Currently, mapreduce does not support namespaces well and takes takes longer time to setup and code. Any idea, on the the dataflow when it will be for public? Will it support for processing namespaces data in appengine? -Aswath On Fri, Jun 27, 2014 at 12:18 AM, 'Tom Kaitchuck' via Google App Engine google-appengine@googlegroups.com wrote: No -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Will cloud Dataflows replace the MapReduce/Pipeline API?
Currently, mapreduce does not support namespaces well and takes takes longer time to setup and code. Any idea, on the the dataflow when it will be for public? Will it support for processing namespaces data in appengine? -Aswath On Fri, Jun 27, 2014 at 12:18 AM, 'Tom Kaitchuck' via Google App Engine google-appengine@googlegroups.com wrote: No -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Will cloud Dataflows replace the MapReduce/Pipeline API?
Mapreduce does support namespaces: https://code.google.com/p/appengine-mapreduce/source/browse/trunk/java/src/main/java/com/google/appengine/tools/mapreduce/inputs/DatastoreInput.java#28 On Thu, Jul 24, 2014 at 12:28 AM, Aswath Satrasala aswath.satras...@gmail.com wrote: Currently, mapreduce does not support namespaces well and takes takes longer time to setup and code. Any idea, on the the dataflow when it will be for public? Will it support for processing namespaces data in appengine? -Aswath On Fri, Jun 27, 2014 at 12:18 AM, 'Tom Kaitchuck' via Google App Engine google-appengine@googlegroups.com wrote: No -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
Re: [google-appengine] Will cloud Dataflows replace the MapReduce/Pipeline API?
No -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
[google-appengine] Will cloud Dataflows replace the MapReduce/Pipeline API?
Hi, At the keynote of the Google I/O this year was mentioned about a new API/service called Cloud Dataflow. Will it replace the current (and still in preview) MapReduce API? I don't want to have to refactory my code suddenly. Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
