Just Curious, For other pub/sub-style systems where you want to write to the Datastore, the trick is to use list properties to track the subscribers you've published to. So for instance, instead of writing a single entity per subscriber, you write one entity with 1000-2000 subscriber IDs in a list. Then all queries for that list with an equals filter for the subscriber will show the entity. This lets you pack a lot of information into a single entity write, thus minimizing Datastore overhead, cost, etc. Does that make sense?
So if you have over the 5000 limit in the subscribers would you write the entity twice? Each with differnt subscriber id's? Paul 2009/3/13 Brett Slatkin <brett-appeng...@google.com> > > Heyo, > > Good finds, peterk! > > pubsubhubbub uses some of the same techniques that Jaiku uses for > doing one-to-many fan-out of status message updates. The migration is > underway as we speak > (http://www.jaiku.com/blog/2009/03/11/upcoming-service-break/). I > believe the code should be available very soon. > > 2009/3/11 peterk <peter.ke...@gmail.com>: > > > > The app is actually live here: > > > > http://pubsubhubbub.appspot.com/ > > http://pubsubhubbub-subscriber.appspot.com/ > > > > (pubsubhubbub-publisher isn't there, but it's trivial to upload your > > own.) > > > > This suggests it's working on appengine as it is now. Been looking > > through the source, and I'm not entirely clear on how the 'background > > workers' are actually working..there are two, one for pulling updates > > to feeds from publishers, and one for propogating updates to > > subscribers in batches. > > > > But like I say, I can't see how they're actually started and running > > constantly. There is a video here of a live demonstration: > > > > http://www.veodia.com/player.php?vid=fCNU1qQ1oSs > > > > The background workers seem to be behaving as desired there, but I'm > > not sure if they were just constantly polling some urls to keep the > > workers live for the purposes of that demo, or if they're actually > > running somehow constantly on their own.. I can't actually get the > > live app at the urls above to work, but not sure if it's because > > background workers aren't really working, or because i'm feeding it > > incorrect urls/configuration etc. > > Ah sorry yeah I still have the old version of the source running on > pubsubhubbub.appspot.com; I need to update that with a more recent > build. Sorry for the trouble! It's still not quite ready for > widespread use, but it should be soon. > > The way pubsubhubbub does fan-out, there's no need to write an entity > for each subscriber of a feed. Instead, each time it consumes a task > from the work queue it will update the current iterator position in > the query result of subscribers for a URL. Subsequent work requests > will offset into the subscribers starting at the iterator position. > This works well in this case because it's using urlfetch to actually > notify subscribers, instead of writing to the Datastore. > > For other pub/sub-style systems where you want to write to the > Datastore, the trick is to use list properties to track the > subscribers you've published to. So for instance, instead of writing a > single entity per subscriber, you write one entity with 1000-2000 > subscriber IDs in a list. Then all queries for that list with an > equals filter for the subscriber will show the entity. This lets you > pack a lot of information into a single entity write, thus minimizing > Datastore overhead, cost, etc. Does that make sense? > > > @bFlood: Indeed, the async_apiproxy.py code is interesting. Not much > to say about that at this time, besides the fact that it works. =) > > -Brett > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
