The urls at /work/pull_feeds and /work/push_events seem to manually trigger the 'background' work of pulling feeds and pushing events..but I've still yet to actually get it working :p In the demo vid it seems to work beautifully, with instantaneous results.
There is a tiny group for pubsubhubub, currently with no messages..I could leave one asking about it, but it seems somehow wrong for me to go in with the first post being some noob question :p I know Brett Slatkin watches this group too.. emi420 - thanks for the link, I actually came across your site yesterday during my research. How do you manage feeding updates to followers? Do you actually write updates n times to n different message queues? On Mar 12, 12:30 pm, bFlood <bflood...@gmail.com> wrote: > right, i can't really tell either. it looks like the churn of incoming > requests is powering the "background" workers, the real change is the > async proxy (AsyncAPIProxy.start_call) to fire the workers off without > affecting the incoming request (at least I think thats what is > happening...) > > it would be nice to see the code behind this import (since it contains > the AsyncRPC class) > from google3.apphosting.runtime import > _apphosting_runtime___python__apiproxy > > cheers > brian > > On Mar 11, 6:17 pm, peterk <peter.ke...@gmail.com> wrote: > > > The app is actually live here: > > >http://pubsubhubbub.appspot.com/http://pubsubhubbub-subscriber.appspo... > > > (pubsubhubbub-publisher isn't there, but it's trivial to upload your > > own.) > > > This suggests it's working on appengine as it is now. Been looking > > through the source, and I'm not entirely clear on how the 'background > > workers' are actually working..there are two, one for pulling updates > > to feeds from publishers, and one for propogating updates to > > subscribers in batches. > > > But like I say, I can't see how they're actually started and running > > constantly. There is a video here of a live demonstration: > > >http://www.veodia.com/player.php?vid=fCNU1qQ1oSs > > > The background workers seem to be behaving as desired there, but I'm > > not sure if they were just constantly polling some urls to keep the > > workers live for the purposes of that demo, or if they're actually > > running somehow constantly on their own.. I can't actually get the > > live app at the urls above to work, but not sure if it's because > > background workers aren't really working, or because i'm feeding it > > incorrect urls/configuration etc. > > > On Mar 11, 8:01 pm, bFlood <bflood...@gmail.com> wrote: > > > > great find peterk! cant help but notice the very interesting > > > async_apiproxy.py code in that project either...async db and url calls > > > would be awesome > > > > cheers > > > brian > > > > On Mar 11, 3:22 pm, peterk <peter.ke...@gmail.com> wrote: > > > > > I've done a good bit of googling about what exactly it is I'm trying > > > > to achieve, the correct terminology etc. and turns out I'm really > > > > looking for an efficient way to do pub/sub, publication/subscription, > > > > on appengine. > > > > > Which led me to this eyebrow-raising little app from Brett Slakin and > > > > Brad Fitzpatrick: > > > > >http://code.google.com/p/pubsubhubbub/ > > > > > Jaiku folk seem to be involved too. > > > > > Basically they have an experimental pub/sub system running on GAE, > > > > using http instead of xmpp. The goals are somewhat broader than my > > > > requirements - basically you have an atom feed at one url you post > > > > data to, the atom feed updates a hub, and the hub then updates your > > > > subscribers at different urls, so it's all a lot more distributed than > > > > I need. > > > > > I imagine one could do something similar rolled into one application > > > > that handles publishing, the hub, and subscription (e.g. like jaiku or > > > > twitter). But the big minus point that sticks out here for me in > > > > pubsubhubub is that it completely avoids the question of the total > > > > write-cost per update. It can afford to do that because subscribers > > > > here are distributed all over the place on different domains and > > > > servers etc. so the write load gets spread across all those different > > > > urls/machines..it's not the concern of the hub running on appengine. > > > > But if pubs/subs/hub are all rolled into one application on appengine, > > > > then your application will be soaking up the total write cost. The hub > > > > handles updating each subscriber in the background, I think, so it > > > > does solve the issue of writing to n subscribers in the space of one > > > > request..the publisher issues its update to the hub, returns > > > > immediately, and then the hub partitions out the subscriber updates > > > > and works on updating everyone over time. But it doesn't solve the > > > > issue of monetary cost of doing n writes for n subscribers...if one > > > > publisher had lots of subscribers (e.g. hundreds of thousands or > > > > millions), each of their updates could cost $$$! > > > > > But I guess one might be safe in assuming the average number of subs > > > > per publisher will be much lower than that..and make the idea > > > > financially feasible.. > > > > > Anyway, I thought this project looked very interesting...it's giving > > > > me a lot to chew on now.. > > > > > On Mar 11, 3:50 pm, peterk <peter.ke...@gmail.com> wrote: > > > > > > Hmm. Good thinking Andrew. I've done a bit more digging along those > > > > > lines..I don't think Jaiku has publically said if and what messaging > > > > > protocols they might use, but there seems to be some speculation that > > > > > it uses XMPP. > > > > > > That might also neatly align with their comment that they are in some > > > > > cases "extending" GAE in their port..and the recent news that XMPP is > > > > > scheduled to come to GAE at some point in the next 6 months. Perhaps > > > > > it's the work on the Jaiku port that's bringing that functionality... > > > > > > I didn't know anything about messaging protocols or XMPP until I dug > > > > > around after your post, so it's all new to me. I found a project > > > > > purporting to provide XMPP functionality in advance of Google's > > > > > official support, athttp://imified-demo.appspot.com/, but looking at > > > > > the code for it, I cannot for the life of me see how this could be > > > > > used to address the problems discussed in my original email. That > > > > > project seems only to take instant messages via a POST to a URL and > > > > > save them to the datastore..but there doesn't seem to be any of the > > > > > 'instant distribution' of updates to people subscribed to your > > > > > presence etc. that other XMPP software boast about. Using that xmpp > > > > > project linked, there doesn't seem to be anything there about pushing > > > > > updates to subscribed users etc. > > > > > > I dug around the app gallery for other open source apps along similar > > > > > lines..there are some would-be twitter/jaiku clones..all I've seen use > > > > > non-messaging approaches, but they all make one compromise or another > > > > > similar to solutions I'd come up with myself (e.g. taking the last 5 > > > > > updates from friends and sorting them yourself in memory by > > > > > date..which has its own issues, for example, if one user had made >5 > > > > > updates recently..+ you'd inevitably hit a roadblock on all these > > > > > reads beyond a certain number of friends..there's only so many friends > > > > > you could do this for before it gets too slow). > > > > > > Any thoughts/ideas...? > > > > > > On Mar 11, 1:54 pm, Andrew Badera <and...@badera.us> wrote: > > > > > > > you'd have to wonder if there's a push out to Gnip somewhere ... or > > > > > > if > > > > > > protocol buffers are involved .. > > > > > > > Thanks- > > > > > > - Andy Badera > > > > > > - and...@badera.us > > > > > > - (518) 641-1280 > > > > > > - Tech Valley Code Camp 2009.1:http://www.techvalleycodecamp.com/ > > > > > > - Google me:http://www.google.com/search?q=andrew+badera > > > > > > > On Wed, Mar 11, 2009 at 9:21 AM, peterk <peter.ke...@gmail.com> > > > > > > wrote: > > > > > > > > I just read on your blog (from January) the intention to release > > > > > > > the > > > > > > > appengine port of Jaiku as open source when the port is > > > > > > > finished..but > > > > > > > I was wondering if I could be so cheeky as to jump ahead with a > > > > > > > couple > > > > > > > of questions about it. The requirements of Jaiku seem to line up > > > > > > > roughly similarly with issues I'm having in a slightly different > > > > > > > context, that I'm finding pretty challenging to implement > > > > > > > efficiently > > > > > > > on app engine. > > > > > > > > With your service, you seem to track updates for friends and other > > > > > > > people I follow..so I might have a long list of people I'm > > > > > > > following, > > > > > > > and you feed me their updates. > > > > > > > > How do you implement this on GAE? > > > > > > > > I've been toying with a very similar problem for some time now. It > > > > > > > seems to me you cannot chain together queries such as > > > > > > > me.friends.updates.order(..) to get your friends' latest updates, > > > > > > > for > > > > > > > example. You can't make n writes to n update queues for n people > > > > > > > following you, since writes are so costly. If I store my friends > > > > > > > in a > > > > > > > list of keys, this limits the number of friends I can query at a > > > > > > > given > > > > > > > time to 30. e.g. updates.all().filter('user IN', me.friends) is > > > > > > > limited to 30 subqueries. I may have many more friends, so this > > > > > > > approach doesn't seem to be sufficient. > > > > > > > > I've been scratching my head over a similar problem for some time > > > > > > > now, > > > > > > > coming up with various hairbrained schemes that have been overly- > > > > > > > complex, none of which deliver scalability to the nth degree. So > > > > > > > I'd > > > > > > > really, really, really appreciate any insight you could provide in > > > > > > > implementing this kind of data model on GAE!! Many thanks! --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
