Re: [google-appengine] How to speed up the task queue execution?

2010-06-23 Thread djidjadji 
It's not a 1 second limitation. But if you hammer with concurrent
transactions on a single entity group you get a lot of "collisions"
and retries and that slows the whole process down. A rule of thumb is
don't try to update a single entity group more then once a second for
a long period of time. It's not about the kind but the individual
instances of a kind. If you have 50 request/s and they all want to
update 50 different objects with transactions it should work. It might
slow down a bit if they all hit the same tablet server but you won't
have collisions

The example at the TaskQueue man page also mentions the 1/s update by
transaction.

2010/6/24 dflorey :
> Back to my initial question... ;-)
> @djidjadji: Thanks for your post! Do you know more about this "1
> second limitiaon"?
> I'm working only on separate entity groups (but they are of the same
> kind) only.
> I've been in that talk at the Google IO but I'm still confused why I'm
> not able to execute more than ~2 tasks per second.
> Is there some global transaction/s limit per app?
> Or is the problem just that my app has low traffic for a long time and
> then I'm starting ~1500 concurrent tasks?
> But doing this for some time, I do not get any warnings about new
> allocated nodes, but it is still not getting faster.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] How to speed up the task queue execution?

2010-06-23 Thread dflorey 
Back to my initial question... ;-)
@djidjadji: Thanks for your post! Do you know more about this "1
second limitiaon"?
I'm working only on separate entity groups (but they are of the same
kind) only.
I've been in that talk at the Google IO but I'm still confused why I'm
not able to execute more than ~2 tasks per second.
Is there some global transaction/s limit per app?
Or is the problem just that my app has low traffic for a long time and
then I'm starting ~1500 concurrent tasks?
But doing this for some time, I do not get any warnings about new
allocated nodes, but it is still not getting faster.


On Jun 23, 5:45 pm, Harshal  wrote:
> Good question Ross. :)
>
> +1
>
> On Wed, Jun 23, 2010 at 9:11 PM, Ross M Karchner 
> wrote:
>
>
>
> > I would love to know how much traffic (or better, how many "topics"
> > and subscriptions) PubSubHubBub serves, and how much it costs (even if
> > nobody really pays for it).
>
> > On Wed, Jun 23, 2010 at 11:09 AM, Geoffrey Spear 
> > wrote:
>
> > > On Jun 22, 9:49 pm, Tristan  wrote:
> > >> User sends a request for a page.. cold start, need new instance  ~
> > >> 10,000 ms = 10s.
> > >> User stares at it for 2 seconds, clicks an interaction... we're
> > >> unlucky, new instance ~10,000 ms = 10s.
> > >> User executes a task that delegates something to the queue. Oh no,
> > >> start of a task, new instance ~10,000 ms = 10s.
>
> > > If your instances are taking 1ms of CPU to start up, you're doing
> > > something horribly wrong.  Period.
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > > To post to this group, send email to google-appeng...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com > e...@googlegroups.com>
> > .
> > > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appeng...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com > e...@googlegroups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] How to speed up the task queue execution?

2010-06-22 Thread dflorey 
I'm using task queue to execute a lot of transactions on different
entity groups in parallel.
But even if I specify 50/s execution rate I end up with only ~120
executed tasks per minute.
I don't get any "too much contention..." exceptions so I have no clue
where the bottleneck is.
Any ideas how to speed things up?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.