from T2-T4, the attempts to add tasks to the queue will fail because the Task Queue Stored Task Bytes is at or over quota.
The solution is to give yourself head room. You do this by setting your limit to, say, 500 MB with the follwing in queue.yaml total_storage_limit: 500M Alternatively, in queue.xml <total-storage-limit> 500M </total-storage-limit> Note that your Task Queue Stored Task Bytes also counts toward your Total Stored Data. More Total Stored Data can be obtained by enabling billing. Nick Verne On Thu, Dec 9, 2010 at 2:14 PM, Eric Ka Ka Ng <ngk...@gmail.com> wrote: > Hi Nick, > Thank for your reply! guess i have more understanding on how the Task Queue > Stored Task Bytes be counted > as Task Queue Stored Task Bytes increases in 'near real time' when task is > being added, but decreases 'every couple of hours', so within this 'couple > of hours', this counter is sort of accumulative? > for example, would the following scenario be possible? > assume T4-T3=T3-T2=T2-T1=T1-T0=30 mins, and the counter decreases 2 hours > after T0 (at T4) > before T0, there were no tasks, Task Queue Stored Task Byte = 0 > from T0-T1, many tasks are added, and Task Queue Stored Task Byte = 100MB > from T1-T2, all previous added tasks has been finished processing, leaving > the queue empty. Task Queue Stored Task Byte is still 100MB > from T2-T4, some tasks are added, but since Task Queue Stored Task Byte = > 100MB (over the quota), those tasks would be accumulated in the queue > but cannot be executed, making Task Queue Stored Task Byte continue to be > increasing to let say 200MB > at T4, Task Queue Stored Task Byte started to decrease, by the amount of > task being executed from T1-T2. Task Queue Stored Task Byte = 100MB (over > quota). newly added tasks since T2 still cannot be executed > > > anyway, seems being able to increase the quota for Task Queue Stored Task > Byte would be a MUST-HAVE for us, in order to avoid the tasks being stuck > and hanging the system when there is a short burst of tasks . could you > advise how we could configure this quota on GAE? > Many thanks, > Eric Ng > On 8 December 2010 10:15, Nicholas Verne <nve...@google.com> wrote: >> >> Eric, >> >> Task Queue Stored Task Count is updated in "near real" time. Each task >> that is added to a queue is counted at that time, and each task that >> is deleted after sucessful execution is subtracted. >> Task Queue Stored Task Bytes increases as each task is added, but >> decreases only every couple of hours when a process runs which takes >> an accurate measurement of real usage. >> >> To estimate the average number of bytes/task, you could pause the >> relevant queue, then watch the Task Queue Stored Task Bytes change as >> you add tasks. >> >> Nick Verne >> >> On Tue, Dec 7, 2010 at 8:46 PM, Eric Ka Ka Ng <ngk...@gmail.com> wrote: >> > Once upon a time we experienced problem when we reach the Task Queue >> > Stored >> > Task Bytes Quota Limit (i believe is 100MB for paid application) . It >> > happened when 'producer' produce too many tasks into the queue before >> > the >> > 'consumer' can consume them, leaving many many tasks being in the queue. >> > Although we fixed the problem for 'producer' faster than 'consumer' >> > problem, >> > this trigger us to have a better understanding on how the quota 'Task >> > Queue >> > Stored Task Bytes' be counted and configured >> > >> > Specially i have 3 questions >> > 1. how to count the 'bytes' of a task queue stored task? >> > let say each task consumes 50K, as we should have 100MB of such quota, >> > it >> > should have the capacity to store 20000 tasks. but how could we >> > calculate / >> > estimate the task stored size? >> > 2. is this a counter showing the 'real-time current' total stored task >> > size, >> > or a counter being updated regularly, or an accumulative counter being >> > reset >> > in daily basis? >> > i was thinking it should not be an accumulative counter, but sometimes >> > when >> > i check the system we dont have any tasks in the queue, but the "Tasek >> > Queue >> > Stored Task Bytes" is not 0 >> > >> > 3. how to configure the quota for Task Queue Stored Task Bytes' >> > in GAE >> > >> > documentation http://code.google.com/appengine/docs/quotas.html#Task_Queue, >> > it is stated that we could configure it should be configurable up to >> > "Stored >> > Data" (billable) . however from our current GAE admin console and >> > billing >> > setting, we found no way to configure it. >> > >> > appreciateif anyone can share (partial) answers for these or your >> > thoughts >> > >> > Regards, >> > Eric ng >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Google App Engine" group. >> > To post to this group, send email to google-appeng...@googlegroups.com. >> > To unsubscribe from this group, send email to >> > google-appengine+unsubscr...@googlegroups.com. >> > For more options, visit this group at >> > http://groups.google.com/group/google-appengine?hl=en. >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
