Re: [google-appengine] Task Queue Strange Errors/Defects

[nischalshetty]

You can also schedule a cron job if calling a particular URL every n seconds 
is your only aim. Having said that, tasks run only once. But in cases of 
failures, they can be run again. Not sure, but is there a way to set a limit 
on the number of tasks in a queue? If there's an option like that, try 
setting the size to 1. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Advice on calculating statistics on Datastore entities

[nischalshetty]

IMO a map reduce would be good. But then it runs over all your entities of 
the specified kind so you need to decide if you are ready for the overhead. 
If you have a lot of data it can be a pain. 

I read somewhere that mapreduce over a subset will someday be implemented, 
of course someday can mean anything. Interested to see how you end up 
solving this because in a few months we will be wanting to do something 
similar. Keep this thread updated on your trials and erros.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Advice on calculating statistics on Datastore entities

[Jamie]

I'm currently storing entities of a particular kind (e.g.
TrafficHits).  I would like to begin removing older entities (e.g.
older than a month), but would like to calculate summary statistics on
these records before deleting them.

What's the recommended approach for this (I'm using Java JDO)?

1) Perform a query to retrieve entities older than a month.  Loop
through records and calculate stats.  Delete entities using a delete
query with same where clause.  This seems somewhat inefficient.

2) A mapreduce job?

Any help would be appreciated.

Thanks,
J

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Are we doing it right with the "label" implementaion?

[Nischal Shetty]

I started allocating negative numbers as ids to my entities(these would be
part of the key) and it seems to work.

The appengine returns results sorted in the ascending order of keys and
hence if I provide keys as -8, -9, -10 etc., I would get the results as -10,
-9, -8 and so on which suffices my use case :)



On 30 April 2011 22:53, Robert Kluin  wrote:

> Hey Nischal,
>   You could insert some records with known key names in various
> orders and a common value for a property to query on.  If the results
> come back out key-ordered, that would be a good indication that the
> method might work.  Also, I *think* there are some videos, groups
> posts, and references in the docs (user or code) mentioning entities
> in key order.
>
>
>
> Robert
>
>
>
>
> On Sat, Apr 23, 2011 at 23:59, Nischal Shetty 
> wrote:
> > @Robert Thank you so much! Any idea how I can confirm on the timestamp +
> key
> > thingy? I always need it based on timestamp (descending). If that works
> then
> > it would help me a lot!
> > I have looked into exploding indexes. My query would always be equality
> > checks over list properties exclusively. I'm hoping that would not be a
> > problem?
> > Example:
> > select from post
> > where listproperty := label1 && listproperty := label2
> > sortby time desc
> > This is what we currently have. If the timestamp thingy is true then we
> > would be able to remove the last line and that would be so damn cool :)
> > On 24 April 2011 08:24, Robert Kluin  wrote:
> >>
> >> You probably need to be aware of exploding indexes if you're indexing
> >> a list property.  It is just something you should consider and be
> >> aware of.
> >>
> >>
> http://code.google.com/appengine/docs/python/datastore/queries.html#Big_Entities_and_Exploding_Indexes
> >>
> >> If you're always planning to sort by timestamp, you could consider
> >> prepending the timestamp on the Post entity keys.  Your entities
> >> should then be returned to the app sorted by timestamp without
> >> explicitly specifying the sort order.  At least that is the case if I
> >> recall correctly.
> >>
> >>
> >> Robert
> >>
> >>
> >>
> >>
> >> On Sat, Apr 23, 2011 at 13:14, nischalshetty  >
> >> wrote:
> >> > We're building a new web product.
> >> > 1. We have the concept of label in them.
> >> > 2. Each post gets one or more label.
> >> > 3. The posts have to be displayed sorted according to timestamp in
> >> > descending order
> >> > We're making use of List (ListProperty) to save these labels
> for
> >> > each post. It's working well for us. We can now query over multiple
> >> > labels
> >> > and retrieve posts.
> >> > However, we need the posts to be sorted on timestamp in descending
> >> > order.
> >> > Due to this our query would have the timestamp as the sort order along
> >> > with
> >> > the label names. Due to this, we need indexes. And the indexes are
> being
> >> > built for combination of the labels and timestamp.
> >> > So, if our query has 2 labels and the sort order then the index would
> be
> >> > built for these. If we query for 3 labels and sort order we would have
> a
> >> > new
> >> > index and so on.
> >> > I wanted to know if there is any other way such that we can avoid the
> >> > indexes? Are indexes cool if you anticipate a LOT of data everyday? Is
> >> > there
> >> > a way to have the posts sorted based on timestamp in a different way?
> >> >
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "Google App Engine" group.
> >> > To post to this group, send email to
> google-appengine@googlegroups.com.
> >> > To unsubscribe from this group, send email to
> >> > google-appengine+unsubscr...@googlegroups.com.
> >> > For more options, visit this group at
> >> > http://groups.google.com/group/google-appengine?hl=en.
> >> >
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-appengine@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengine+unsubscr...@googlegroups.com.
> >> For more options, visit this group at
> >> http://groups.google.com/group/google-appengine?hl=en.
> >>
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> > http://groups.google.com/group/google-appengine?hl=en.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http:

Re: [google-appengine] Re: Blobstore upload handler is Base64 encoding strings?

[Brandon Donnelson]

Is http://code.google.com/p/googleappengine/issues/detail?id=4265 related? 
I'm getting base64 back after the upload, which I pretty sure the server 
should decode it. I'm not sure about character issues. Also, blobinfo is 
recording the wrong file byte size for base64 files, which after comparing 
the sizes of files it appears the headers are being included in the 
calculation. 

My goals is to upload images drawn in HTML5 canvas.

I made a demo and source to show that base64 encoded upload is failing.
http://demogwtcanvas.appspot.com
http://code.google.com/p/gwt-examples/source/browse/trunk/DemoGwtCanvas/src/org/gonevertical/democanvas/client/UploadImage.java
http://code.google.com/p/gwt-examples/wiki/gwt_hmtl5 - wiki stuff about it

http://code.google.com/p/googleappengine/issues/detail?id=4265

What are your thoughts on this?

Brandon Donnelson
http://gwt-examples.googlecode.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Files API continuous random errors,

[JH]

Here is what I am doing:
files.finalize(file_name)
blob_key = files.blobstore.get_blob_key(file_name)
#fix to wait for blob_key consistency
while not blob_key:
time.sleep(1)
blob_key = files.blobstore.get_blob_key(file_name)

There is an issue open and Google says they are working on a fix

On Apr 30, 2:43 pm, Kaan Soral  wrote:
> How can we prevent it?
> Do you think retrying until null is a good approach?
>
> On Apr 30, 5:56 pm, Brandon Donnelson  wrote:
>
>
>
>
>
>
>
> > Mee too! I'm wondering how long it will take. I switched to uploading via
> > posting on the server side.
>
> > Brandon Donnelsonhttp://gwt-examples.googlecode.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Task Queue Strange Errors/Defects

[Ravi Sharma]

Hi Robert,
How naming the task will help me?Sorry i didnt understand it. But anyways i
have named them now.

In My code i have made sure that whatever exception comes from my code will
not be thrown out of my controller and will just print it in log. So in what
scenerio it failed and retried and also before failing completely it added
next task in the queue which is my last statment in my controller.(Thats how
i got more then 1 task in my queue)
And if it failed to add Task then it should retry the current task and i
should still have 1 active task only.

and today also i found that my task stopped. I dont know how can i make sure
that i can run one task every N seconds and i dont need to monitor it
manually.

Thanks
Ravi.


On Sat, Apr 30, 2011 at 9:01 PM, Robert Kluin wrote:

> Hi Ravi,
>  You should name your tasks.  If one fails and retries for some
> reason, or runs multiple times, then the tasks will start to fan-out
> (think fork-bomb).
>
> http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api/taskqueue/TaskOptions.html#taskName(java.lang.String)
>
>
>  To control the delay you can set a countdown or eta.  Note that the
> time you specify is like a request, I've not seen tasks run before
> than time, but tasks will sometimes run later than requested.
>
> http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api/taskqueue/TaskOptions.html#countdownMillis(long)
>
>
> Robert
>
>
>
>
>
> On Sat, Apr 30, 2011 at 02:13, Ravi Sharma  wrote:
> > Hi ,
> > I have a chained task which runs every 1 second(Still wondering how can i
> > run it every 3 seconds)
> > Task code is something like this
> > try{
> > //Do something
> >//read data store and update datastore
> > }catch(Throwable ex){
> > logger.log(Level.Sever,ex.getMessage(),ex);
> > }finally{
> > //add next task to queue to be run next
> > Queue queue = QueueFactory.getQueue("queuename");
> >
> >
> queue.add(TaskOptions.Builder.withUrl("/tasks/mytask/"+auctionId).method(Method.GET));
> > }
> >
> > Now i expect that one task will be finished and next will be added to the
> > queue. 2 days back i found 4-5 tasks in the queue i dont know how they
> got
> > there
> >
> > and today i found tasks queue empty. How it can be possible that queue
> can
> > get more then 1 task at a time or it can get empty.I still have quota
> left
> > for the day.
> >
> > My App Name is edurectory
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group at
> > http://groups.google.com/group/google-appengine?hl=en.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Task Queue Strange Errors/Defects

[Robert Kluin]

Hi Ravi,
  You should name your tasks.  If one fails and retries for some
reason, or runs multiple times, then the tasks will start to fan-out
(think fork-bomb).

http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api/taskqueue/TaskOptions.html#taskName(java.lang.String)


  To control the delay you can set a countdown or eta.  Note that the
time you specify is like a request, I've not seen tasks run before
than time, but tasks will sometimes run later than requested.

http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api/taskqueue/TaskOptions.html#countdownMillis(long)


Robert





On Sat, Apr 30, 2011 at 02:13, Ravi Sharma  wrote:
> Hi ,
> I have a chained task which runs every 1 second(Still wondering how can i
> run it every 3 seconds)
> Task code is something like this
> try{
>             //Do something
>    //read data store and update datastore
>         }catch(Throwable ex){
>             logger.log(Level.Sever,ex.getMessage(),ex);
>         }finally{
>             //add next task to queue to be run next
>             Queue queue = QueueFactory.getQueue("queuename");
>
> queue.add(TaskOptions.Builder.withUrl("/tasks/mytask/"+auctionId).method(Method.GET));
>         }
>
> Now i expect that one task will be finished and next will be added to the
> queue. 2 days back i found 4-5 tasks in the queue i dont know how they got
> there
>
> and today i found tasks queue empty. How it can be possible that queue can
> get more then 1 task at a time or it can get empty.I still have quota left
> for the day.
>
> My App Name is edurectory
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Files API continuous random errors,

[Kaan Soral]

How can we prevent it?
Do you think retrying until null is a good approach?

On Apr 30, 5:56 pm, Brandon Donnelson  wrote:
> Mee too! I'm wondering how long it will take. I switched to uploading via
> posting on the server side.
>
> Brandon Donnelsonhttp://gwt-examples.googlecode.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Prospective search query size limit

[jbmusso]

> The maximum query length is 4kb.  This implies the maximum number of
> conditions depends on the structure of the query.  At most you could
> probably have around 200 terms.

Thank you for the answer, I see how this can limit my design and I
will code accordingly. Considering the fact that my users may
subscribe for alerts up to potentially thousands of words, I'll have
to add more subscriptions.

> We are still working to figure out the pricing.  The cost will likely be
> proportional to the number of matches x registered subscriptions BUT as I
> said we are still working through this is still very tentative.  It sounds
> like the first is likely a better option simply because of the limitation in
> terms of the number of terms.  As we have more information on how the
> pricing will work, we'll announce it.

I'm not sure to fully understand the "number of matches x registered
subscriptions". Say, for example, that I have a million registered
subscriptions, but only 0,1% of them concentrate most of the match
calls. Wouldn't the formula you posted above make these almost-never-
used 99,9% remaining subscriptions make my prospective-search billing
quotas skyrocket ?

In other words, "idle/never matched" subscriptions would greatly
increase the cost of those of the most active subscriptions that are
matched on a regular basis. I picked exaggerated numbers on purpose,
though some applications may be designed that way (including mine,
hehe...).

I never realized before that pricing formulas could influence code/
application design that much.

Regards,

-jbmusso

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Aw: Re: [google-appengine] Index Building Stuck

[Manuel Hennig]

Thanks. I will do that

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Re: GAE Hauling the Freight

[Brandon Wirtz]

Two of our deployments cruise along at 60 request per second 12-14 hours a
day every day.  And site at about 30 instances all of that time. Thus far
running on HR they have had Zero down time.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Jay
Sent: Saturday, April 30, 2011 8:15 AM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Re: GAE Hauling the Freight

 

I had a similar experience this month. Due to advertising related to a movie
release. Requests/sec. spiked up to 50 per second and held there for a few
hours. Absolutely no problem on app engine. We didn't have to touch
anything. (of course we had thought about it before and maintained a very
light weight approach). But operationally, we could just watch it go. GAE
spun up 29 instances to handle the traffic at that point if memory serves.  

-- 
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Index Building Stuck

[Robert Kluin]

Try filling out a production issue:
  
http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue







On Sat, Apr 30, 2011 at 13:13, Manuel Hennig
 wrote:
> Hi
> I added a few (170) indexes a week ago and they are all still in the
> building state with no change. The entity has only 90 entries. I tried to
> vacuum the indexes, but that didn't work (No error message though, it just
> says "0 indexes deleted").
>
> Is there another way to get rid of them without deleting the whole GAE and
> creating a new one?
>
> app id is: planetofslettal
>
> Thanks for any help
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Are we doing it right with the "label" implementaion?

[Robert Kluin]

Hey Nischal,
   You could insert some records with known key names in various
orders and a common value for a property to query on.  If the results
come back out key-ordered, that would be a good indication that the
method might work.  Also, I *think* there are some videos, groups
posts, and references in the docs (user or code) mentioning entities
in key order.



Robert




On Sat, Apr 23, 2011 at 23:59, Nischal Shetty  wrote:
> @Robert Thank you so much! Any idea how I can confirm on the timestamp + key
> thingy? I always need it based on timestamp (descending). If that works then
> it would help me a lot!
> I have looked into exploding indexes. My query would always be equality
> checks over list properties exclusively. I'm hoping that would not be a
> problem?
> Example:
> select from post
> where listproperty := label1 && listproperty := label2
> sortby time desc
> This is what we currently have. If the timestamp thingy is true then we
> would be able to remove the last line and that would be so damn cool :)
> On 24 April 2011 08:24, Robert Kluin  wrote:
>>
>> You probably need to be aware of exploding indexes if you're indexing
>> a list property.  It is just something you should consider and be
>> aware of.
>>
>>  http://code.google.com/appengine/docs/python/datastore/queries.html#Big_Entities_and_Exploding_Indexes
>>
>> If you're always planning to sort by timestamp, you could consider
>> prepending the timestamp on the Post entity keys.  Your entities
>> should then be returned to the app sorted by timestamp without
>> explicitly specifying the sort order.  At least that is the case if I
>> recall correctly.
>>
>>
>> Robert
>>
>>
>>
>>
>> On Sat, Apr 23, 2011 at 13:14, nischalshetty 
>> wrote:
>> > We're building a new web product.
>> > 1. We have the concept of label in them.
>> > 2. Each post gets one or more label.
>> > 3. The posts have to be displayed sorted according to timestamp in
>> > descending order
>> > We're making use of List (ListProperty) to save these labels for
>> > each post. It's working well for us. We can now query over multiple
>> > labels
>> > and retrieve posts.
>> > However, we need the posts to be sorted on timestamp in descending
>> > order.
>> > Due to this our query would have the timestamp as the sort order along
>> > with
>> > the label names. Due to this, we need indexes. And the indexes are being
>> > built for combination of the labels and timestamp.
>> > So, if our query has 2 labels and the sort order then the index would be
>> > built for these. If we query for 3 labels and sort order we would have a
>> > new
>> > index and so on.
>> > I wanted to know if there is any other way such that we can avoid the
>> > indexes? Are indexes cool if you anticipate a LOT of data everyday? Is
>> > there
>> > a way to have the posts sorted based on timestamp in a different way?
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>> >
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] ConcurrentModificationException

[Robert Kluin]

Hey Jeff,
  Here's a summary of what is happening: 1) open a transaction and
insert two transactional tasks, 2) each of the newly inserted tasks
inserts two transactional tasks.  When the two new tasks run, the
transactions happen within the 'same entity group' (which was never
set because there are *no* datastore operations) causing the
concurrent modification exception when inserting the next set of
tasks.

  So one easy solution to this is to do a datastore operation (like a
fetch by key) to set a unique entity group.  The question is, is there
a way in the Java SDK or Objectify to set the entity group without the
need to actually perform a datastore operation.  I know how to do this
in Python, but I'm not familiar enough with the Java SDK or Objectify
to know if it is possible.


Robert






On Mon, Apr 25, 2011 at 14:17, Jeff Schnitzer  wrote:
> I really don't understand what's being asked.
>
> You don't manually set an entity group; the rule is the same for
> low-level datastore transactions - for any txn, you can only access
> one entity group.  You get exceptions when you violate this rule.
>
> An Objectify (class) instance is created with a transactional context.
>  Want a new transaction?  Create a new instance.  You can't recycle
> them.  If there's something confusing about the documentation, let me
> know.
>
> Jeff
>
> On Sat, Apr 23, 2011 at 7:58 AM, Robert Kluin  wrote:
>> Hey Matija,
>>  I'm not familiar with Objectify's source, but perhaps Jeff could
>> suggest a simple / more efficient way to set the connection's entity
>> group for Java apps.
>>
>>
>>  For Python apps it is possible to get the transaction connection
>> object then 'manually' set the entity group; however, you need to use
>> 'internal' methods / attributes to do that (at least with the
>> interface I'm familiar with).
>>
>>
>>
>> Robert
>>
>>
>>
>>
>> On Sat, Apr 23, 2011 at 15:59, Matija  wrote:
>>> Hi Robert. Bravo. This approach solved problem.
>>> We are using objectify and for quick test reason I have used this to set
>>> this internal 'connection' entity group:
>>> try {
>>>      dao.ofy().getDatastore().get(KeyFactory.createKey("Random", new
>>> Random().nextLong()));
>>> } catch (EntityNotFoundException e) {
>>>       // Ignore
>>> }
>>> Deferred implementation that we use doesn't have access to request headers
>>> (but on Monday or Tuesday maybe it will have ;)). There is no reason to use
>>> costly random class when task name is unique.
>>> Now question is how to set their internal transaction entity group so that I
>>> don't need to use one get datastore operation with throwing exception?
>>>
>>> Tnx, MATijA.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine+unsubscr...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Index Building Stuck

[Manuel Hennig]

Hi
I added a few (170) indexes a week ago and they are all still in the 
building state with no change. The entity has only 90 entries. I tried to 
vacuum the indexes, but that didn't work (No error message though, it just 
says "0 indexes deleted").

Is there another way to get rid of them without deleting the whole GAE and 
creating a new one?

app id is: planetofslettal

Thanks for any help

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] HTTP DELETE requests returning 400 Bad Request from AppEngine Front Ends

[master outside]

I am getting 400 http status on deletes. As far as I can tell this
effects all apps. Now I am using custom domains. Is anyone not having
this problem. Also see issue 
http://code.google.com/p/googleappengine/issues/detail?id=4956

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] OpenID+OAuth for devices

[Robert Kluin]

Hi Brice,
  The section in the docs you'll be interested in is:
http://code.google.com/appengine/docs/java/oauth/overview.html

  However, if I'm not mistaken, the built-in OAuth stuff will only
work with Google accounts.  So if your app works with other OpenID
providers you might need to do something a little different.  Also,
last time I tested the built-in OAuth with Apps accounts it did not
work correctly (the authentication works, but the app did not
recognize the user).



Robert






On Mon, Apr 25, 2011 at 00:20, Brice  wrote:
> Good evening,
>
> I would appreciate some pointers on how to accomplish what I think the
> gentleman in this thread is doing:
> http://groups.google.com/group/google-appengine/browse_thread/thread/78dadccc8b8a644d/836d75c50a6be4ae?lnk=gst&q=openid+oauth#836d75c50a6be4ae
>
> Specifically, I'm trying to follow the suggestions from here:
> https://sites.google.com/site/oauthgoog/UXFedLogin/nobrowser/input-capable-devices
>
> I have a new AppEngine (Java) site using federated login. Everything
> appears to be working with that setup - I've implemented /_ah/
> login_required/ and am able to successfully login/logout when
> accessing "secured" URLs in my application.
>
> What I'm not getting (maybe because I'm not understanding something),
> is how to get OAuth working ... my understanding is that I need OAuth
> to provide my desktop application with a long-lived access token that
> it can use. I've read reams of documentation, it seems, but I'm not
> putting it together for some reason.
>
> I have registered my AppEngine application with Google ... I have a
> key/secret and the RSA cert/key from that process. I have no idea how/
> where to use this ... presumably this is part of the hybrid openid
> +oauth mechanism, but following the instructions here:
> http://code.google.com/apis/accounts/docs/OpenID.html#oauth -- I'm not
> seeing where/how.
>
> My goal is pretty straightforward ... users install the desktop app,
> which needs to access secured URLs in my app engine app - instead of
> providing username/password in the desktop app, users go to a URL in
> my app engine app ... they login (using any OpenID provider),
>  happens, then they get a
> key to enter into the desktop app ... the desktop app turns this
> around and gets the long-lived OAuth token for accessing secured URLs
> in my app engine app.
>
> The  bit in the middle is what I'm apparently
> missing ...
>
> Any guidance (code samples/flows) would be excellent and much
> appreciated.
>
> Thanks,
> Brice
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] How to change permissions to access an application

[Robert Kluin]

Yeah, you can switch back-and-forth between Google Accounts and
Federated Login, but if you want to restrict logins to a specific Apps
domain it has to be done at creation time.

I'm not sure how the changes to Apps accounts are going to impact
this.  I guess it will make it possible to log into 'Google Accounts'
apps using an Apps account?



Robert






On Sat, Apr 30, 2011 at 10:37, Brandon Donnelson
 wrote:
> You can also change it in the Dashboard and select Federated Login or Google
> Authentication.
> Your right, just hard code it into the server side stuff. Maybe the hard
> parts choosing? Or you can make your own user login system too.
> Brandon Donnelson
> http://gwt-examples.googlecode.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: GAE Hauling the Freight

[Jay]

I had a similar experience this month. Due to advertising related to a movie 
release. Requests/sec. spiked up to 50 per second and held there for a few 
hours. Absolutely no problem on app engine. We didn't have to touch 
anything. (of course we had thought about it before and maintained a very 
light weight approach). But operationally, we could just watch it go. GAE 
spun up 29 instances to handle the traffic at that point if memory serves. 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Task Queue Strange Errors/Defects

[Brandon Donnelson]

I'm wondering if the task was retrying due to exception thrown. One way I 
found the task will stick to the que for longer period of time. 

http://code.google.com/appengine/docs/java/config/queue.html#Configuring_Retry_Attempts_for_Failed_Tasks
 
- looks like you setup a que xml. I haven't tried this yet so I have no 
experience in it :).

Brandon Donnelson
http://gwt-examples.googlecode.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Task Queue Strange Errors/Defects

[Ravi Sharma]

I have not mentioned any retry count...so i guess whatever the default one
is being used. (Actually i never want to retry if one fails next one will
take care .. Do you know how to never retry a task.)


On Sat, Apr 30, 2011 at 3:48 PM, Brandon Donnelson
wrote:

> Did they have a high retry count?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Files API continuous random errors,

[Brandon Donnelson]

Mee too! I'm wondering how long it will take. I switched to uploading via 
posting on the server side.

Brandon Donnelson
http://gwt-examples.googlecode.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Appropriate way to save hundreds of thousands of ids per user

[Nischal Shetty]

It's a simple twitter app http://www.justunfollow.com

On 30 April 2011 20:13, Brandon Donnelson  wrote:

> Whats your app?
>
> Brandon
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>



-- 
-Nischal
twitter: NischalShetty 
facebook: Nischal 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Task Queue Strange Errors/Defects

[Brandon Donnelson]

Did they have a high retry count?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: GAE Hauling the Freight

[Thomas Wiradikusuma]

Man this is awesome, thanks for sharing with us!
It makes us who choose GAE feel good, after these "OMG cloud is
falling" stuff :)

On Apr 29, 3:44 pm, "Brandon Wirtz"  wrote:
> 900 Megabytes per second Burst for about 10 minutes, with 375 MBps for about
> 9 hours.  That's 7.2Gigabits burst.  To do that on a traditional hosting
> provider would be 8 Dedicated machines serving Gigabit each, and a 10 gig
> load balancer.  Would have been about $15k a month with a traditional
> hosting provider for this capacity.  Client will spend about $1500 this
> month instead, after our markup of the service.
>
> This is partly me bragging, but more it was a great demonstration of the
> elasticity of the product.  Look at the traffic before and after the spike.
> GAE really shines on sites where you only need 100x the capacity 10 minutes
> a week. And 50x for 6 hours a week.
>
>  image001.png
> 45KViewDownload

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Appropriate way to save hundreds of thousands of ids per user

[Brandon Donnelson]

Whats your app?

Brandon 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] How to change permissions to access an application

[Brandon Donnelson]

You can also change it in the Dashboard and select Federated Login or Google 
Authentication. 

Your right, just hard code it into the server side stuff. Maybe the hard 
parts choosing? Or you can make your own user login system too.

Brandon Donnelson
http://gwt-examples.googlecode.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: how to work around this eror: "DeadlineExceededError"

[Erwin Streur]

Sasha, you are absolutely right.

Based on the orignal post I assumed that the number of Authority
records belonging to a given twitterUser is 0 or 1, because the get
method is used which returns the first result only (according to doc)
and not the fetch method which can return multiple results.
In case the code sniplet is wrong and large amount of data are being
fetched(and this is a real requirement) it become a whole different
ballgame.

Linh Truong, can clarify what exactly you are trying to do and how
much data/entities you have in the datastore and how many you expect
to retrieve using the query?

It is important to realize that the datastore is not a regular
(relational) database. Large amount of in the datastore is not/should
not be a problem, because GAE doesn't allow you to preform a full
table scan, but only index based retrieval.
Returning large amount of data in a single operation is a problem and
should be tackled by mechanisms as paging or sharding(MapReduce)

On 29 apr, 07:05, Sasha  wrote:
> DeadlineExceeded usually just means that you tried to do too much work
> in one request, and you need to choose a more efficient design.
> Usually this also means the user is waiting many seconds for a
> response, which will frustrate them, so it's good to avoid it. You
> need to push work out of the request into tasks, split up the work
> across multiple requests, or simply do less work in each request.
> Almost certainly, you don't want a huge SELECT statement.
>
> How many Authority records should normally have a given twitterUser?
> Do you really need to retrieve all of them at once? If it isn't really
> so many you are fetching, then it may just be that you are querying
> inefficiently, and this is why you should profile (for a start, see
> appstats:http://code.google.com/appengine/docs/python/tools/appstats.html).
> If the bottleneck is in how you are querying, you can make it fast in
> some simple ways, like reducing the number of round trips to the
> datastore, getting things by key name or key id instead of querying,
> storing frequently-read information in memcache, using __key__ queries
> where possible... not to mention having your application do some kinds
> of work incrementally and in advance rather than all at once. If you
> do many fetches instead of just one at a time, it will be slower.
>
> But... if the real cause of the problem is that the query returns too
> many records, and it won't work to use key names or __key__ queries to
> fetch records,
> then the solution strongly depends on WHY you are trying to get so
> many records at once. Probably, you won't really want to get all those
> records at once.
>
> If you want to show the results to the user, it's customary to show
> only a few at each time, in 'pages.' You should consider this
> carefully.
> A pretty simple and efficient way is to use query cursors, for this
> seehttp://code.google.com/appengine/docs/python/datastore/queries.html#Q...
> If the user can't stand to click 'Next,' then you could always write
> some Javascript to request each chunk after the last one completed, or
> retrieve it as the user scrolls down, so the user sees an 'infinite
> page'
> and the server can still be happy because each request does not take
> so long. If you don't like query cursors, you can use other tricks
> like inequality filters (rank >= 10 and rank < 20 limits you to only
> data from ranks 10-19) but this adds to the complexity.
>
> If you are NOT displaying data to users on a web page - if you need to
> do a mass export of some data to another computer, or to carry out an
> operation on every user, or generate a big report based on lots of
> data, then you were on the right track to look at the bulk loader
> (http://code.google.com/appengine/docs/python/tools/
> uploadingdata.html) and mapreduce. These tools really just split up
> big jobs into multiple requests themselves in a way that only makes it
> look simple. Regardless of the approach you choose, you will have to
> somehow ensure that each request takes only a short time to carry
> out.
>
> If you try to get all matching keys, that will take you a little
> farther but you will still hit a limit of how many you can grab. It's
> better to find a way to only ask the datastore for fewer records at
> each request. If something like mapreduce or pipeline or the bulk
> loader will work well for you, then by all means use those. It
> strongly depends on whether and why you really need to fetch a large
> number of records at once.
>
> If you really need a LOT of work to be done by the task queue, and
> cannot avoid it, you can always tell the user that it will take a
> while. Then when the job is done, you can refresh the page (in
> javascript) or send a notification. That is not unreasonable at all
> for tasks which are very computationally expensive.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send 

[google-appengine] Re: GAE Hauling the Freight

[Brandon Donnelson]

Brag all you want here. Thanks for sharing! Am I missing something in the 
blank rectangle in the middle?

Brandon Donnelson
http://gwt-examples.googlecode.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: how to work around this eror: "DeadlineExceededError"

[Brandon Donnelson]

You won't have to store the list, just get the count of the keys to start 
with.

public long queryTotal() {
long total = 0;
PersistenceManager pm = sp.getPersistenceManager();
try {
  Query q = pm.newQuery("select stuffIdKey from " + 
ThingStuffJdo.class.getName());
  List ids = (List) q.execute();
  total = ids.size();
  q.closeAll();
} catch (Exception e) { 
  e.printStackTrace();
  log.log(Level.SEVERE, "", e);
} finally {
  pm.close();
}
return total;
  }

Once you get the total count, they you can page through your regular query 
using a range or 

like in http://code.google.com/appengine/docs/java/datastore/queries.html 
fetching results section.

Brandon Donnelson
http://gwt-examples.googlecode.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Add Your Profile Picture Challenge (for this #GAE community).

[Brandon Donnelson]

Wahoo, new groups native UI. Its fun to see more faces. Don't be shy, add 
your profile picture to spice this up :).

https://groups.google.com/forum/#!topic/google-appengine/EodCvLQmpl0 - this 
post in the native ui

I think the java community compared to the python community has more profile 
pictures, way to go! :)

Brandon Donnelson
http://gwt-examples.googlecode.com

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Set task queue rate at 1 every 45 seconds.

[Prashant]

sorry, my bad :( was in lil bad mood while thinking abt it ...

try 1.33/min with bucket size one ...

On 30 April 2011 19:08, Prashant  wrote:

> shuldn't it be 0.022/min instead of /sec ?
>
>
> On 30 April 2011 16:51, Daniel Woelfel  wrote:
>
>> > Try setting rate to 0.022 per second. (1/45)
>>
>> I tested it, but it seems to round up. Tasks were executing at 1/
>> second.
>>
>> One odd thing, though. I accidentally entered "sec" instead of "s" and
>> I got this regular expression in the error logs:
>>
>>   Value '0.022/sec' for rate does not match expression '^^(0|[0-9]+(\.
>> [0-9]*)?/[smhd])$'
>>
>> Why would they allow fractions in the queue.yaml file if they just
>> round up?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Set task queue rate at 1 every 45 seconds.

[Prashant]

shuldn't it be 0.022/min instead of /sec ?

On 30 April 2011 16:51, Daniel Woelfel  wrote:

> > Try setting rate to 0.022 per second. (1/45)
>
> I tested it, but it seems to round up. Tasks were executing at 1/
> second.
>
> One odd thing, though. I accidentally entered "sec" instead of "s" and
> I got this regular expression in the error logs:
>
>   Value '0.022/sec' for rate does not match expression '^^(0|[0-9]+(\.
> [0-9]*)?/[smhd])$'
>
> Why would they allow fractions in the queue.yaml file if they just
> round up?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Seemingly random TemplateNotFound errors only on specific instances

[kamens]

We've started seeing bursts of TemplateNotFound errors when invoking
fairly standard template rendering code:
__

from django.template.loader import render_to_string

path = os.path.join(os.path.dirname(__file__),
'video_comments_content.html')
html = render_to_string(path, {})
__

and the error that is seen somewhat randomly:
__

/base/data/home/apps/khanexercises/de514da0ee02.350078430120358844/
discussion/video_comments_content.html
Traceback (most recent call last):
  File "/base/python_runtime/python_lib/versions/1/google/appengine/
ext/webapp/__init__.py", line 634, in __call__
handler.get(*groups)
  File "/base/data/home/apps/khanexercises/
de514da0ee02.350078430120358844/discussion/comments.py", line 39, in
get
html = render_to_string(path, template_values)
  File "/base/python_runtime/python_lib/versions/third_party/
django-0.96/django/template/loader.py", line 101, in render_to_string
t = get_template(template_name)
  File "/base/python_runtime/python_lib/versions/third_party/
django-0.96/django/template/loader.py", line 79, in get_template
source, origin = find_template_source(template_name)
  File "/base/python_runtime/python_lib/versions/third_party/
django-0.96/django/template/loader.py", line 72, in
find_template_source
raise TemplateDoesNotExist, name
TemplateDoesNotExist: /base/data/home/apps/khanexercises/
de514da0ee02.350078430120358844/discussion/video_comments_content.html
__

For some reason these errors only show up on specific instances. We've
tried redeploying, but after a few hours I come back to check the
console and see tons of these errors. When I look at the "Instances"
panel, I can see that all instances are fine (0 or 1 errors) and one
or two are disastrous (100+ errors). It's extremely hard to reproduce
the crash because sometimes we have 100 instances running, so
reloading the page over and over almost always works (I was only able
to see the crash once, presumably when my request was routed to a
problematic instance, but I did verify that it's happening).

It's almost as if some of the instances are brought up either without
that specific template file or with a different template loading
environment. Note that the path in that error report ".../discussion/
video_comments_content.html" looks correct to me.

Thanks in advance. appid: khanexercises

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Set task queue rate at 1 every 45 seconds.

[Daniel Woelfel]

> Try setting rate to 0.022 per second. (1/45)

I tested it, but it seems to round up. Tasks were executing at 1/
second.

One odd thing, though. I accidentally entered "sec" instead of "s" and
I got this regular expression in the error logs:

   Value '0.022/sec' for rate does not match expression '^^(0|[0-9]+(\.
[0-9]*)?/[smhd])$'

Why would they allow fractions in the queue.yaml file if they just
round up?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Sharing a Task queue between two different applications

[Stephen]

On Fri, Apr 29, 2011 at 11:36 AM, footy  wrote:
>
> It is critical for me to present the front end (App1: Producer) to the
> users under heavy load. Thus the producer needs to be scaled
> differently. for eg: I might need 25 instances.


You can't control the number of instances directly, but you don't need to:

http://code.google.com/appengine/docs/python/blobstore/overview.html#Using_the_Blobstore

"...the user's browser uploads the specified files directly to the
Blobstore. The Blobstore rewrites the user's request and stores the
uploaded file data, replacing the uploaded file data with one or more
corresponding blob keys, then passes the rewritten request to the
handler at the URL path you provided to create_upload_url()."

ie. the App Engine infrastructure handles blob uploads asynchronously
and only calls your app with the blob key once all XMB has completely
uploaded.

The only thing your handler does is enqueue a task with the blob key.
As this takes only a trivial amount of time App Engine will easily
scale to the required number of instances.


> As for the worker back end (App2: Consumer) it is not that critical,
> and I am okay with a bit of delay as the consumer churns through the
> work load of various tasks queued in the task queue.
> Thus the consumer needs to be scaled differently. for eg: I might only
> need 5 instances to run through the tasks.


Here you have some control, although again not directly the number of instances:

http://code.google.com/appengine/docs/python/config/queue.html#Defining_Queues_and_Processing_Rates

queue:
- name: process-photos
  rate: 5/m


Whether you actually achieve a rate of 5 tasks per minute, or whatever
you set this to, depends on how long your task handler takes to
process a photo as this is one of the factors Google uses to decide
how many instances to run concurrently. If you can't process photos
fast enough your queue will back up.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.