[google-appengine] Re: App Engine VM-based Backends - Trusted Tester Sign-up

[Robert Kluin]

This is a really exciting feature.

Will the durability of these backends look and feel more like GCE 
(long-term / stable) or will the be more like GAE's current backends (flaky 
and unstable)?


Robert



On Thursday, June 20, 2013 2:21:06 PM UTC-5, Takashi Matsuo (Google) wrote:
>
>
> Fellow App Engine Gurus,
>
> We're happy to announce the next generation of App Engine Managed 
> Backends. These Backends utilize the App Engine VM Runtime, allowing 
> developers to run Backends on Compute Engine VMs. By building on top of 
> Compute Engine VMs, developers can:
>
>
>- take advantage of higher CPU and memory
>- rely on longer-lived processes
>- utilize a local filesystem
>- communicate via native network stacks
>- execute external processes
>- access the entire JRE
>- upload arbitrary Python extensions
>
>
> Given that these are App Engine Backends, you can still use all the App 
> Engine APIs to access the existing managed services (Datastore, Task 
> Queues, Memcache, etc.)
>
> Updating existing Backends to run on Compute Engine VMs is a simple config 
> change:
>
> app.yaml
> 
> application: app-id
> version: v1
> runtime: python27
> *vm: true*
> *
> *
> *manual_scaling:*
> *  instances: 1*
>
>
> That’s all you need to get started. We’ll pick a deafult VM machine type 
> and spin everything up your behalf. Of course there other options that you 
> can set (including machine type) and these are documented in the Getting 
> Started 
> Guide
> .
>
> In order to build a great product, we need quality feedback from brave 
> early adopters. If you’re interested test driving, please sign-up at the 
> link below and we’ll take care of the rest.
>
> App Engine VM-based VM Runtime - Trusted Tester 
> Sign-up
>
> Also, if you have any questions, please feel free to send an email to:
> appengine-...@googelgroups.com 
>
> Thanks!
>
> -- 
> Takashi Matsuo | Developers Programs Engineer | tma...@google.com
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Sunsetting double wildcard SSL certificate support for applications serving from appspot.com

[Robert Kluin]

How will this impact people using secure only URLs that are testing a
new version of the app before making it default?  I guess the
"versions" page in the admin console will be updated to use the -dot-
notation, so it will still be easy?


Robert





On Thu, Jan 10, 2013 at 5:47 PM, Christina Ilvento  wrote:
> Hi All,
>
> As part of our continued commitment to security and a standard platform,
> Google App Engine will soon stop publishing double wildcard SSL certificates
> for *.*.appspot.com, (e.g., https://version.application.appspot.com or
> https://www.application.appspot.com). This change will take effect no later
> than April, 2013. Please note that this only affects applications serving
> from appspot.com, and that any applications serving from custom domains will
> continue using their existing certificates. Furthermore, this only affects
> HTTPS access to your application and non-secure HTTP traffic will not be
> affected.
>
> If you rely on HTTPS access to such URLs for your application, please change
> any application logic to use “-dot-” instead of “.”. For example, to access
> version “1” of application “myapp” use “https://1-dot-myapp.appspot.com”
> instead of “https://1.myapp.appspot.com”.
>
> Developers for applications using this pattern as of 1/7/2013 have already
> received a notification email with instructions for identifying this pattern
> in their application.
>
> Based on our analysis, fewer than 2,000 App Engine applications are
> currently using double wildcard SSL certificates on appspot.com and the
> majority of them are using the pattern of https://www.appid.appspot.com,
> which can safely be replaced with https://appid.appspot.com.
>
> Thank you for your continued support of App Engine. If you have any
> questions or concerns about these changes, please feel free to email us at
> appengine-ssl-certificate-wildcard-questi...@google.com with your
> application-id and we’ll be happy to assist you.
>
>
> Regards,
> Christina Ilvento on behalf of the Google App Engine Team
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Taskqueue stops running tasks several times a day

[Robert Kluin]

Hey Richard,
  Did you ever get an explanation / resolution for this?  I have been
seeing this same behavior across a lot of apps that I work with.  We
will see one or more queues just stop executing tasks for, typically,
several minutes, but I have seen this last as long as 20 minutes.


Robert





On Fri, May 18, 2012 at 10:19 PM, Richard Steventon  wrote:
> Nick,
>
> I shared a private google docs spreadsheet with you detailing average
> number of players per game for this year.
>
> Start at week 21.  Column A = (server) time.  Anytime the number of
> players in a game suddenly goes to zero... is because the taskqueue
> stopped running.
>
> When it restarts, one of two things happen:
> 1) It clears the queue (NO spike in user numbers ie: all
> unprocessed results from previous games get deleted/lost)
> 2) It runs the queue (spike in the user count ie: all unprocessed
> results from previous games get processed)
>
> -Bye
> -Richard
>
>
> On Fri, May 18, 2012 at 6:32 PM, Nicholas Verne  wrote:
>> Please give us your app id so we can investigate. Also, it would help
>> us if you could give some specific dates/times of outages.
>>
>> Thanks,
>>
>> Nick Verne
>>
>> On Sat, May 19, 2012 at 3:50 AM, Mind  wrote:
>>> I have an Android game whose server side is Python on AppEngine.
>>>
>>> The app processes the client requests.  These requests result in a need to
>>> update the client profile (player stats).  I offload this to a task.
>>>
>>> Tasks run fine some days.  On other days, the task queue randomly stops
>>> running and refuses to process any tasks.  Synchronous cron jobs also start
>>> late or not at all during this time.
>>>
>>> Outages can last several minutes to 10's of minutes.  They are not dependent
>>> of the volume of client requests (I graph everything) or time of day.  Some
>>> days it happens with 20 clients connected ... and other days it is just fine
>>> with over 200 clients connected at the same time.
>>>
>>> The statistics I don't care too much about, but I ALSO use tasks and CRON to
>>> control game flow, cycle to the next game board/etc.  So broken task queue
>>> means all my Android user's game clients stop working.
>>>
>>> Has anyone else observed anything similar or got any suggestions ?
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msg/google-appengine/-/2nvta8dI8gwJ.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine+unsubscr...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to 
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at 
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Unexplained traffic loss

[Robert Kluin]

Did anyone file an issue on this?  I have also seen this behavior
before and would like to hear about the cause / resolution.



Robert




On Mon, May 21, 2012 at 10:59 AM, Robert Schuppenies  wrote:
> Have you filed a production issue?:
> http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue
>
>
> On Mon May 21 04:03:31 GMT-700 2012, Dylan
>  wrote:
>>
>> We are having a similar issue.
>> Requests timing out or taking more than 10 seconds to return a 200kb
>> json document from the cache.
>> Our instance graph also looks the same for the past 24hrs, sharp
>> spikes in instances.
>>
>> Anyone know how to get in touch with Google?
>>
>> On May 21, 8:43 pm, Naresh Talluri  wrote:
>> > Even i can see this behavior which i never seen before.
>> > my app-id : beta-sb.
>> >
>> > Thanks,
>> > Naresh T
>> >
>> > On Fri, May 18, 2012 at 10:49 PM, Nathan Skone
>> > wrote:
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > > Another note about this issue is that the admin graphs listed zero
>> > > instances during these times, even though I had the minimum idle
>> > > slider set
>> > > to 4. This means that the Google App Engine incorrectly shut down
>> > > instances
>> > > that I specified should be running.
>> >
>> > > Thanks,
>> > > Nathan Skone
>> >
>> > > On Thursday, May 17, 2012 2:51:59 PM UTC-7, Nathan Skone wrote:
>> >
>> > >> We have over 300 machines distributed across the US calling an API
>> > >> once a
>> > >> minute, which means we should always have a base level of traffic of
>> > >> at
>> > >> least 5 or 6 requests per second.  However, today we are seeing pits
>> > >> in our
>> > >> requests per second graph that show the traffic dropping to zero,
>> > >> then
>> > >> spiking directly afterward.  We have not deployed any new code
>> > >> recently, or
>> > >> otherwise touched our application in any way.
>> >
>> > >> This traffic pattern has doubled our costs compared to a normal day.
>> > >> A
>> > >> screenshot is attached.
>> >
>> > >> Thanks,
>> > >> Nathan Skone
>> >
>> > >  --
>> > > You received this message because you are subscribed to the Google
>> > > Groups
>> > > "Google App Engine" group.
>> > > To view this discussion on the web visit
>> > >https://groups.google.com/d/msg/google-appengine/-/1jKeQMTTMmUJ.
>> >
>> > > To post to this group, send email to
>> > > google-appengine@googlegroups.com.
>> > > To unsubscribe from this group, send email to
>> > > google-appengine+unsubscr...@googlegroups.com.
>> > > For more options, visit this group at
>> > >http://groups.google.com/group/google-appengine?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Scatters with NDB

[Robert Kluin]

Have you done any testing with this on an appspot instance?

It has been a while since I last looked at the SDKs handling of this,
but it was a little different last I knew.



Robert



On Sat, May 19, 2012 at 12:14 AM, Kaan Soral  wrote:
> I am trying to rewrite my custom mapreduce with NDB
> For the above code:
>    gg=AModel.query()
>    #gg=gg.order(ndb.GenericProperty('__scatter__'))
>    logging.info(len(gg.fetch(5000,keys_only=True)))
> The result is ~200
>
> If you remove the comment and fetch only properties with __scatter__'s the
> result is ~100 (~50%)
>
> I have wiped the DB, created it again several times and the results were
> consistent.
>
> Normally the __scatter__ probability should be 0.2%
>
> What could be the reason? [The experiment is on SDK]
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/Wh7r-9vacQ8J.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Outages?

[Robert Kluin]

Do you mean Python or Java apps sitting at over 100mb after spin up?

I work on some extremely large Python apps, they can sit around ~75mb.
 If your Python app is heavier than that at startup... you are
probably doing stuff in a very questionable way.


Robert



On Fri, Apr 13, 2012 at 11:02, Brandon Wirtz  wrote:
>> My guess is that the number is close to 60s.  If GAE gets marginally
>> slower, it pushes you over the edge.
>
>
>
> I’ll be doing a video very shortly about how to keep your Instances from
> dying on startup. But one easy way to tell if this is the issue is to up
> your instance size AND test that you are having the same issue on a Dynamic
> Backend.  If your issues magically go away, but your startups take 65
> seconds, you know that you need to optimize your startup.
>
>
>
> One should also watch the Instance memory usage in the dash board. Often
> people have instances that just spun up and are sitting at 119 of 128m of
> memory usage and then wonder why their instances recycle so often.
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] 500 Server Error

[Robert Kluin]

Hey Gowtham,
  You need to take a look at your apps logs and see what the exception
is being thrown.


Robert





On Thu, Apr 19, 2012 at 05:24, Gowtham Gutha  wrote:
> I tried a sample java servlet application and deployed it to Google App
> Engine through Eclipse. My sample project worked well on my localhost but on
> appengine it did not.
>
> http://guthagowtham.appspot.com/ is my project url. This is simple test it
> takes input from the user and prints it on the other page.
>
> When i typed something and clicked on submit, it is unable to display the
> output. Here is what i got
> http://guthagowtham.appspot.com/simple?name=sample&submit=submit
>
> The following message is displayed
>
> Error: Server Error
>
> The server encountered an error and could not complete your request.
>
> If the problem persists, please report your problem and mention this error
> message and the query that caused it.
>
> Is there any solution for this.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/YDMQyt2P0goJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: How to filter images on GAE?

[Robert Kluin]

I suggest you post your actual app.yaml and the full stack trace from
the appspot logs.



Robert



On Wed, Apr 18, 2012 at 14:56, Computer_Engineer  wrote:
> Brian Quinlan: i tried to upload it into my application on GAE, the error
> still???what can i do?
>
>
> On Tuesday, April 17, 2012 5:11:44 PM UTC+3, Computer_Engineer wrote:
>>
>> I'm working on python google app engine application,i'm using python2.5, i
>> want to deal with images on GAE(filter images), initially i tried wit PIL
>> library and i installed it successfully and i tested it and it works
>> correctly on my computer,but when i test it on localhost(run on GAE), i have
>> an error thatNotImplementedError: Unable to find the Python PIL library,then
>> i tried to deal with images using Images Python API , as in this
>> tutorial:https://developers.google.com/appengine/docs/python/images/overview,
>> and i tested resizing images service,and it works correctly,but the main
>> application that i want is to filter the images,the question is :if this
>> service(image filtering) supported in Images Python API in GAE?and if not
>> how can i filter images on GAE?i saw that this service available in PIL,but
>> this library is not supported by GAE
>> I tried as a second choice to use python2.7 such that it supports PIL
>> library and i edited the app.yaml configuration file,but i don't know how to
>> use this library in the python page(main.py)?
>> app.yaml:
>> application: app_id
>> version: 1
>> runtime: python27
>> api_version: 1
>> threadsafe: false
>> handlers:
>> - url: /.*
>>   script: main.py
>> libraries:
>> - name: PIL
>>   version: "1.1.7"
>>
>> ...any suggestions are welcome...Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/Gy6_kxeqsY0J.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Datastore Small Operations - datastore.get(key)

[Robert Kluin]

Hi,
  Memcache is a cache, so you should not depend on it being there.  If
you need something persisted, you will need to write it to the
datastore.  You can stick it in memcache at write time, which will
help your reads though.


Robert






On Wed, Apr 18, 2012 at 02:36, cloudpre  wrote:
> Jeff - thanks for the note. I have been trying to add few items in
> memcache manually.
>
> Can put operations also be saved? Let's say I am updating the same
> query again after few seconds.
>
> Does it work flawlessly in the production? The last thing I want to
> see is our thousands of customers coming back and complaining.
>
> Cheers.
>
> On Apr 18, 12:17 am, Jeff Schnitzer  wrote:
>> Get-by-key is a read operation, not a small operation.  Small
>> operations are things like count() index walks and keys-only queries
>> (although there is still 1 read operation per query).
>>
>> Most read-heavy apps can benefit significantly from memcache.  If
>> you're using the low-level api, try this:
>>
>> http://code.google.com/p/objectify-appengine/wiki/MemcacheStandalone
>>
>> Jeff
>>
>>
>>
>> On Wed, Apr 18, 2012 at 2:27 AM, cloudpre  wrote:
>> > Hi
>>
>> > We have a pretty big app which has not been optimized till date. We
>> > spend over $40/day just on the reads.
>>
>> > This is our current stats
>>
>> > Datastore Read Operations                46.74 Million Ops       46.69   
>> > $0.70/ Million
>> > Ops      $32.69
>> > Datastore Small Operations               0.00 Million Ops        0.00    
>> > $0.10/ Million
>> > Ops      $0.00
>>
>> > I have been reading that small operations are much cheaper and they
>> > are done based on key.
>>
>> > The following function is used at many places - but I do not see
>> > counter moving from zero for small operations.
>>
>> > Is datastore.get(key) a small operation? If not, what should I change
>> > the function to make it count towards small operations.
>>
>> > Thanks.
>>
>> > // Get Entity
>> >        public static Entity getEntityFromId(String entityName, String id)
>> >        {
>> >                // Get Datastore
>> >                DatastoreService datastore =
>> > DatastoreServiceFactory.getDatastoreService();
>>
>> >                // Get Entity
>> >                try
>> >                {
>> >                        Key key = KeyFactory.stringToKey(id);
>> >                        return datastore.get(key);
>> >                } catch (Exception e)
>> >                {
>> >                        return null;
>> >                }
>> >        }
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups 
>> > "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to 
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group 
>> > athttp://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: How to filter images on GAE?

[Robert Kluin]

When you deploy to appspot, does your code work?

You need to use Python 2.7 locally, and you need to ensure PIL is
installed for Python 2.7.


Robert






On Wed, Apr 18, 2012 at 06:45, Computer_Engineer  wrote:
> Brian Quinlan:Thanks for your response,i tried as :from PIL import Image ,
> but the problem still as :No module named PIL
>   args = ('No module named PIL',)
>   message = 'No module named PIL'
>
>
> On Tuesday, April 17, 2012 5:11:44 PM UTC+3, Computer_Engineer wrote:
>>
>> I'm working on python google app engine application,i'm using python2.5, i
>> want to deal with images on GAE(filter images), initially i tried wit PIL
>> library and i installed it successfully and i tested it and it works
>> correctly on my computer,but when i test it on localhost(run on GAE), i have
>> an error thatNotImplementedError: Unable to find the Python PIL library,then
>> i tried to deal with images using Images Python API , as in this
>> tutorial:https://developers.google.com/appengine/docs/python/images/overview,
>> and i tested resizing images service,and it works correctly,but the main
>> application that i want is to filter the images,the question is :if this
>> service(image filtering) supported in Images Python API in GAE?and if not
>> how can i filter images on GAE?i saw that this service available in PIL,but
>> this library is not supported by GAE
>> I tried as a second choice to use python2.7 such that it supports PIL
>> library and i edited the app.yaml configuration file,but i don't know how to
>> use this library in the python page(main.py)?
>> app.yaml:
>> application: app_id
>> version: 1
>> runtime: python27
>> api_version: 1
>> threadsafe: false
>> handlers:
>> - url: /.*
>>   script: main.py
>> libraries:
>> - name: PIL
>>   version: "1.1.7"
>>
>> ...any suggestions are welcome...Thanks
>
>
> On Tuesday, April 17, 2012 5:11:44 PM UTC+3, Computer_Engineer wrote:
>>
>> I'm working on python google app engine application,i'm using python2.5, i
>> want to deal with images on GAE(filter images), initially i tried wit PIL
>> library and i installed it successfully and i tested it and it works
>> correctly on my computer,but when i test it on localhost(run on GAE), i have
>> an error thatNotImplementedError: Unable to find the Python PIL library,then
>> i tried to deal with images using Images Python API , as in this
>> tutorial:https://developers.google.com/appengine/docs/python/images/overview,
>> and i tested resizing images service,and it works correctly,but the main
>> application that i want is to filter the images,the question is :if this
>> service(image filtering) supported in Images Python API in GAE?and if not
>> how can i filter images on GAE?i saw that this service available in PIL,but
>> this library is not supported by GAE
>> I tried as a second choice to use python2.7 such that it supports PIL
>> library and i edited the app.yaml configuration file,but i don't know how to
>> use this library in the python page(main.py)?
>> app.yaml:
>> application: app_id
>> version: 1
>> runtime: python27
>> api_version: 1
>> threadsafe: false
>> handlers:
>> - url: /.*
>>   script: main.py
>> libraries:
>> - name: PIL
>>   version: "1.1.7"
>>
>> ...any suggestions are welcome...Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/f2BduRaG_CsJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Master/Slave Datastore, thanks for all your hard work

[Robert Kluin]

I think this is a really good idea.  It will encourage people with
apps to move, and help further persuade new apps to not use
master-slave.  I have not spoken to anyone who regretted the move to
high-replication.


Robert




On Wed, Apr 4, 2012 at 15:42, Chris Ramsdale  wrote:
> Hey App Engine Users,
>
> Almost 4 years after launch, we’ve seen fantastic growth both in Google App
> Engine and the apps that run on it. And although the Master/Slave Datastore
> was a big part of our early success, it's time to announce the deprecation
> of the Master/Slave Datastore in favor of the High-Replication Datastore
> (HRD). HRD has provided us with higher availability and better, more
> predictable performance. Many upcoming features will be HRD-only and we
> strongly encourage you to migrate all your applications as soon as possible
> using the migration tools found in the Application Settings tab of the
> Administration Console. The deprecation period will follow the guidelines
> set in our terms of service. If you have any questions or concerns, please
> contact us at ms-datastore-deprecat...@google.com.
>
> Chris
>
> Product Manager, Google App Engine
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Fwd: 1.6.4 Prerelease Now Available

[Robert Kluin]

Awesome.  Tons of good features here.





On Mon, Mar 19, 2012 at 19:10, Marzia Niccolai
 wrote:
> Hi,
>
> The 1.6.4 prerelease SDKs are now available to download for Python and Java
> here: http://code.google.com/p/googleappengine/downloads/list
>
> Please see release notes below.
>
> -Marzia
>
> Python Version 1.6.4
> ===
> - Datastore statistics now show the amount of storage used by application
>  indexes.
> - Billed applications which have specified additional logs storage over 1 GB
>  are now being charged for that storage at $0.24/GB/month.
> - We have released an experimental utility for backing up your application's
>  blobs at the same time you migrate your datastore data. You can opt in to
>  blob migration in the Admin Console when you start your migration.
> - Using the Blobstore API's serve_blob method, your application can serve
>  objects hosted on Google Storage for Developers.
> - We have updated the experimental Backup/Restore functionality to include
>  the option to backup and restore to Google Storage for Developers.
> - The NDB datastore API is now generally available.
> - In the Python 2.7 runtime, Background threads are available as an
>  experimental release when using App Engine backends.
> - In an application's Admin Console, there is a Memcache viewer available
>  that lists Memcache stats and has the ability to view Memcache content
>  based on key.
> - In the Capabilities API stub in the SDK, you can now enable or disable
>  a capability using SetPackagedEnabled.
> - The Windows installer now prompts to install Python 2.7 instead of
>  Python 2.5.
> - The Testbed API now supports the Capabilities API.
> - GQL queries in the Admin Console no longer throw an error when a trailing
>  semi-colon is included.
> - The Datastore API now includes a NonTransactional decorator to ensure that
>  a function is run outside of a transaction. Existing transactions are
> paused
>  while the function is executing.
> - The Datastore Admin now shows entities from every namespace.
>    http://code.google.com/p/googleappengine/issues/detail?id=3962
> - Fixed an issue with _strptime when threadsafe was specified.
>    http://code.google.com/p/googleappengine/issues/detail?id=6489
> - Fixed an issue where DatastoreFileStub.__del__ fails on tempfile.msktemp.
>    http://code.google.com/p/googleappengine/issues/detail?id=6749
> - WebOb 1.1.1 is now included in the SDK, and used by default there when
>  Python 2.7 is specified.
>    http://code.google.com/p/googleappengine/issues/detail?id=7014
> - Fixed an issue where the index.yaml file was cleared if your skip_files
> entry
>  differs from the default skip_files list.
>    http://code.google.com/p/googleappengine/issues/detail?id=7031
>
> Java Version 1.6.4
> =
> - Datastore statistics now show the amount of storage used by application
>  indexes.
> - Billed applications which have specified additional logs storage over 1 GB
>  are now being charged for that storage at $0.24/GB/month.
> - We have released an experimental utility for backing up your application's
>  blobs at the same time you migrate your datastore data. You can opt in to
>  blob migration in the Admin Console when you start your migration.
> - Using the blobstoreService.serve method, your application can serve
> objects
>  hosted on Google Storage for Developers.
> - We have updated the experimental Backup/Restore functionality to include
>  the option to backup and restore to Google Storage for Developers.
> - Version 2.0 of the App Engine DataNucleus plugin is now available as an
>  experimental release. This plugin contains support for JDO 3 and JPA 2 and
>  over 40 bug fixes. For a full list see:
>
>  http://code.google.com/p/datanucleus-appengine/source/browse/branches/2_0_0/dist/RELEASE_NOTES.ORM
> - Threads are available as an experimental feature via
>  com.google.appengine.api.ThreadManager.
> - In an application's Admin Console, there is a Memcache viewer available
>  that lists Memcache stats and has the ability to view Memcache content
>  based on key.
> - The method ApiProxy.getRemainingMillis() returns the amount of
> milliseconds
>  remaining before a request hits the request deadline.
> - New datastore query related callbacks - PreGet(), PreQuery(), and
> PostLoad()
>  are now available.
> - Omitting the  directive from appengine-web.xml now results in
>  a warning. Starting with the 1.6.5 release, omitting this directive will
>  result in an error.
> - You can set an RPC deadline for the Task Queue API's leaseTasks method.
> - GQL queries in the Admin Console no longer throw an error when a trailing
>  semi-colon is included.
> - Fixed an issue where  did not work.
>    http://code.google.com/p/googleappengine/issues/detail?id=5774
> - The Datastore Admin now shows entities from every namespace.
>    http://code.google.com/p/googleappengine/issues/detail?id=3962
> - Fixed an issue in the Java Development Server where Task Qu

Re: [google-appengine] Re: 1:1 with a member of the App Engine community: Robert Kluin

[Robert Kluin]

Hey Guys, glad you found it "entertaining" (not sure the right word).
Just wanted to note that I'm at pycon this evening hacking around on a
couple things.  For anyone here, feel free to drop by!  I'll be taking
off tomorrow mid-morning.


Robert






On Tue, Mar 13, 2012 at 14:24, Kaan Soral  wrote:
> I'm a big fan of robertk, loved the interview
>
>
> On Tuesday, March 13, 2012 12:13:25 AM UTC+2, Johan Euphrosine (Google)
> wrote:
>>
>> FYI,
>>
>> I just posted an interview with our very Robert Kluin to Google+:
>> https://plus.google.com/111042085517496880918/posts/8CHm8KKqXWU
>>
>> Feel free to comment there.
>> --
>> Johan Euphrosine (proppy)
>> Developer Programs Engineer
>> Google Developer Relations
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/BjsPhqc6ZNsJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Tasks in queue counts wildly inflated (1000 => 700k)

[Robert Kluin]

Agreed.  Makes it sometimes makes it hard to troubleshoot.





On Thu, Mar 8, 2012 at 12:21, Jan Zawadzki / Hapara
 wrote:
> So the queue task counts seem to be pulling wrong queues, or
> miscalculating the number by a huge margin at times.
>
> We've observed queues with ~100 tasks displaying counts of 700,000+.
>
> When the incorrect count is displayed it decrements correctly as tasks
> execute.  Pausing and restarting the queue has no effect.
>
> It's not a show stopper but it's disconcerting :)  ('better bump that
> quota again... ugh...')
>
> Jan
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: App Engine Chat Time!

[Robert Kluin]

Hi,
  1) Start a new thread.  You responded to a completely unrelated post
from many months ago.
  2) Be more specific.  It is not clear if you want to diagram the
relationship between your data, or you want to diagram the schema.


Robert




On Mon, Mar 5, 2012 at 12:30, Edixon Polanco  wrote:
> Hi,
>
> How I can generate a diagram of my database from the data store google app
> engine
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/D2kLAPtPqFMJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] No matching index found

[Robert Kluin]

You're not showing us the entire query here.  Please show us the
actual query or at least the index it says you're missing.



Robert



On Mon, Feb 27, 2012 at 08:32, ramesh  wrote:
> Hi All,
>
> I am trying to fetch the data by applying the sort order(Ascending) on
> the property.
>
> Ex:
>
>  Query query = new Query("DBNAME");
>  query .addFilter("Property",FilterOperator.GREATER_THAN_OR_EQUAL,
> "Propertyvalue");
>  query.addSort("Property", SortDirection.ASCENDING);
>
>
>  When i am executing the above query, it saying no matching index
> found.
>
>  Please let me know what i am doing wrong.
>
>  Thanks
>  Ramesh
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Email when quota is reached?

[Robert Kluin]

Perhaps this could be tied into my request for a persistent log of quotas hit.
  http://code.google.com/p/googleappengine/issues/detail?id=6296


Would be nice to be able to "register" an developer to get an email
when quotas are getting hit.



Robert



On Mon, Feb 27, 2012 at 04:36, Felix E. Klee  wrote:
> On Mon, Feb 27, 2012 at 12:25 AM, sb  wrote:
>> Writing a small app shouldn't be too hard.
>
> How?
>
> Of course one could write an app that repeatedly checks if app engine
> replies with 200 OK for a certain page. But this wouldn't be sufficient.
> There may be quotas that are hit only later, e.g. during a certain XHR.
>
>> How long between checks do you think is good? Every 2 hours?
>
> As frequent as possible is my preference, perhaps up to every minute.
>
> In fact, it would be cool if there was an API to the billing system, in
> order to automatically enable billing when needed.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Unable to create entity on Google App Engine Datastore

[Robert Kluin]

Are you making requests against your local server, or the app running
on appspot?




On Sun, Feb 26, 2012 at 04:24, ani  wrote:
> Hello, I’m trying to connect an android application with Google app
> engine. I am using Google App Engine Datastore as my “database”. The
> problem is that I am having difficulties in building entities on the
> GAE datastore.
>
> I’m using JDO to create my entity and I am following the same code
> pattern as on code.google.com
> (http://code.google.com/appengine/docs/java/datastore/jdo/
> dataclasses.html).
>
> The whole thing runs without error but when I open it on Datastore
> viewer at http://localhost:/_ah/admin, where I’m suppose to see
> all my entities, none of them are being displayed. Instead I’m
> receiving the following message:
> “Datastore has no entities in the Empty namespace. You need to add
> data programmatically before you can use this tool to view and edit
> it.”
>
> I even tried to store a simple data object in the datastore for the
> entity I created, but it does not work, so I am confusing whether the
> entity has been created or not.
> (http://code.google.com/appengine/docs/java/datastore/jdo/
> creatinggettinganddeletingdata.html)
>
> Below is my Entity Category
>
> import javax.jdo.annotations.IdentityType;
> import javax.jdo.annotations.IdGeneratorStrategy;
> import javax.jdo.annotations.PersistenceCapable;
> import javax.jdo.annotations.Persistent;
> import javax.jdo.annotations.PrimaryKey;
>
>
> @PersistenceCapable(identityType = IdentityType.APPLICATION)
> public class Category{
>
>         @PrimaryKey
>         @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
>         Long id;
>         @Persistent
>         private String ccode;
>         @Persistent
>         private String cname;
>
>         public Category(String ccode, String cname){
>                 this.ccode = ccode;
>                 this.cname = cname;
>         }
>
>         public Long getID() {
>                return id;
>            }
>         public void setID(Long id) {
>                this.id = id;
>            }
>         public String getCcode() {
>                return ccode;
>            }
>         public void setCode(String ccode) {
>                this.ccode = ccode;
>            }
>         public String getCname() {
>                return cname;
>            }
>         public void setCname(String cname) {
>                this.cname = cname;
>            }
>
> }
>
> Here i"m trying to add an object data in Entity category
>
> Class PMF
> import javax.jdo.JDOHelper;
> import javax.jdo.PersistenceManagerFactory;
> import javax.jdo.PersistenceManager;
> import com.google.appengine.api.datastore.Key;
> import com.google.appengine.api.datastore.KeyFactory;
> import our.fy.project.Category;
>
> public final class PMF {
>    private static final PersistenceManagerFactory pmfInstance =
>        JDOHelper.getPersistenceManagerFactory("transactions-
> optional");
>    private PMF() {}
>    public static PersistenceManagerFactory
> getPersistenceManagerFactory() {
>                return pmfInstance;
>    }
>    public void addCategory(Category c) {
>                PersistenceManager pm =
> getPersistenceManagerFactory().getPersistenceManager();
>                 c = new Category("abc","xyz");
>                Key key =
> KeyFactory.createKey(Category.class.getSimpleName(), "011");
>
>        c.setKey(key);
>
>                try {
>                        pm.makePersistent(c);
>                } finally {
>                        pm.close();
>                }
>        }
>
>
> Can anyone help, whether I am missing something or not? Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: many "server error" responded in the past hour

[Robert Kluin]

Hey Tapir,
  I've recently noticed some strangeness around the instance
management / scheduler too.  I feel like instances are being killed
almost immediately after serving a request, and I've been periodically
seeing instances that don't seem fully loaded (based on mem use and no
requests servered).


Robert




On Mon, Feb 27, 2012 at 02:33, Tapir  wrote:
> About 30% visits will encounter cold startup.
> Why AppEngine doesn't triy to use the resident instance but often try
> to create a new instance to handle new visits?
>
> This problem doesn't happen before, but it is serious today.
>
> On Feb 27, 2:31 pm, Tapir  wrote:
>> Many such logs:
>>
>> A serious problem was encountered with the process that handled this
>> request, causing it to exit. This is likely to cause a new process to
>> be used for the next request to your application. If you see this
>> message frequently, you may be throwing exceptions during the
>> initialization of your application. (Error code 104)
>>
>> There are 2 instances running for about one hour:
>> 0.000   0.0 ms  30      1       0:43:54         98.5 MBytes     Resident 
>> Icon Resident
>> 0.000   0.0 ms  10      0       1:25:04         95.1 MBytes     Dynamic Icon 
>> Dynamic
>>
>> The visits of my app is very small, about one pageview/minute.
>> The 2 running instances are idle at most time.
>> But AppEngine still often tries to create a third one even if the 2
>> running ones are idle so that there are many long-time instance
>> startups.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Python 2.7 Instance problems today ?

[Robert Kluin]

Take screenshots and submit a billing issue.

When you see stuff like that, I'd also submit a production issue.


Robert




On Fri, Feb 24, 2012 at 11:10, Mind  wrote:
> My app serves queries to control a free Android multiplayer game.  QPS
> is normally around 1/sec.  Responses are typically around 1k or less
> of JSON data.  Login, get game data, post score, etc
>
> This morning, I went from the normal 1-2 instances to 10+ instances
> (which I am being charged for!).  Latency is also up to 120 seconds
> (but only 1 instance ?!?) according to the dashboard.
>
> I have 8 instances with 0 QPS and 0 latency sitting around doing doing
> nothing but being billed for.
>
> Did someone break something ?
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Does free quota still allow for "roughly 5 million pageviews a month for an efficient application"?

[Robert Kluin]

I guess the kind of apps I work with tend to be more like Jeff's.
Even with heavy leveraging of memcache, the datastore quota is
probably my biggest limiting factor.  Of course, you'll rapidly hit
the instance hour limit too.





On Tue, Feb 28, 2012 at 09:01, Jeff Schnitzer  wrote:
> On Tue, Feb 28, 2012 at 12:54 AM, sb  wrote:
>>
>> You can do quite well if your app doesn't have a lot of database
>> operations and you make good use of memcache.
>
>
> You get 50k datastore operations per day.
>
> 50k * 30 = 1.5M datastore operations per month.
>
> On average, you are allowed a little less than one datastore operation for
> every 3 pages served.
>
> Possible?  Yes... if you take, as someone mentioned earlier, a very literal
> interpretation of "pages served".  And memcache performs flawlessly the
> entire month.  In practice, few developers will be able to pull this off,
> even with an app which has a very generous read/write ratio.
>
> I love GAE.  It's worth paying for.  But 5M pages per month is not going to
> happen on the free tier... you will need to enable billing long before you
> get to that point.
>
> Jeff
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Thought : Request Queue Management in Google app Engine

[Robert Kluin]

This is related to what the pending request latency controls adjust in
your application's settings.  If you turn the min-pending latency up,
it will be more resistant to spinning up instances.  Turn the max
pending latency down, more willing to spin up instances.  You'll need
to experiment to figure out what works for you since, as Gopal said,
they don't disclose the exact algorithms used.


Robert





On Sat, Feb 25, 2012 at 03:37, pradeep  wrote:
> Hi Google guys,
>
> I have some thought in the Request Queue Management
>
>  * any of the first call takes more time to server and if the age of
> instance increases then it serving fast,
>
> Like we can do the prior detection of the queue and serving level ,
> the instance can be created and loaded the app in to instance  after
> few seconds (consider 30 secs/60 secs) then if we add that instance
> for serving all the call will take less time so that the we can have
> the throughput of the server will be high
>
> If I was wrong please correct me
>
> please let me know where I can find some more information of the
> documentation for how the request queue is managed.
>
> Thanks
> Krishnan
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Entities Dissappeared in Datastore Admin after "Copy to another app"

[Robert Kluin]

Goto the billing settings page of your app.  Change the setting there.


Robert





On Sat, Feb 25, 2012 at 06:52, Sacha Vieux-Roy  wrote:
> How do I so this?
>
> On Feb 24, 12:41 am, Robert Kluin  wrote:
>> HiSacha,
>>   It sounds like you may want to increase your quota in order to
>> finish the copy.  Or, throttle the copy so it isn't burning your quota
>> so fast.  The other issues are all due to running out of quota.
>>
>> Robert
>>
>> On Thu, Feb 23, 2012 at 02:08,SachaVieux-Roy
>>
>>  wrote:
>> > Yes, I am hitting a quota.  The entities showing up in the other app.
>>
>> > I tried to recover the entities using GAE Datastore Backup Utility but got
>> > quota errors.
>>
>> > Can you help?
>>
>> > attached command output.
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "Google App Engine" group.
>> > To view this discussion on the web visit
>> >https://groups.google.com/d/msg/google-appengine/-/WJbhQvM3YUwJ.
>>
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] BTW: There is a great thread about python 2.7 performance on google-appengine-python

[Robert Kluin]

Very nice Andrin.




On Fri, Feb 24, 2012 at 10:24, Andrin von Rechenberg  wrote:
> If you've missed the thread about Python 2.7 performance on
> google-appengine-python:
> http://groups.google.com/group/google-appengine-python/browse_thread/thread/59578235d6a3f48
>
> Cheers,
> -Andrin
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Understanding channels and costs

[Robert Kluin]

Hey David,
  Note that this probably won't work as you're hoping.  You can't have
a broadcast channel:
http://code.google.com/appengine/docs/python/channel/overview.html#Caveats



Robert




On Thu, Feb 23, 2012 at 16:15, David Hardwick
 wrote:
> And how are you?
>
> I see that we get charged for $0.01/100 opening on a channel.
>
>> Opened Channels
>> $0.01/100 Opens
>
>
> So if I have a 'Notification Channel' which I used to send updates on the
> status of a completed backend process, then every time I need to open a
> channel on the client-side it would cause that channel open ticker to
> increment...is that correct?
>
> In use-case terms, if a person logins into the application, and the first
> thing my code does is connect to the Notification channel for that newly
> logged in user (so that the user can now listen for any notifications), then
> I would be incrementing my 'Open Channel' ticker each time someone starts a
> session on my application...is that correct?
>
> Rock on,
>   -Hardwick
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] For information about how to identify the cause of the error to start up the backend

[Robert Kluin]

Sorry, I'm not sure I understand.  Are you saying you got this working
or that it still isn't working?  If it isn't working, what errors are
you getting in your logs?


Robert





On Thu, Feb 23, 2012 at 21:19, linkseed  wrote:
> I have a script for initialization, and you hook up to the start of
> backends, was successful to start the backend.
>
> backends.yaml
>
> backends:
> - name: backend
>   class: B1
>   options: dynamic
>   instances: 1
>   start: warmup.py
>
>
> But, in the hook to the warmup, you could not be started.
>
>
> You need to make carefully advice many times, thank you very much.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/WcrGwDtL9MkJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Store Data from .csv into Database -> Which one?

[Robert Kluin]

Hi Jan,
  Well, there is the datastore:
http://code.google.com/appengine/docs/python/datastore/

  If you don't want to write something to load and parse the csv in to
entities yourself, you can use the bulkloader.
http://code.google.com/appengine/docs/python/tools/uploadingdata.html


  You have really provided any details about what you're doing, so it
is hard to offer any more specific ideas.


Robert





On Thu, Feb 23, 2012 at 05:47, Jan  wrote:
> Hi,
>
> i want to read some data out of .csv files -> so i have rows of data.
> First i thought i store this in a sql database. Problem here: To use
> the google cloud sql, you have to request access for that. I guess
> this takes some time.
>
> Is there any alternative way to store this data and query it
> performant.
>
> Thanks a lot!
> Jan
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Update section of a web page

[Robert Kluin]

Hey Thomas,
  This isn't really an App Engine question, but more of a javascript
question.  I'd suggest asking at something like stack overflow.  Also,
google it:
   
https://www.google.com/search?sourceid=chrome&ie=UTF-8&q=dependent+select+menus



Robert




On Thu, Feb 23, 2012 at 15:32, Thomas Guirden  wrote:
> hi
>
> Is there a neat way to update a section of a web page when working
> with Java and GAE? And passing parameters (e.g. To load data from the
> datastore according to some criteria)?
>
> I want to show a second drop down list ( tag) depending on the
> selection done on the first dropdown ( tag).  E.g. First drop
> down selects the customer, second drop down selects the order, all on
> the same screen.
>
> I thought of two ways to do this... Both a little bit cumbersome and
> my instinct tells me there MUST be a cleaner way to do things.
>
> Both methods use the HTML 5 object tag (or iframe tag) and onChange of
> the first dropdown, I change innerHTML of the object tag to load the
> 2nd sub-page (containing the 2nd  tag).  However to pass
> parameters from the first page to the second using this approach I
> must either build and call a form using Java/JavaScript (to pass
> parameters using POST method) or else save the parameter into the
> datastore from within the first page, and then read this parameter
> from the 2nd sub-page.  And then build the  tag using this
> parameter.
>
> I cannot simply use .innerHTML='etc...'
> because I get problems (I think) with too many single and double
> quotes when building the first page
> (resp.getwriter().println(" .innerHTML =' \"\">etc...'.
>
> Setting innerHTML to simple text without using <> and quotes works
> fine.  I will try to post code later from my development PC.
>
> Thanks for any ideas you may have!
>
> T
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: is app engine down? I'm seeing a huge number of deadline errors and our app failing to server, hr datastore

[Robert Kluin]

If this happens again in the future, you might want to file a production issue:
   
http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue




On Thu, Feb 23, 2012 at 06:45, Phong Luu  wrote:
> Is this problem reoccuring again today? Can anyone look into this ASAP as im
> serving thousands of users globally and it affects our advertisers.
> app id(yt-carousel-gadget.appspot.com). Here is a Youtube channel where it
> exposes the error publically. http://www.youtube.com/user/danonecanal
>
>
> thanks
>
>
>   File
> "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py",
> line 592, in get_result
> return self.__get_result_hook(self)
>   File
> "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
> line 1428, in __get_hook
> self.check_rpc_success(rpc)
>   File
> "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
> line 1182, in check_rpc_success
> rpc.check_success()
>   File
> "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py",
> line 558, in check_success
> self.__rpc.CheckSuccess()
>   File
> "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py",
> line 133, in CheckSuccess
> raise self.exception
>
> Error: An error occurred for the API request datastore_v3.Get().
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/45FXpuJI3jMJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] How to unregister on AppSpot.com? (verification problems with cellphone no)

[Robert Kluin]

Fill out the sms issue form:
   http://code.google.com/appengine/kb/sms.html#error



Robert



On Thu, Feb 23, 2012 at 03:11, chrisno1  wrote:
> I just registered an account on AppSpot.com where I used my own
> cellphone number for verification. However, the Gmail account I used
> is a special student mail and needs another domain for logging in. It
> can't be used for deploying the application.
>
> Now, when I want to register a new account on Google App Engine, I
> can't verify it since my cellphone number has already been used.
> My friend has the same problem.
>
> Hope you understand and thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] After clicking "Copy to another app" in "Datastore Admin" Entities Dissapeared from "Datastory Admin".

[Robert Kluin]

Hi Sacha,
  The entities are probably still there.  You're just burning through
the quota too fast.  Cancel the copy job, and/or pause your queues,
wait a day, then see if everything is showing up again.
Alternatively, go increase your daily budget.



Robert




On Wed, Feb 22, 2012 at 14:21, Sacha Vieux-Roy  wrote:
> After checking 1 small entity and clicking "Copy to another app" in
> "Datastory Admin" Entities Dissapeared from "Datastory Admin".
>
> I created a Google App Engine website in PYTHON.
>
> When I clicked "Copy to another app" in Datastore Admin, all entities
> dissapeared a couple of days later, from the Datastore Admin.
>
> Most of the time, when I click Datastore Viewer I get
> "Server Error
>
> A server error has occurred.
>
> Return to Applications screen."
>
>
> I think the jobs are still running and I have no way to cancel the
> jobs.
>
> Every so often, I can see the entities in the Datastore Viewer
>
>
> Already Tried:
> To Download the entities, but I get "fancy_urllib_InvalidCertException
> = fancy_urllib.InvalidCertificateException AttributeError: 'module'
> object has no attribute 'InvalidCertificateException'"
>
> Is there a way for to recover the entities?
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Verify Your Account by SMS from Uruguay

[Robert Kluin]

http://code.google.com/appengine/kb/sms.html#error




On Wed, Feb 22, 2012 at 10:13, Mondel  wrote:
> Hi could activate my account to google app engine?
> I'm from Uruguay and the service does not work here.
> thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/UixKrWuZVagJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] A single deploy crash all deployed server versions

[Robert Kluin]

Hi Victor,
  Were you deploying to the version that was actively serving traffic
or a new version?  You might want to look into versions.



Robert





On Thu, Feb 23, 2012 at 15:06, Victor Osorio  wrote:
> I was deploying a application on GAE and the "Verifying availability"
> routing was not responding after 12 minutes I cancel, and all my server
> versions crashs.
>
>
> This error is showed after canceling operation:
>
>     com.google.appengine.tools.admin.HttpIoException: Error posting to URL:
> https://appengine.google.com/api/appversion/isready?app_id=app_id&version=1-3-2f&;
>     409 Conflict
>     No transaction is in progress.
>
> I made a deploy on my development server and everything was ok.
>
> The production server are down!
>
>
> The log
>
>> Unable to update:
> com.google.appengine.tools.admin.HttpIoException: Error posting to URL:
> https://appengine.google.com/api/appversion/isready?app_id=app_id&version=1-3-2f&;
> 409 Conflict
> No transaction is in progress.
> at
> com.google.appengine.tools.admin.AbstractServerConnection.send1(AbstractServerConnection.java:281)
> at
> com.google.appengine.tools.admin.AbstractServerConnection.send(AbstractServerConnection.java:234)
> at
> com.google.appengine.tools.admin.AbstractServerConnection.post(AbstractServerConnection.java:213)
> at
> com.google.appengine.tools.admin.AppVersionUpload.send(AppVersionUpload.java:606)
> at
> com.google.appengine.tools.admin.AppVersionUpload.isReady(AppVersionUpload.java:576)
> at
> com.google.appengine.tools.admin.AppVersionUpload.access$100(AppVersionUpload.java:33)
> at
> com.google.appengine.tools.admin.AppVersionUpload$1.call(AppVersionUpload.java:527)
> at
> com.google.appengine.tools.admin.AppVersionUpload$1.call(AppVersionUpload.java:525)
> at
> com.google.appengine.tools.admin.AppVersionUpload.retryWithBackoff(AppVersionUpload.java:657)
> at
> com.google.appengine.tools.admin.AppVersionUpload.commit(AppVersionUpload.java:525)
> at
> com.google.appengine.tools.admin.AppVersionUpload.doUpload(AppVersionUpload.java:141)
> at
> com.google.appengine.tools.admin.AppAdminImpl.doUpdate(AppAdminImpl.java:328)
> at
> com.google.appengine.tools.admin.AppAdminImpl.update(AppAdminImpl.java:52)
> at
> com.google.appengine.eclipse.core.proxy.AppEngineBridgeImpl.deploy(AppEngineBridgeImpl.java:265)
> at
> com.google.appengine.eclipse.core.deploy.DeployProjectJob.runInWorkspace(DeployProjectJob.java:144)
> at
> org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:38)
> at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/79pA4moESVMJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Google app engine - strategic investment for Google

[Robert Kluin]

Hey Hugues,
  This might be of interest to you:

http://googleappengine.blogspot.com/2011/11/app-engine-160-out-of-preview-release.html


Robert





On Thu, Feb 23, 2012 at 15:19, hugues2  wrote:
> Hi all,
>
> After experimenting with GAE/GWT, we will start a development effort soon
> using these technologies. Our main worry is about whether or not GAE is a
> strategic investment for Google and the commitment that Google offers for
> these products.
>
> Is there any view on whether or not Google sees these products as part of
> it's strategic product portfolio ? Maybe another close question is how GAE
> is comparing to competitors such as Amazon or Heroku.
>
> Thanks a lot for sharing your views !
>
> Hugues
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/quvATcn6P0EJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Which task queue exceptions raised from Queue.add() make sense to retry?

[Robert Kluin]

Hey Andy,
  In Python, I retry internal errors.  I believe I have seen timeouts
masquerading as those.

  I'm not sure about the transactional error.  I do not think I have
ever seen one.


Robert




On Thu, Feb 23, 2012 at 11:00, AndyD  wrote:
> Thanks, Robert.
>
> FYI, you're right, a little code spelunking in QueueApiHelper.java shows
> that in Java, the tombstone error becomes a TaskAlreadyExistsException:
>
> case TASK_ALREADY_EXISTS:
>
> return new TaskAlreadyExistsException("Task name already exists : "
> + detail);
>   case TOMBSTONED_TASK:
> return new TaskAlreadyExistsException("Task name is tombstoned : " +
> detail);
> The same file also shows that QueueFailureException is basically a generic
> catch-all for anything not handled explicitly:
>
>  default:
> return new QueueFailureException("Unspecified error (" + errorCode +
> ") : " + detail);
>
> So that seems worth a retry if it pops up.
>
> Regarding TransactionalTaskException, it's used for any Datastore errors
> that arise.  I'm not sure if those would only arise when the task is part of
> an enclosing transaction or not, but based on the name, I'm going to guess
> that's the case.  I got a little lost trying to trace back to the possible
> sources where that could be raised.
>
> Similarly I wasn't able to track down the source of an
> InternalFailureException.  If anybody has an additional insight into those,
> please chime in.
>
> -Andy
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/60AaEXo1XrQJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Generating 50k chunks for sitemap

[Robert Kluin]

Hey Jeff,
  I haven't tried to solve this problem in practice, so this amounts
to thinking out loud; maybe it is helpful, maybe not. ;)

  So each chunk is limited to 50k urls and 10mb uncompressed.  In
other words, you've got to constraints to consider.  In very
simplistic testing, I can compress about 50k urls that are right
around 10mb of generated sitemap data down to around 4mb.  I'd say it
depends pretty heavily on your url schema how many you'll be able to
safely fit in one sitemap.  As  I recall, there is no ordering to your
sitemap, right?  I've got no clue how this might impact SEO type
stuff.  I'm assuming this is an issue because you've got well over 50k
urls; presumably a full "sitemap index" file is sufficient to list
2,500,000,000 urls (ie 50k * 50k).

  If you want to compute this thing on-demand, maybe you could use the
datastore stats to get an idea of how many entities you have
(presumably this is the driver), then figure out how many sitemaps
you'll need.  Then, here's the iffy part, perhaps you could use
something similar to how mapreduce works to shard the key-space into
roughly equal shards.  Basically you'd start fetching ordering by
__scatter__, but you could need a *lot* of keys to make this work
right.  And I mean a lot, probably several hundred k worth of keys.
Then split those up such that you have roughly 50k entities per shard.
 You would need to compute the numbers based on the total count.  This
is just sounding messy, hard, and possibly unreliable.  Precomputing
is sounding better to me.  ;)  If you know something about the
keyspace, you could probably do better.

  How do you get the stuff you want to include in the sitemap?  Is it
fairly static, or always coming in, or is it periodically added in
very large batches?

  If it is pretty static, you can just regenerate it periodically.
Getting the total count is easy enough, then you'll just need to
allocate urls to map files.  You could do this linearly, or break it
up in some way.  Possibly in a way similar to map-reduce.  Just split
the key-space into shards, then within each set of shard start
allocating to sitemaps.  If you're really wanting to maximize urls /
sitemap, you could probably come up with a way to combine the
left-overs from each shard into one (or more) full sitemaps.

  If you're periodically getting the data in large batches, do it in
chunks.  Keep track of you're last partially-full sitemap and start
adding to it until it is full.  Once it is full write it off to the
blobstore and open a new sitemap to start filling.  Keep the current
active sitemap in the datastore so you can add new urls to it, serve
the others from the blobstore.


Just some random thoughts...



Robert



On Thu, Feb 23, 2012 at 08:30, Jeff Schnitzer  wrote:
> The sitemaps protocol, like most things on the internet, seems to be
> slightly retarded.
>
> Ok, the rule for sitemaps is "no more than 50k items".  What do you do when
> you have more?  You need to break it down into 50k chunks and reference them
> from a sitemap index... but how do you get 50k chunks?
>
> The obvious answer is:  Take the max id, divide by 50k, and that's the
> number of sitemaps in your index.  The first sitemap is 0-49,999, the second
> is 50,000-99,, etc.  That works as long as your entities have simple
> numeric ids.  What happens when they don't, either because they have
> ancestors or because they have string keys?
>
> This would be a lot easier if the last line in a sitemap could be a pointer
> to another sitemap, allowing them to chain... but no, the internet was
> designed with the idea that your website is a bunch of files ok a disk.
>  Cursors shmursors.
>
> What do you do?  Precalculate your maps into blobs?
>
> Jeff
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Need documentation of various limits/quotas regarding concurrent requests

[Robert Kluin]

Hey Markus,
I've looked at using the Channel API or third party solutions in
several projects.  In one I went with the Channel API to test it out a
little.  I've still not made heavy use of it though.  I've looked at
the third party solutions a bit, but have not used one in a production
app.  I'm sometimes very lazy, and in these cases old-fashioned
polling was sufficient.  So,  I'm not sure about self-hosting the
requisite javascript libs.  I'd love to hear about your thoughts if
you dig into one though.


Robert





On Thu, Feb 23, 2012 at 05:49, Markus  wrote:
> Hey Robert,
>
> what a small world! Maybe we should post a feature request to the channels
> api people.
>
> Do you know if it's possible with any of these other solutions to host their
> javascript libraries from your server as well? Preferably in a way that you
> won't have to check every hour whether they updated their server-side code
> to break compatibility with your version of the client library?
>
> Which solution did you end up using?
>
>
> Thanks,
>
> Markus
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/41i3jv2Y5b0J.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Unneeded instance started

[Robert Kluin]

Hey Francois,
  Interesting.  On apps where I'd set it to auto, I generally see the
billed line track with the total line.  I also generally see instances
linger for a long, long time being largely unused.



Robert





On Thu, Feb 23, 2012 at 03:58, Francois Masurel  wrote:
> Hi Robert,
>
> I have set Max Idle Instances setting to full Automatic and my frequent
> instance restarts have disappeared.  No more "loading_request=1" in my logs
> every 3 minutes.
>
> I now have to dynamic instances permanently "active" but only one seems to
> be "billed" (cf. attachments).  So, for the moment, it does not cost me more
> than before.
>
> But is it normal behavior ?  I better keep an eye on it.
>
> François
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/ONbZnH5_VSEJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Entities Dissappeared in Datastore Admin after "Copy to another app"

[Robert Kluin]

Hi Sacha,
  It sounds like you may want to increase your quota in order to
finish the copy.  Or, throttle the copy so it isn't burning your quota
so fast.  The other issues are all due to running out of quota.


Robert





On Thu, Feb 23, 2012 at 02:08, Sacha Vieux-Roy
 wrote:
> Yes, I am hitting a quota.  The entities showing up in the other app.
>
> I tried to recover the entities using GAE Datastore Backup Utility but got
> quota errors.
>
> Can you help?
>
> attached command output.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/WJbhQvM3YUwJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Understanding "The App Engine Way"

[Robert Kluin]

More used items will remain in cache longer.  So stuff that is
infrequently used should naturally fall out.

If you've got entities that are frequently written to and seldom read,
memcaching them will obviously have a lower value.  Particularly if
they are updated very often.  Note that it may still make sense to
memcache these in some cases, since it could still save a trip to the
datastore.




Robert




On Thu, Feb 23, 2012 at 21:43, Thomas Wiradikusuma
 wrote:
> I haven't actually used it other than the one provided by Objectify :D
>
> I'm thinking of putting more (mostly rendered pages, recently accessed
> items, lookups), but afraid memcache will become full and evicting more
> important cache objects.
>
> What kind of misuse that you're talking? If I know I would avoid it.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/lQgcuPRdWgIJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: 1.6.3 Prerelease SDK Available

[Robert Kluin]

I totally agree this is an excellent feature.  Should give us good
options for making the rollout of app updates a little safer.



2012/2/22 sebastián serrano :
> Wow, traffic splitting is a really nice feature.
>
> -Sebastian
> www.devsar.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/poENbFXYWFsJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Caching a list on high replication datastore

[Robert Kluin]

For the case of adding, you can also track what was just added, then
ensure that it is included in the results.  So you'll run your normal
query (without the additional lookup step), then just ensure the new
item is in the list.  The same can actually work for updates.  If you
update one entity, ensure that the next page view updates or replaces
the version of that entity from the query results.

In practice, this type of method may be cheaper since you're just
doing one query and not a query followed by a batch get (which will
potentially be many RPCs).  Of course, if you want everything to be
consistent then Josh's method is probably about the best you can do.
You could probably improves it a bit by caching the entities in
memcahce too; in other words, check memcache then only goto the
datastore for keys not in memcahce.


Robert



On Wed, Feb 22, 2012 at 17:44, Joshua Smith  wrote:
> It depends how you get the list of projects.
>
> Queries are eventually-consistent, but fetches are consistent right away.
>
> So if the list for a user is stored as a list of ID's in the user's record,
> you can fetch that, then fetch the items listed, and all will be consistent
> always.
>
> But if you are querying for projects that happen to have a user listed
> (WHERE user = :1), then you need to get clever.
>
> Delete and rename are actually pretty easy to handle if you can afford
> another round trip to the database. Instead of querying for the records
> completely, you query for the keys, and then fetch those records. Here's the
> code I use:
>
> class HRModel(db.Model):
>   @classmethod
>   def gql_with_get(cls, query_string, *args, **kwds):
>     return filter(None, db.get(db.GqlQuery('SELECT __key__ FROM %s %s' %
> (cls.kind(), query_string), *args, **kwds)))
>
> It works just like Model.gql() but it does the two-step. The db.get will get
> the consistent data. The filter is needed to handle deletes.
>
> This will not detect new records appearing, however. There is no pretty way
> to deal with that. You have to somehow let the process doing the query know
> that if it doesn't see a certain record, it should retry.
>
> In practice, I work around this through a trick in most cases. When you
> create a new project, the user probably needs to fill some stuff in. So
> create the record right away, and put in enough info so it will appear in
> your query results. Then have the user edit the existing record to specify
> the rest of the data, and put it back to the datastore.
>
> By the time the user has done their part, eventual consistency will have
> occurred, and Bob's your uncle.
>
> -Joshua
>
> On Feb 22, 2012, at 5:25 PM, Andrew Richardson wrote:
>
> I'm considering switching to the high replication datastore, and I want to
> make sure I understand what needs to be cached in order to deal with
> eventual consistency. I know this question has been asked many times...just
> want to make sure I'm understanding after reading some other answers.
>
> I have a page that displays a list of projects which a user is part of. From
> this page, they can rename a project, leave a project, or create a new
> project. If they perform any of these actions, the page is refreshed, and
> obviously should reflect the change.
>
> This list is already being cached...so if user #1 is logged in, memcache may
> have "user-1-projects" stored. On the master-slave datastore, I can simply
> delete this value if they rename/leave/create a project, and the next page
> view will cause it to be rebuilt from a datastore query. But as I understand
> it, on the HRD I will have to modify the cached value in-place rather than
> deleting it...ie I retrieve the "user-1-projects" list from the cache,
> rename/create/delete the relevant item, then write the updated
> "user-1-projects" entry back to the cache. If I just delete it as I'm
> currently doing, the subsequent query may return stale results.
>
> Is this correct?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/Aen4pAMD2LAJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To u

Re: [google-appengine] Entities Dissappeared in Datastore Admin after "Copy to another app"

[Robert Kluin]

Are you hitting a quota?  Are the entities showing up in the other app?



On Wed, Feb 22, 2012 at 09:40, Sacha Vieux-Roy
 wrote:
> I created a Google App Engine website in PYTHON.
>
> When I clicked "Copy to another app" in Datastore Admin, all entities
> dissapeared a couple of days later, from the Datastore Admin.
>
> Most of the time, when I click Datastore Viewer I get
> "Server Error
>
> A server error has occurred.
>
> Return to Applications screen."
>
>
> I think the jobs are still running and I have no way to cancel the jobs.
>
> Every so often, I can see the entities in the Datastore Viewer
>
> Already Tried:
> To Download the entities, but I get "fancy_urllib_InvalidCertException =
> fancy_urllib.InvalidCertificateException AttributeError: 'module' object has
> no attribute 'InvalidCertificateException'"
>
> Is there a way to recover my entities?
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/Cph1uH3L5L0J.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] For information about how to identify the cause of the error to start up the backend

[Robert Kluin]

That will cause a request to /_ah/warmup.  As I recall, you should
handle that.  The whole point in warmup requests is to give you a way
to initialize your app.



Robert






On Wed, Feb 22, 2012 at 10:04, linkseed  wrote:
>> Could you include your app.yaml and corresponding url mapping for the
>> warmup request?
>
>
> Do you need something other than this description to app.yaml?
>
> inbound_services:
> - warmup
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/7_pUTOL7L-MJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Unneeded instance started

[Robert Kluin]

Hey Francois,
  I'm curious, have you noticed an increase since setting this to
automatic?  On several apps where I've bumped into this issue in the
past week or so I've seen a bunch of instances hang around through low
traffic times.



Robert



On Wed, Feb 22, 2012 at 09:20, Francois Masurel  wrote:
> I have set Max Idle instances to Automatic, just hoping it won't cost me too
> much and hit my daily budget.
>
> Thanx for helping.
>
> François
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/vEjQDF-eBz4J.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Which task queue exceptions raised from Queue.add() make sense to retry?

[Robert Kluin]

I would also suggest you handle TaskAlreadyExistsException.  In some
cases you may want to swallow that error, and in others raise it.  In
Python we also have a TombstonedTaskError, I guess that is handled by
TaskAlreadyExistsException in Java.

Actually, in Python we have a great deal more exceptions.  I've got a
list of exceptions similar to yours that I retry.  These also include
deadline exceeded and timeout errors.



Robert





On Wed, Feb 22, 2012 at 08:28, AndyD  wrote:
>
> [hmm, this question didn't attract any interest on StackOverflow..., thought
> I'd give it a try here]
>
> I'm looking to add some defensive exception handling/retry logic around my
> (Java) code that enqueues tasks in App Engine. In reviewing the
> documentation for potential exceptions raised, I see several candidates that
> might make sense to catch and handle with a retry, but the documentation is
> a bit vague, so I thought I'd see what others have found to be worthwhile.
>
> Here are the ones I'm looking at, and their descriptions from the docs:
>
> InternalFailureException - Internal task queue error.
> TransientFailureException - Intermittent failure.
> QueueFailureException - Unspecified queue failure.
> TransactionalTaskException - Queue operation failure caused by Datastore
> exception.
>
> The only one for which the docs explicitly suggest that retrying might be
> helpful is TransientFailureException.
>
> Also, regarding TransactionalTaskException, is this something that would
> only arise when enlisting a task in an enclosing datastore transaction, or
> could this arise when enqueuing a standalone task?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/JV4cR4o2BNYJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: [appengine-python] 2012 US PyCon -- GAE Related Sprints

[Robert Kluin]

On Wed, Feb 22, 2012 at 12:18, Guido van Rossum  wrote:
> On Mon, Feb 20, 2012 at 22:46, Robert Kluin  wrote:
>> Hey Everyone,
>>  Myself and several colleagues are going to be at pycon, and sticking
>> around for a few days of sprints after the main conference is over.
>> We're planning to discuss and work on several App Engine related
>> projects during that time.  Some of the projects are pretty neat, and
>> should benefit the larger App Engine Python community.  We'd also love
>> to have some help or more ideas for cool things to work on!  If you're
>> going to be at PyCon and sticking around, feel free to sign up or just
>> catch one of us at the conference.  Feel free to stop by even if you'd
>> just like to nerd-out about App Engine / Python!
>
> Hi Robert,
>
> Let's make sure we meet up in Santa Clara! I'll try to make it to some
> of the sprints though I may have to divide my attention between Python
> core and App Engine...
>
>>  Some current items on the project list:
>>    - An ext.db mock for "unit" tests that offers basic functionality
>>      (including querying!) without the poor performance of the SDK stub.
>
> I hope you didn't literally mean ext.db -- please mock at a lower
> level so that NDB can also transparently benefit from this. (Note that
> the latest trend seems to be partial use of NDB, possibly prompted by
> the User model in webapp2_extras.)

Hey Guido,
  Sounds great.  Perhaps we can discuss how to best accomplish this a
little bit at pycon!  I've got some basic ideas, but hopefully
together we can come up with a better strategy.


Robert



>
>>    - Work on PyAMF and RTMPy.
>>    - Since we'll soon have pull-queue tagging, I want to get out the
>>      updates I've been holding back for slagg.
>>    - I'd like to put together a Cassandra stub for the SDK.  I made
>>      good progress on this last year, but it has fallen behind.
>>
>>
>>  I've added our "spint" to the pycon sprints page, see "APP ENGINE
>> RELATED PROJECTS (COMMUNITY RUN)"
>>    https://us.pycon.org/2012/community/sprints/projects/
>>
>>
>>
>> Robert
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "google-appengine-python" group.
>> To post to this group, send email to 
>> google-appengine-pyt...@googlegroups.com.
>> To unsubscribe from this group, send email to 
>> google-appengine-python+unsubscr...@googlegroups.com.
>> For more options, visit this group at 
>> http://groups.google.com/group/google-appengine-python?hl=en.
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Need documentation of various limits/quotas regarding concurrent requests

[Robert Kluin]

Hey Markus,
  We actually have *exactly* the same problem with Channel API.
There's no way for us to use it since it is going to
talkgadget.google.com.

  If I remember correctly, beaconpush, or one or more of their
competitors, actually allows you to use your own domain.  It will look
a lots less suspicious if it is hitting "service.yourdomain.com."


Robert



On Wed, Feb 22, 2012 at 11:37, Markus  wrote:
> interesting... beaconpush actually offers pretty much exactly what I'm
> looking for.
>
> There's just one big problem, that the Channels API (and probably beaconpush
> as well) have for me: I'm working on a security-focused service, and the
> last thing that I can afford to do is have my page open random non-https
> requests to any domains other than the one from the main page. Especially
> not TALKGADGET.google.com. Even though "beaconpush.com" already sounds a lot
> less gossipy, I still don't think it would be a good idea. At least I would
> find such a page highly suspicious if I cared about my data being safe with
> them. Even including external javascript libraries is something I'd really
> like to avoid.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/3SUBVUny9a0J.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Python 2.7: Instance memory limitations with concurrent requests

[Robert Kluin]

Hey Alex,
  I should probably have stated this better as memory is not always
handled well.  For example, the ext.db code keeps many copies of the
data in various forms.  This can cause rapid and unexpected memory
blowups, and the result is something that appears similar to a memory
leak.  As Brian noted, this is partially caused by Python's handling
of memory.

  However, there are a number of scenarios where you get real memory
leaks.  For example there have recently been several posts / issues
from people having issues with the blobstore leaking memory.  Some of
these are quite detailed and the repro code is very simple.  If I'm
not mistaken, in the past, we've observed this happening with heavy
datastore use as well, though I don't have simple repro cases for
those.



Robert





On Tue, Feb 21, 2012 at 16:13, alex  wrote:
> The whole point of this topic was python27 runtime, multithreading and
> concurrent requests. Your specific case, plus python 2.5, doesn't
> necessarily means memory leaks in the runtime itself. I'd profile my code
> that handles most frequently accessed URLs to start off.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/ZyB9_mGBPDQJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] problem in upload data using remote_api and loader

[Robert Kluin]

Hi,
  You didn't really give details about the command you're actually
trying to run, but you might try calling appcfg with "-A s~finaldidi"


Robert



On Tue, Feb 21, 2012 at 04:42, didi  wrote:
> hello everyone,
>
> I referenced the artical 
> http://code.google.com/intl/en/appengine/docs/python/tools/uploadingdata.html
>
> but it shows error message: BadRequestError: app s~finaldidi cannot
> access app finaldidi's data
>
> Could anyone helps me?thanks
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] The API call file.Create() took too long to respond and was cancelled

[Robert Kluin]

It is also reported in:
  http://code.google.com/p/googleappengine/issues/detail?id=6888

There have been several others reporting similar issues lately:
   
https://www.google.com/search?sourceid=chrome&ie=UTF-8&q=app+engine+groups+http%3A%2F%2Fcode.google.com%2Fp%2Fgoogleappengine%2Fissues%2Fdetail%3Fid%3D6888#hl=en&safe=off&q=+site:groups.google.com+app+engine+groups+The+API+call+file.Create()+took+too+long&sa=X&ei=tZxET4_6K8bUgAfCj9mlBA&ved=0CEQQrQIwBA&bav=on.2,or.r_gc.r_pw.r_cp.r_qf.,cf.osb&fp=7f381450be81b469&biw=1573&bih=957





On Tue, Feb 21, 2012 at 18:04, Hernan Liendo  wrote:
> Hi there!
>
> we're getting this exception sometimes when trying to write a file to
> the blobstore. It usually appears for one or two minutes and then
> everything gets back to normal again.
>
> It's reported on 
> http://code.google.com/p/googleappengine/issues/detail?id=5723
> but marked as closed.
>
> Do you guys experienced anything like this?
>
> The complete stacktrace is:
>
> com.google.apphosting.api.ApiProxy$ApiDeadlineExceededException: The
> API call file.Create() took too long to respond and was cancelled.
>        at java.lang.Thread.getStackTrace(Thread.java:1495)
>        at
> com.google.apphosting.runtime.ApiProxyImpl.doSyncCall(ApiProxyImpl.java:
> 218)
>        at com.google.apphosting.runtime.ApiProxyImpl.access
> $000(ApiProxyImpl.java:67)
>        at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:
> 163)
>        at com.google.apphosting.runtime.ApiProxyImpl$1.run(ApiProxyImpl.java:
> 161)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at
> com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:
> 161)
>        at
> com.google.apphosting.runtime.ApiProxyImpl.makeSyncCall(ApiProxyImpl.java:
> 67)
>        at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:101)
>        at com.google.apphosting.api.ApiProxy.makeSyncCall(ApiProxy.java:50)
>        at
> com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:
> 557)
>        at
> com.google.appengine.api.files.FileServiceImpl.create(FileServiceImpl.java:
> 479)
>        at
> com.google.appengine.api.files.FileServiceImpl.createNewBlobFile(FileServiceImpl.java:
> 99)
>        at
> com.google.appengine.api.files.FileServiceImpl.createNewBlobFile(FileServiceImpl.java:
> 81)
>        at
> ar.com.zupcat.lib.util.BlogstoreUtil.saveToBlogstoreImpl(BlogstoreUtil.java:
> 69)
>
> Thanks!
> Hernán
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Need documentation of various limits/quotas regarding concurrent requests

[Robert Kluin]

Why are you doing this rather than using a service such as beaconpush.com?

As for wanting limits thoroughly documented, I could not agree more.
This is some thing that should be done, and I frankly can't understand
why it isn't.  I would like to see something like:
   http://code.google.com/p/googleappengine/issues/detail?id=5677

Along the same lines, it is very annoying to hit temporary
(undocumented!) quotas, such as datastore operations per minute, then
have no record of exactly what quota was hit.  I'd love to see a daily
log of *exactly* every quota an app hits with enough detail for me to
tell Google what I'm hitting:
http://code.google.com/p/googleappengine/issues/detail?id=6296


Robert




On Tue, Feb 21, 2012 at 14:58, Markus  wrote:
> Ouch! Really??? This is a MAJOR deal-breaker for one of my projects.
>
> But if that's the case, what good is ? Is it
> really only there to limit the number of requests even further?
> Wouldn't it be brilliantly sensible and useful to use this setting to
> specify the size of the thread pool?
>
> Do you know if this is a soft-limit? I.e. could I request an increase?
> With the app I'm working on, I'd love to have many 10s of thousands of
> requests simultaneously locked in a wait() on a single backend. This
> should consume virtually no CPU and probably not much memory either.
> But with 10 threads (i.e. 10 requests) per instance, there is no way
> this would work.
>
> In case you guessed it: I am indeed trying to build my own much
> improved version of the Channels API.
>
> Thank you for the data-point. This might also explain the 6-request-
> limit for my backend, if there are 4 other things going on in parallel
> that I didn't initiate.
>
> Markus
>
>
>
> On Feb 21, 8:46 pm, Jeff Schnitzer  wrote:
>> I suspect one of the missing pieces of information is that multithreaded
>> Java instances can only handle 10 threads right now (or so I have been
>> told).
>>
>> Jeff
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Feb 21, 2012 at 2:05 PM, Markus  wrote:
>> > Hi,
>>
>> > first of all, let me say that I think AppEngine is one of the greatest
>> > ideas ever. I love the concepts behind it; even some of the
>> > restrictions it comes with that allow it to be so scalable.
>>
>> > Unfortunately, though, I keep running into random undocumented limits,
>> > that force me to rewrite my code frequently, or to even abandon GAE
>> > for certain projects altogether.
>>
>> > Right now, I am fighting with concurrent requests and have found
>> > several limits that I didn't see documented anywhere and I am not sure
>> > whether I should be seeing them, or whether I am doing something
>> > wrong. So I am hoping one of the Google engineers is available and can
>> > give some background info regarding some of the limits/quotas/strange
>> > behaviors I found with one of my test apps:
>>
>> > - Frontend instances can handle only about 30 requests/s (qps). (I've
>> > seen that number mentioned in posts somewhere, but I can't find it in
>> > the docs.)
>>
>> > - Frontend instances keep getting created by the scheduler "randomly",
>> > sometimes already when I only have 2-3 requests pending, even if I set
>> > the respective Application settings to max 1 idle instance and max 15s
>> > request latency.
>>
>> > - Backend instances can handle only 10 requests/s (qps), independent
>> > of instance class. (I haven't seen this mentioned anywhere.)
>>
>> > - Backend instances can only have 6 requests open simultaneously,
>> > independent of how I set max-concurrent-requests. (Someone mentioned
>> > this in a post in 2009 for a frontend instance, but I can't find any
>> > infos on backends.)
>>
>> > - My backend sends "503 Service unavailable" errors randomly, if there
>> > are too many requests pending. Unfortunately I can't get a number on
>> > when this happens (how many requests is too many). Also, I found this
>> > surprising since I thought that the whole point of backends was to not
>> > have these sorts of limits.
>>
>> > Any light anyone can shed on these questions would be much
>> > appreciated.
>>
>> > I am most interested in the  setting in
>> > backends.xml. Should this do something? I guess if I had written GAE
>> > (I wish), this would specify the size of the thread-pool for request
>> > servlets? Is there a limit/quota for how high I can set this number?
>>
>> > Are there any differences in these limits between paid and free apps?
>> > How about Python vs. Java?
>>
>> > Do the 9 hours per day of free backend use also apply for paid apps?
>> > (Or more general: are all free quotas still free for paid apps?)
>>
>> > And, I guess, the question that non-Google-engineers can answer to
>> > help me in this quest is:
>>
>> > - Has anyone been able to exceed any of the above mentioned limits? If
>> > so, do you know how?
>>
>> > Thank you very much for your help,
>>
>> >  Markus
>>
>> > PS: Is there a chance we can get all quotas/pricing information on a
>> > single page? Currently there are:
>> 

Re: [google-appengine] Re: 1.6.3 Prerelease SDK Available

[Robert Kluin]

Hey Marzia,
  Hopefully it will find its way to the dashboard soon though ?   :)


Robert





On Tue, Feb 21, 2012 at 14:21, Marzia Niccolai
 wrote:
> Hi Robert,
>
> On Mon, Feb 20, 2012 at 11:30 PM, Robert Kluin 
> wrote:
>>
>> These are some of the best features in recent releases:
>>  - Datastore statistics now show the amount of storage used by application
>>    indexes.
>
>
> Sorry, this was my fault. This feature was meant to be removed from the
> notes that I posted, and aren't in the SDK.
>
> But hopefully gives you a taste of things to come...
>
> -Marzia
>
>>
>>  - The instances screen in the Admin Console now has a button to shut down
>>    a specific instance.
>>  - Each application log has a link to the instance that served the
>> request. If
>>    the instance is no longer serving requests, a message will be
>> displayed.
>>  - The TaskQueue API now supports the ability to tag a Pull Queue task.
>> You may
>>    then lease tasks by tag.
>>
>> Really looking forward to this going live!
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Feb 20, 2012 at 21:04, Jeff Schnitzer  wrote:
>> > See thread subject.
>> >
>> > Prerelease != Release
>> >
>> > Jeff
>> >
>> >
>> > On Mon, Feb 20, 2012 at 2:14 PM, GAE  wrote:
>> >>
>> >>
>> >> I just deployed a java pp. However I can't see that shutdown button on
>> >> the
>> >> Instances page in the Admin Console. Where is it suppoed to be?
>> >>
>> >>
>> >> On Monday, February 20, 2012 7:03:01 PM UTC+1, Marce (Google) wrote:
>> >>>
>> >>> - The instances screen in the Admin Console now has a button to shut
>> >>> down
>> >>>   a specific instance.
>> >>> - Each application log has a link to the instance that served the
>> >>> request. If
>> >>>   the instance is no longer serving requests, a message will be
>> >>> displayed.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "Google App Engine" group.
>> >> To view this discussion on the web visit
>> >> https://groups.google.com/d/msg/google-appengine/-/z53DnK1Uef8J.
>> >>
>> >> To post to this group, send email to google-appengine@googlegroups.com.
>> >> To unsubscribe from this group, send email to
>> >> google-appengine+unsubscr...@googlegroups.com.
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/google-appengine?hl=en.
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: App Engine Takes 20 secs to serve some times

[Robert Kluin]

Just a note, you can confirm that this is indeed the issue by looking
at the logs.  If it is just your apps startup time you'll see a
"loading_request=1" following the latency info (when you expand the
log entry).


Robert




On Wed, Feb 22, 2012 at 00:59, Krishnan PA  wrote:
>
>
> On Tue, Feb 21, 2012 at 3:57 PM, pradeep  wrote:
>>
>> Hi All,
>>
>> Can any one help me to find the issue
>>
>> actually the Time of service changes every time I not getting what was
>> the reason
>>
>> It takes 20s for few times and it takes 65-100ms most of the time for
>> same service call
>>
>> And I am using multithread as true
>> and the cache is basically the asynchronous memcache
>> App engine version is 1.6.1
>>
>> If you found it, what would be cause of issue in this can share with
>> me please
>>
>> Thanks
>> Krishnan
>
>
> Thanks Guys,
>
> --
> Regards
> Krishnan
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Task Queue Craziness

[Robert Kluin]

I agree that it is dangerous to blindly catch all errors.  However, in
this case you could catch and log the full trace.  That way you get
your logging info, but you don't get dupes.

  try:
 # do ... lots ... of ... things
 raise Exception('Ka Powww!')
  except:
 logging.exception('A task crapped out.')



Robert



On Tue, Feb 21, 2012 at 15:02, James Gilliam  wrote:
> I usually don't usually catch all errors because it hides diagnostic
> info ... but since I can't limit the retries, i usually get the error
> twice ...
>
> On Feb 21, 11:20 am, Jeff Schnitzer  wrote:
>> If you don't want a task to retry, catch any errors and simply return
>> normally as if the task completed.
>>
>> Jeff
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Feb 21, 2012 at 2:15 PM, James Gilliam  wrote:
>> > one other task queue question ...
>>
>> > i have ...
>>
>> >  retry_parameters:
>> >    task_retry_limit: 1
>>
>> > is there any way i can say no retries?
>>
>> > a while back, i tried task_retry_limit: 0 but didn't work
>>
>> > thanks
>>
>> > On Feb 21, 10:56 am, Robert Kluin  wrote:
>> > > Why not leave the rate at 1/s, but set max_concurrent_requests to 1 as
>> > > Nick suggested?
>>
>> > > On Tue, Feb 21, 2012 at 12:56, James Gilliam 
>> > wrote:
>> > > > Yes ... I saw the 5 per sec in the admin ...
>>
>> > > > Also, I just realized why I think gae created so many instances when i
>> > > > had the queue at 1/s ... I had assumed the tasks would take much less
>> > > > than a second (which was true at the beginning); but as the number of
>> > > > entity writes continued, the latency for the writes got larger and
>> > > > eventually took longer than a second ... at which point gae started
>> > > > new instances to keep up with the 1 sec rate ... i had assumed it
>> > > > would not consider starting a new task until the previous one in the
>> > > > queue finished -- this appears to not be the semantics
>>
>> > > > My solution is to slow down the tasks even further ... to say 15 /
>> > > > min ... so that new instances will not be created ...
>>
>> > > > On Feb 19, 12:21 pm, Nicholas Verne  wrote:
>> > > >> James,
>>
>> > > >> The misreporting of number of tasks in a queue is a known bug that
>> > > >> we're working on.  It causes undue alarm for our users.
>>
>> > > >> How do you know that GAE changed the target rate to 5 / sec ? Did you
>> > > >> observe this in the admin console?
>>
>> > > >> The queue's rate is the rate at which tasks are dispatched to your
>> > > >> app. If your queue is set to run 1 / sec and your task latency is
>> > > >> longer than 1 second, you should expect to have more instances spin
>> > > >> up.
>>
>> > > >> If you wish to prevent more than one task running at a time, set your
>> > > >> queue's max_concurrent_requests to 1.
>>
>> > > >> Nick Verne
>>
>> > > >> On Mon, Feb 20, 2012 at 3:35 AM, James Gilliam 
>> > wrote:
>> > > >> > I am using task queue in python.
>>
>> > > >> > I added 10,000 tasks for a queue set to run at 1/sec and a bucket of
>> > > >> > 1 ... they were short tasks and I didn't want them to run
>> > > >> > immediately ... I didn't even want them to run multiples per second
>> > > >> > even though the latency was low ... it was ok if it took hours as I
>> > > >> > knew it would at 1/sec.  I figured it would create a second instance
>> > > >> > as the site was continuing to run ... but I was pretty surprised
>> > with
>> > > >> > what actually happen.
>>
>> > > >> > At first the queue started processing fine ... approx 60 tasks
>> > running
>> > > >> > per minute ...
>>
>> > > >> > Then GAE decided to run them at a faster rate, change the target
>> > rate
>> > > >> > to 5 / sec and created lots of new instances ...
>>
>> > > >> > When there were about 3000 tasks left ... GAE went crazy and said
>> > > >> > there were nearly 200,000 tasks in the queu

Re: [google-appengine] Re: Task Queue Craziness

[Robert Kluin]

Why not leave the rate at 1/s, but set max_concurrent_requests to 1 as
Nick suggested?



On Tue, Feb 21, 2012 at 12:56, James Gilliam  wrote:
> Yes ... I saw the 5 per sec in the admin ...
>
> Also, I just realized why I think gae created so many instances when i
> had the queue at 1/s ... I had assumed the tasks would take much less
> than a second (which was true at the beginning); but as the number of
> entity writes continued, the latency for the writes got larger and
> eventually took longer than a second ... at which point gae started
> new instances to keep up with the 1 sec rate ... i had assumed it
> would not consider starting a new task until the previous one in the
> queue finished -- this appears to not be the semantics
>
> My solution is to slow down the tasks even further ... to say 15 /
> min ... so that new instances will not be created ...
>
> On Feb 19, 12:21 pm, Nicholas Verne  wrote:
>> James,
>>
>> The misreporting of number of tasks in a queue is a known bug that
>> we're working on.  It causes undue alarm for our users.
>>
>> How do you know that GAE changed the target rate to 5 / sec ? Did you
>> observe this in the admin console?
>>
>> The queue's rate is the rate at which tasks are dispatched to your
>> app. If your queue is set to run 1 / sec and your task latency is
>> longer than 1 second, you should expect to have more instances spin
>> up.
>>
>> If you wish to prevent more than one task running at a time, set your
>> queue's max_concurrent_requests to 1.
>>
>> Nick Verne
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Feb 20, 2012 at 3:35 AM, James Gilliam  wrote:
>> > I am using task queue in python.
>>
>> > I added 10,000 tasks for a queue set to run at 1/sec and a bucket of
>> > 1 ... they were short tasks and I didn't want them to run
>> > immediately ... I didn't even want them to run multiples per second
>> > even though the latency was low ... it was ok if it took hours as I
>> > knew it would at 1/sec.  I figured it would create a second instance
>> > as the site was continuing to run ... but I was pretty surprised with
>> > what actually happen.
>>
>> > At first the queue started processing fine ... approx 60 tasks running
>> > per minute ...
>>
>> > Then GAE decided to run them at a faster rate, change the target rate
>> > to 5 / sec and created lots of new instances ...
>>
>> > When there were about 3000 tasks left ... GAE went crazy and said
>> > there were nearly 200,000 tasks in the queue ...
>>
>> > As far as I could tell, the number of tasks being reported was just an
>> > anomaly ... but it was alarming given that GAE was creating instances
>> > like crazy.
>>
>> > The tasks would add new records (sometimes) and I noticed that the
>> > latency of these tasks increased the longer the tasks ran.  At first
>> > just 200ms ... by the end nearly 1400 ms.
>>
>> > So --
>>
>> > 1. The most important question -- Why did GAE not respect my desire to
>> > run the queue at 1 / sec ?  Is there a way I can stop GAE from
>> > overriding my wishes?
>>
>> > 2. Was the latency increasing because the records being added were
>> > slowing new records from being added?
>>
>> > 3. The invalid number of tasks seems like a harmless bug since new
>> > phantom tasks were not actually created (as far as I know).
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups 
>> > "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to 
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group 
>> > athttp://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] For information about how to identify the cause of the error to start up the backend

[Robert Kluin]

Could you include your app.yaml and corresponding url mapping for the
warmup request?


Robert



On Tue, Feb 21, 2012 at 11:31, linkseed  wrote:
> Thank you for advice.
>
>
>>   Have you tried to define a warmup handler?  You just need something
>> to handle that request.
>
>  I tried to define, the results did not change.
> Was the same even if you change to an undefined further.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/25mbdIqDkGAJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Python 2.7: Instance memory limitations with concurrent requests

[Robert Kluin]

To be clear, the scheduler can still dispatch multiple requests for
that url to the same instance.  Only one request will execute at a
time though.  One of the biggest issues is that the Python runtime
leaks memory like mad, so in this case you may wind up with 1)
increased latency and 2) still blowing mem limits fast.


Robert



On Tue, Feb 21, 2012 at 09:36, Jeff Schnitzer  wrote:
> On Mon, Feb 20, 2012 at 5:33 AM, Johan Euphrosine  wrote:
>>
>> On Mon, Feb 20, 2012 at 11:25 AM, Andrin von Rechenberg
>>  wrote:
>>>
>>> I guess that's the same solution as just deploying two different
>>> versions. A threadsafe one and a non threadsafe one. Or did
>>> I misunderstand you?
>>
>>
>> appcfg.py provide commands to help you manage your backends deployment and
>> configuration, with `backends.yaml` and `appcfg backends update`.
>>
>> That make the solution more convenient than using multiple versions on
>> frontend instances.
>
>
> This adds the significant downside of no auto-scaling.  I don't see how
> that's more convenient :-)
>
> Andrin:  In javaland I would simply synchronize the people-search function
> so that at most one thread can execute that routine in a single instance at
> once.  It means all people-search requests in that instance will queue up in
> serial, which could cause undesirable waits if the function takes
> significant time, but if the requests are spread out among enough instances
> it probably won't be an issue.
>
> I don't know what the python equivalent of 'synchronized' is.
>
> Jeff
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] More than 1800 QPS

[Robert Kluin]

Hey Herman,
  You're hitting the timeout on the get from memcache?  One suggestion
would be to figure out what the typical response time is then set a
lower deadline, such as 5 seconds.  That will let you catch the
deadline error on the cache request sooner and get the value from the
datastore.  I've successfully used this method in high-qps situations
several times.

  Also, the user-request deadline is now 60 seconds.
http://code.google.com/appengine/docs/python/runtime.html#Quotas_and_Limits

  Congrats on getting into these types of issues.  ;)

Robert



On Mon, Feb 20, 2012 at 10:25, Hernan Liendo  wrote:
> Hi there!,
>
> six month ago we've decided to use AppEngine to host our games. After
> a successful launch we are starting to worry about its limits.
> We have a growing game with approximately 30.000 daily unique users
> and this is just the beginning.
>
> We are serving more than 40 instances with a 300ms latency and 1.800
> QPS. The problem is that we're having 3 to 4 timeout errors per minute
> in our log. Most of them are related to the cache get method.
> Our cache statistics are 548.459.776 cache hit, 1.810.909 cache missed
> having 5.305 objects living in it.
>
> If we fail to get the object from the cache, we're still trying to get
> it from the DataStore. However we always reach the 30secs timeout.
>
> Do you guys have any suggestions? Are we missing anything? How can we
> deal with this?
>
> Thanks in advanced,
> Hernan
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Cannot sign in plugin from behind proxy

[Robert Kluin]

Perhaps this can help:
  
http://code.google.com/appengine/docs/java/tools/uploadinganapp.html#Using_an_HTTP_Proxy





On Mon, Feb 20, 2012 at 13:30, Chuck Lega  wrote:
> Also, I am running ubuntu.
>
> On 20 Feb, 17:21, Chuck Lega  wrote:
>> I have installed the google plugin for eclipse (Google Plugin for
>> Eclipse 3.7     2.5.1.v201201120043-rel-r37).
>> When clicking "Sign in to google" I end up with an empty window,
>> finally saying "Unable to load page...".
>> Problem is probably that I am behind a proxy that wont let the https
>> request through. But there is no way of setting the proxy (and usr/
>> pwd). The plugin does not seem to reuse the eclipse-network-settings
>> (which works fine).
>> Does anyone know of a workaround for this?
>> /CL
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Python 2.7: Instance memory limitations with concurrent requests

[Robert Kluin]

Hey Andrin,
  If you hit a hard limit it will be killed immediately, no exception,
no nothing, just dead.  If you hit a soft limit it will be killed
after the request finishes.  You'll have no way within the app to know
that either case happened.

  I view this as a very serious (security) problem within tasks, for
some more info:
http://code.google.com/p/googleappengine/issues/detail?id=6616

  Also, please star that.  ;)


Robert




On Mon, Feb 20, 2012 at 05:25, Andrin von Rechenberg  wrote:
> I guess that's the same solution as just deploying two different
> versions. A threadsafe one and a non threadsafe one. Or did
> I misunderstand you?
>
> My main concern is having two versions running.
>
> @Brandon: Do you know if an instance is killed during handling requests or
> afterwards?
> And what happens with if there are concurrent requests?
>
>
> Cheers,
> -Andrin
>
>
> On Mon, Feb 20, 2012 at 11:20 AM, Johan Euphrosine 
> wrote:
>>
>> Hi Andrin,
>>
>> Have you considered using a dynamic B2 backend?
>>
>> If the request originate from a form you can even address the backend by
>> name directly with:
>>
>>     backendname.appid.appspot.com
>>
>> Hope that helps.
>>
>> On Mon, Feb 20, 2012 at 11:15 AM, Andrin von Rechenberg
>>  wrote:
>>>
>>> Hey there
>>>
>>> I'm planning to migrate to Python 2.7.
>>>
>>> I'm now running on Python2.5. There is one kind of request
>>> (lets call it people-search) that uses A LOT of ram.
>>> From time to time instances get killed because they exceed
>>> the allowed memory allocation. These requests do a lot of
>>> RPCs. One people search uses about 100mb of ram.
>>>
>>> In Python 2.7 I would guess that one instance will handle
>>> many people-searches at the same time, because they
>>> mostly wait for RPCs. That means that an instance
>>> will go out of memory much more often, because every
>>> pending request will use 100mb. You handle 2 at the same
>>> time, an F1 instance dies. I'm afraid that I will end up in a
>>> crash loop pretty soon.
>>>
>>> Am I right in my assumption?
>>>
>>> Is there a way to limit the amount of people-searches
>>> that one instance handles at the same time?
>>> Like threadsafe:false for a certain handler?
>>>
>>> The only solution I see at the moment is to deploy 2 versions
>>> of my app. The main version with threadsafe enabled and
>>> a special version with threadsafe disabled.
>>> The main app would then not handle people-searches directly,
>>> but do a urlfetch to the threadsafe disabled version for
>>> people-searches. This way one instance would always
>>> only do one people-search because threadsafe is disabled.
>>>
>>> But it's s ugly from a maintenance point of view.
>>> Really ugly.
>>>
>>> Any better suggestions?
>>>
>>> Cheers,
>>> -Andrin
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine+unsubscr...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>>
>>
>> --
>> Johan Euphrosine (proppy)
>> Developer Programs Engineer
>> Google Developer Relations
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: shell.appspot.com version with Python2.7

[Robert Kluin]

Awesome.  I use that constantly as well.






On Mon, Feb 20, 2012 at 08:17, Andrin von Rechenberg  wrote:
> Ryan, maybe you could add a link from shell.appspot.com to
> shellpy27.appspot.com just like the later does.
>
> Cheers,
> -Andrin
>
>
> On Mon, Feb 20, 2012 at 1:47 PM, Andrin von Rechenberg 
> wrote:
>>
>> Hey there
>>
>> I frequently use shell.appspot.com
>> A lovely python shell to execute google appengine code.
>>
>> I now created an exact copy that runs Python2.7 here:
>>
>> http://shellpy27.appspot.com/
>>
>> Makes it easier if you want to try out a couple of things
>> that are new in py27.
>>
>> Cheers,
>> -Andrin
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: 1.6.3 Prerelease SDK Available

[Robert Kluin]

These are some of the best features in recent releases:
  - Datastore statistics now show the amount of storage used by application
indexes.
  - The instances screen in the Admin Console now has a button to shut down
a specific instance.
  - Each application log has a link to the instance that served the request. If
the instance is no longer serving requests, a message will be displayed.
  - The TaskQueue API now supports the ability to tag a Pull Queue task. You may
then lease tasks by tag.

Really looking forward to this going live!







On Mon, Feb 20, 2012 at 21:04, Jeff Schnitzer  wrote:
> See thread subject.
>
> Prerelease != Release
>
> Jeff
>
>
> On Mon, Feb 20, 2012 at 2:14 PM, GAE  wrote:
>>
>>
>> I just deployed a java pp. However I can't see that shutdown button on the
>> Instances page in the Admin Console. Where is it suppoed to be?
>>
>>
>> On Monday, February 20, 2012 7:03:01 PM UTC+1, Marce (Google) wrote:
>>>
>>> - The instances screen in the Admin Console now has a button to shut down
>>>   a specific instance.
>>> - Each application log has a link to the instance that served the
>>> request. If
>>>   the instance is no longer serving requests, a message will be
>>> displayed.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msg/google-appengine/-/z53DnK1Uef8J.
>>
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Error 'Python int too large to convert to C long'

[Robert Kluin]

Hi,
  You might try to provide some additional details.  You're using
Python 2.7, maybe snippets from your config, details on how to
reproduce, etc


Robert





On Fri, Feb 17, 2012 at 16:10, googlefox  wrote:
> I am running PIL inside of Googleappengine. When I run my app in
> dev_appserver, it runs fine. However, when I run it inside of GAE, I
> see requests fail with:
>
> ---
> 2012-02-17 13:04:23.960 
> E 2012-02-17 13:04:23.960 ('Python int too large to convert to C
> long',)
> E 2012-02-17 13:04:23.960 Python int too large to convert to C long
> E 2012-02-17 13:04:23.960
> ---
>
> I am not getting a full stack trace, which makes it difficult to know
> why it is failing. On a GS thread, someone from google told me to use
> the new build of webapp2 and include that with my app to get a full
> stack. However, that didnt work either (to get a full stack).
>
> So, can someone from GAE help me figure out why my app is failing with
> an overflow error?
>
> All I am trying to do is to convert an input image into different
> sizes.
>
> Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: URGENT - Accidental Billing Cancelation

[Robert Kluin]

So did they get this fixed for you?






On Sun, Feb 19, 2012 at 07:32, DanMux  wrote:
> Hi Amy.
>
> Thanks for your reply, I filled that in as soon as I found it on
> another thread.
>
> Have had no reply, or to our billing email - or the other form.
>
> This is very very very painful
>
> And such a simple stupid mistake
>
> Users are already complaining, we have removed our apps from the
> Market and iTunes, we look like, and might as well be, complete
> amateurs.
>
> I can't fully express the anger I feel, with myself for making such a
> simple mistake, but worse that it can NOT be undone!
>
> Our action plan in this scenario. sit around and pray to be
> bestowed some attention. We cant even copy data to another instance,
> taking the 30 minutes read only hit, as that too is part of the quota!
>
> Pass me the whiskey, and stir in a bottle of pain killers
>
> Yours in sincere anticipation.
> Dan.
>
>
>
> On Feb 19, 4:44 am, Amy Unruh  wrote:
>> Dan,
>>
>> Start with this, the form for billing 
>> issues:http://code.google.com/support/bin/request.py?contact_type=AppEngineB...
>>
>>
>>
>>
>>
>>
>>
>> On Sun, Feb 19, 2012 at 8:39 AM, DanMux  wrote:
>> > Google staffers PLEASE help urgently.
>>
>> > I accidentally disabled billing on the new app after a hrd migration.
>>
>> > I'm about to hit the free quota.
>>
>> > I just need the cancellation to stop
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] ThreadManager

[Robert Kluin]

It isn't clear if you're talking about Python or Java.  If you're
talking about Python:
  
http://code.google.com/appengine/docs/python/python27/newin27.html#Multithreading

For Java, this is probably in relation to:
  
http://code.google.com/appengine/docs/java/config/appconfig.html#Using_Concurrent_Requests


Robert






On Sat, Feb 18, 2012 at 06:10, Ajax  wrote:
> I was digging around the appengine source, and I came across
> ThreadManager, which is a utility to allow users to create their own
> threads during the lifetime of a request.  I am very interested in
> this feature, as it will enable the asynchronizing of my entire
> service layer for maximum possible performance.
>
> When I use the feature locally, it works fine, but in production, it
> throws FeatureNotAvailable exception, which was expected.  Is there a
> form where we can sign up for trusted tester on this feature?  Should
> I simply ask my project manager to email our appengine support
> contact?
>
> I want to redesign our GData libs to support Future and
> Future, using asynchronous requests.  As I am unable to
> replicate the OAuth signing with UrlFetchAsync, I was hoping
> ThreadManager would allow me to simply wrap the existing GData libs
> with Futures that run synchronously in a background thread, and simply
> block on calls to .get() until the feed or entry is returned.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Exporting entities from GAE to local dev-server

[Robert Kluin]

Hi,
  On the local dev server keys are prefixed with dev~, if I recall
correctly.  With the Python dev_appserver this can be changed with the
"--default_partition" flag.  Perhaps this is your issue.

  Alternatively you might consider recasting all of your keys as you
import the data.


Robert





On Sat, Feb 18, 2012 at 13:11, Mos  wrote:
> Hello,
>
> I'm wondering if it is possible to export Entities from the GAE BigTable
> into the local development server.
> It should be a usual use-case for all people how like to test locally with
> production data.
>
> I'm doing the export with serialized Json. The GAE keys are serialized and
> unserialized via KeyFactory.stringToKey methods.
>
> Hence the export looks like:
> {"key":"ag1zfmtyaXNlbi10YWxrcg8LEgdBcnRpY2xlGMnfAgw","createDate":1329587740394,
> .
>
> The import works fine if the export was done on the same system:
> Export  from "Local Development"  and import to "Local Development"
> Export  from "GAE"  and import to "GAE"
>
> But from "GAE" to "Local Dev" and vice versa it doesn't work.
> Import runs, no error messages, but the entities are not visible.
>
> I think it has something to do with the namespaces.
> I'm using no explicit namespace.
>
> Are there different default namespace between dev-server and GAE
> environment?
>
> How can I import Entities from the GAE into the local dev-server?
>
> Thanks and Cheers
> Mos
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] For information about how to identify the cause of the error to start up the backend

[Robert Kluin]

Hi,
  Have you tried to define a warmup handler?  You just need something
to handle that request.

http://code.google.com/appengine/docs/python/config/appconfig.html#Warmup_Requests



Robert





On Thu, Feb 16, 2012 at 21:26, linkseed  wrote:
> / 301 is returned in the _ah / start, seems to fail to start.
>
> In such a case, what do I do to identify the cause in any way.
>
> - Once the back-end, Stop -> Shutdown -> Start.
> - Dynamic Instance type of a back-end - change to> Resident.
> - Update the application again on the back end.
> - Set new alias the back-end.
> - Purge of the task that was piled up in the back end.
>  (On the back end is processing a task only)
>
> But I went like, however, it does not improve.
>
> Application configuration is as follows.
> In addition, I have attached a screenshot of the management screen.
>
> application: hoge
> version: 1
> runtime: python27
> api_version: 1
> threadsafe: true
>
> inbound_services:
> - warmup
>
> backends:
> - name: backends
>   class: B1
>   options: dynamic
>   instances: 1
>
> Professor Helps to enjoy later.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/roiwXoBiO7sJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Question about datastore batch get and appstats

[Robert Kluin]

Hey Francois,
  You are correct.  This was changed several releases ago.  When you do a
batch fetch the keys will be grouped by entity group, then an async RPC
done for each one.  This should perform better in most cases.  There are
parameters to adjust if you really don't want it to do this... but you
almost certainly do want this.


Robert






On Fri, Feb 17, 2012 at 09:26, Francois Masurel  wrote:

> Here is what I get when I display the detail of one datastore get in
> appstats :
>
>   com.google.appengine.tools.appstats.Recorder:290 makeAsyncCall()  
> com.google.apphosting.api.ApiProxy:184
> makeAsyncCall()  com.google.appengine.api.datastore.DatastoreApiHelper:59
> makeAsyncCall()  
> com.google.appengine.api.datastore.AsyncDatastoreServiceImpl:332
> doBatchGetBySize()  
> com.google.appengine.api.datastore.AsyncDatastoreServiceImpl:381
> *doBatchGetByEntityGroups()*  
> com.google.appengine.api.datastore.AsyncDatastoreServiceImpl:275
> get()  com.google.appengine.api.datastore.DatastoreServiceImpl$1:78
> runInternal()  com.google.appengine.api.datastore.DatastoreServiceImpl$1:75
> runInternal()  com.google.appengine.api.datastore.TransactionRunner:31
> runInTransaction()  com.google.appengine.api.datastore.DatastoreServiceImpl:75
> get()
>
> Does this means that batch get for entities in different entity groups is
> like doing multiple gets ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/ISwF-pZkxUUJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] 2012 US PyCon -- GAE Related Sprints

[Robert Kluin]

Hey Everyone,
  Myself and several colleagues are going to be at pycon, and sticking
around for a few days of sprints after the main conference is over.
We're planning to discuss and work on several App Engine related
projects during that time.  Some of the projects are pretty neat, and
should benefit the larger App Engine Python community.  We'd also love
to have some help or more ideas for cool things to work on!  If you're
going to be at PyCon and sticking around, feel free to sign up or just
catch one of us at the conference.  Feel free to stop by even if you'd
just like to nerd-out about App Engine / Python!


  Some current items on the project list:
- An ext.db mock for "unit" tests that offers basic functionality
  (including querying!) without the poor performance of the SDK stub.
- Work on PyAMF and RTMPy.
- Since we'll soon have pull-queue tagging, I want to get out the
  updates I've been holding back for slagg.
- I'd like to put together a Cassandra stub for the SDK.  I made
  good progress on this last year, but it has fallen behind.


  I've added our "spint" to the pycon sprints page, see "APP ENGINE
RELATED PROJECTS (COMMUNITY RUN)"
https://us.pycon.org/2012/community/sprints/projects/



Robert

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Undocumented quota: "Log Records Received"

[Robert Kluin]

It is not a hard limit.  I suspect they will let you pay for more once
it is out of experimental status.

I've found the log service to be a little slow when fetching a lot of
records in a row.  In my tests it can take 15 minutes of looping (max
batch size) to fetch 100 records.  So if you're using this on an
app with a lot of requests you probably want to scan "often."  (ie
don't pull once a day on an app with 2000 requests per day.)


Robert




On Thu, Feb 16, 2012 at 04:29, Andrin von Rechenberg  wrote:
> @Google: Is this a hard limit that will stay around? Can it be increased?
>
> I'm planning to implement prodeagle using the logservice instead of
> memcache.
> But 100 reads per day is just too low if you have more than 100
> queries
> per day. Are there any other alternatives?
>
> Cheers,
> -Andrin
>
>
> On Tue, Jan 24, 2012 at 8:34 PM, Robert Kluin 
> wrote:
>>
>> You may want to mention why it "all comes to nothing."
>>
>>
>>
>> Robert
>>
>>
>>
>>
>> On Tue, Jan 24, 2012 at 09:25, 333...@gmail.com <333...@gmail.com> wrote:
>> > I've spent two days working on a feature based on the LogService, it
>> > seems it
>> > all comes to nothing :\
>> >
>> > You could at least leave a small note on the documentation related to
>> > it.
>> > here for example:
>> > http://code.google.com/appengine/docs/python/logservice/
>> >
>> > Do you plan to make it billable?
>> >
>> > Daniel C. <http://theosp.me/>
>> >
>> > On Jan 19, 12:08 pm, Alexis  wrote:
>> >> Hi
>> >>
>> >> We noticed a few days ago that starting from the middle-end of the
>> >> day, a new quota line now appears in one of our app dashboard:
>> >>  "Log Records Received  -  95%  - 946,267 of 1,000,000"
>> >>
>> >> This line is not there in other apps or at the beginning of the day.
>> >> So it must appear only because we are going to reach the quota limit.
>> >> That's fine, but this quota does not appear in the "Quota details"
>> >> section of the dashboard, and is not mentioned in the quotas
>> >> documentation neither in the LogService documentation. In other words
>> >> I have no clue of what is it and could not anticipate it.
>> >>
>> >> We only have an average 15 requests/sec so it's more than 1 million
>> >> requests a day, the quota can't be that we can't log more that 1M
>> >> requests right?
>> >>
>> >> I think it may be linked to the new LogService API that we are
>> >> starting to use. Is it because we read too many records using this
>> >> API?
>> >>
>> >> Can someone shed some light on this? Did I miss something?
>> >> Thanks
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>> >
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Spiky pattern on QPS charts since 9:00pm EST

[Robert Kluin]

I *think* the scheduler might be killing instances more aggressively
in the past day or so.  I've seen and heard about some pretty strange
behavior in the past day or so.

You're graphs seem strange, especially the qps graph.  I saw someone
else posting similar graphs a week or two ago.  I never heard about a
resolution.


Robert



On Thu, Feb 16, 2012 at 09:20, nmelo  wrote:
> Hey everyone,
>
> Since yesterday 9:00pm EST we've been noticing "spiky" patterns on various
> charts (QPS, Instance Count, Bytes received) where requests go to 0QPS and
> then shoot back up to where they were every minute or so. They used to be
> very predictable and flat. I've attached a few screen grabs.
>
> Our app still works, there are no errors in the log, we haven't deployed to
> that app in more than a week, and there are no complains from the users so
> far.
>
> Anyone else seen this/has an explanation. Is something we should be
> concerned about?
>
> Thanks!
>
> nmelo
> =
> @nmelo
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/sivWwwUBOpYJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Authentication on a GAE App only for a selected Google Apps domains list

[Robert Kluin]

Hey Antoine,
  Sounds like you've got an issue with how the continue url is
encoded, but that's a total guess.  Does this happen every "first"
time, when the user is not logged in?




Robert




On Fri, Feb 17, 2012 at 01:00, Antoine  wrote:
> Hello,
>
> I succeeded to make it work. I didnt ask for "login:required" because
> i just need one webpage to be secured (one handler for this page which
> asks for openid authentication before showing content).
>
> But now, I have a problem with the authentication and redirection.
>
> Use case :
> - user go on the Google App Engine Application homepage (http://
> example.appspot.com), when he clicks to go on the second webpage
> (secured one)
> - he is asked to choose his Google Apps domain (his provider) by
> clicking on a link
> - then he is redirected to the the login url formatted thanks to the
> providers chosen (ie : www.google.com/accounts/o8/site-xrds?hd=example.com)
> - he is redirected to google which will redirect him to the SSO
> company page (because the Google Apps domains use SSO for the whole
> company users, and for all domains)
> - he authenticate himself on the SSO company page
> - he should be redirected to the Google app engine secured page after
> success
> however, he gets a Google webpage with "The page you requested is
> invalid"
>
> Two things to know also :
> if after that, he re-do everything : go to the homepage, click on the
> secured page link, then click on the provider, he will access the
> secured page automatically (authentication has been successful before,
> even if the redirection shows a Google page with "The page you
> requested is invalid")
> if he opens another tab with Gmail, then no need to authenticate,
> Gmail shows the emails inbox (authentication has been successul
> before...)
>
>
> So I look at this problem and I've found this :
> http://www.google.com/support/forum/p/apps-apis/thread?tid=39a0dedd82b472ec&hl=en
>
> I tried the website : http://www.puffypoodles.com/lso2
>
> I get the same error at first with the Google page "The page you
> requested is invalid" after authentication on the SSO company webpage.
>
> So as it is said by the Google employee, it should be a Google
> Endpoint issue :
> "To verify that there isn't an issue with OpenID on your test domain,
> can you try logging in via OpenID using the test site
> (puffypoodles.com)?[1] If login on this site works, there must be a
> problem with the code. If it does not work, there is probably an issue
> with the Google OpenID endpoints."
>
> Where is my mistake here ?
> I dont really get the Google OpenID enpoints issue.
>
> Thanks in advance,
> Antoine
>
> On 2 fév, 00:11, Robert Kluin  wrote:
>> HiAntoine,
>>   Glad you got that figured out.
>>
>>   You should be able to have login required on your apps, just as
>> before. You'll just need to be sure to define a handler for
>> login_required (as is explained in the article).  Note that you won't
>> get redirected to it on the dev server, so you'll need to directly go
>> to the url to test it out.
>>
>> Robert
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Feb 1, 2012 at 04:42,Antoine wrote:
>> > Ok, I ve found out.
>>
>> > this should be the URL for Google Apps domains :
>>
>> >www.google.com/accounts/o8/site-xrds?hd=example.com
>>
>> > On Feb 1, 4:59 pm,Antoine wrote:
>> >> Thank you for your answer.
>>
>> >> I tried to follow this tutorial (I dont know why I didnt find it
>> >> before... :s ).
>>
>> >> However, I have a question left.
>>
>> >> I decided to let the user choose his domain by clicking on his domain
>> >> link.
>>
>> >> I deleted the "login required" in app.yaml and modified my main.py
>> >> such as...
>>
>> >> My python looks like :
>>
>> >> ---
>> >> #
>> >> #code
>> >> #
>>
>> >> providers = {
>> >>     'prov 1'   : 'google.com/a/domain.com',
>> >>     'prov 2'    : 'google.com/a/subdomain1.com',
>> >>     'prov 3'    : 'google.com/a/subdomain2.com'
>> >>     # add more here
>>
>> >> }
>>
>> >> #
>> >> #code
>> >> #
>>
>> >> else:
>> >>       self.response.out.write('Hello world! Sign in at: ')
>> >>   

Re: [google-appengine] Cron syntax questions (Java)

[Robert Kluin]

See:
  
http://code.google.com/appengine/docs/python/config/cron.html#The_Schedule_Format

If you've got specific questions, show us what you've tried and the error.



Robert





On Wed, Feb 15, 2012 at 18:10, Dai, Gary
 wrote:
> Sorry I don't get it. Could you kindly show me the cron syntax to define
> last Wednesday of each quarter?
>
>  Thanks.
>
> -Original Message-
> From: google-appengine@googlegroups.com
> [mailto:google-appengine@googlegroups.com] On Behalf Of Brandon Wirtz
> Sent: Wednesday, February 15, 2012 5:46 PM
> To: google-appengine@googlegroups.com
> Subject: RE: [google-appengine] Cron syntax questions (Java)
>
> Cron on Wednesdays in code check if it is the last one of the quarter
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] google cloud sql server error

[Robert Kluin]

Hi Mansik,
  Check your app's log in the log viewer on the appspot dashboard.
The error should be detailed there.


Robert





On Wed, Feb 15, 2012 at 12:44, Mansik Kim  wrote:
> Whenever i try to access web built on app engine, it shows me below
> message after deploying
>
> Error: Server Error
> The server encountered an error and could not complete your request.
> If the problem persists, please report your problem and mention this
> error message and the query that caused it.
>
> but if i run the program locally, i can access it. even i can see sql
> changing in Google APIs Console.
>
> I just followed an example. -  
> https://developers.google.com/cloud-sql/docs/developers_guide_java
>
> In addition to Eclips console shows me this message
> (java.security.AccessControlException: access denied
> (java.lang.RuntimePermission accessClassInPackage.sun.jdbc.odbc)
> regardless it works.
>
> I don't know what to do to solve that problem.
> Please help me.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: A question about eventual consistency/transactions

[Robert Kluin]

On Wed, Feb 15, 2012 at 15:04, Jeff Schnitzer  wrote:
> On Wed, Feb 15, 2012 at 2:45 PM, peterk  wrote:
>>
>> Cheers Jeff, that's a relief.
>>
>> As a follow-up... if I have multiple fetch/modify cycles on models
>> within an entity group in a transaction, the fetches won't always
>> necessarily show up-to-date models, right?
>>
>> E.g. within a transaction if I get modelA, update it, put it, then get
>> it later in the transaction again via another (ancestor) query, the
>> earlier modifications may not necessarily show, right?
>>
>> You're only guaranteed to be up-to-date as of the _start_ of the
>> transaction?
>
>
> The answer to this is actually a little more complicated than that.
>
> At the lowest levels of the API, this is true.  If you're using the java
> low-level api (or writing protobufs yourself) you will see the datastore
> "frozen in time" from the start of the transaction (actually, it might be
> from the moment each entity is loaded - I don't recall).  Even after
> changing an entity and saving it, another load within the same transaction
> will see the "old" version.

I believe it is a snapshot from the beginning of the transaction.


>
> But most people work with higher-level APIs.  These higher-level APIs often
> provide a session cache so that when you load the same entity in the same
> session you will get the same object... or at least won't have to go all the
> way back to the datastore.  Objectify does this.  I'm pretty sure JDO/JPA
> does this.  I don't know about Python db or ndb.

ndb has nice built in caching,  ext.db does not.



>
> Jeff
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] fetching by keys (as strings) is kind of slow

[Robert Kluin]

Hey Andreas,
  One thing to note is that PolyModels use lists in the background to
facilitate querying.  If it is deep, it could add some overhead.  Try
this with small entities, if there's a big difference you'll know it
is your data model.



Robert





On Wed, Feb 15, 2012 at 07:56, Andreas  wrote:
> the entities have an average of 10properties but no list properties.
> its a subclass of a PolyModel and there are 2 datetime properties and one 
> objectproperty which stores into the blobstore.
> i didnt try with 'simple' entities yet but i will today i think.
>
> On Feb 15, 2012, at 12:25 AM, Robert Kluin wrote:
>
>> What is the data model like?  Do the entities have a lot of
>> properties, list properties, or datetime properties?  Have you tried
>> this with a very simple entity, perhaps even with no properties?
>>
>>
>>
>> Robert
>>
>>
>>
>>
>>
>> On Tue, Feb 14, 2012 at 15:05, Andreas  wrote:
>>> im just profiling my app and i have to say i get surprising results.
>>>
>>> lets say i have a list with 500 keys as strings not key objects.
>>> db.get(keylist) takes 3seconds??!!!
>>>
>>> how does this take that long? i mean i have the keys already.
>>> would it be (significantly) faster with key objects?
>>>
>>> andreas
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "Google App Engine" group.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> To unsubscribe from this group, send email to 
>>> google-appengine+unsubscr...@googlegroups.com.
>>> For more options, visit this group at 
>>> http://groups.google.com/group/google-appengine?hl=en.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to 
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at 
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] fetching by keys (as strings) is kind of slow

[Robert Kluin]

What is the data model like?  Do the entities have a lot of
properties, list properties, or datetime properties?  Have you tried
this with a very simple entity, perhaps even with no properties?



Robert





On Tue, Feb 14, 2012 at 15:05, Andreas  wrote:
> im just profiling my app and i have to say i get surprising results.
>
> lets say i have a list with 500 keys as strings not key objects.
> db.get(keylist) takes 3seconds??!!!
>
> how does this take that long? i mean i have the keys already.
> would it be (significantly) faster with key objects?
>
> andreas
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Skip requests from cron from getting redirected to my custom domain

[Robert Kluin]

cron requests have a special header indicating that they are a cron
request,  X-AppEngine-Cron: true.

This is documented:
  
http://code.google.com/appengine/docs/python/config/cron.html#Securing_URLs_for_Cron


Robert




On Tue, Feb 14, 2012 at 07:13, praveen  wrote:
> Hi,
>
> I'm using Flask on google appengine
>
> I have a custom middleware which redirects requests from *.appspot.com
> domain to my custom domain
> I'm trying to skip requests from cron service from such redirection.
> (else they would fail)
>
> I have the following code
> http://stackoverflow.com/questions/9277356/skip-requests-from-cron-and-taskqueue-services-from-being-redirected-to-custom-d
>
> Can someone comment on why the check for "X-AppEngine-Cron" isn't
> working?
>
> Thanks
> Praveen
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Entity size and latency

[Robert Kluin]

Last time I benchmarked there was very little, if any difference,
based on the size of an entity (ie 10kb vs 900kb).

What makes a very noticeable difference is serialization.  If you've
got a 2kb entity with a 500 item list it will likely perform worse
than a 900kb entity with a single blob property -- especially if the
list is indexed.  This is why I often serialize data and stick it in a
blob/text property myself when 1) I don't need it indexed and 2) I
don't need to access it every time I access the entity.



Robert





On Tue, Feb 14, 2012 at 13:38, Daniel Florey  wrote:
> I do not have a graph but you should keep in mind that entities with large
> list properties may cause increased serialization time.
> See:
> http://youtu.be/AgaL6NGpkB8?t=14m31s
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/zNpScQ7BHaEJ.
>
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] SMS Verification Issue

[Robert Kluin]

Assuming you're actually talking about App Engine, see the FAQ:
  http://code.google.com/appengine/kb/sms.html#error





On Tue, Feb 14, 2012 at 14:25, Bashie  wrote:
> If there is no phone number that I can provide to Google, is there no way I
> can use Google Apps?
> How is a phone number related to Google Apps?
> And if according to Google's privacy policy that phone number is not going
> to be used for anything else, why can't I receive the code by any other way
> Google already has of contacting me?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/Rcf61_ncHfsJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: How long a query results takes to be updated? HR Datastore

[Robert Kluin]

You have to explicitly set eventually consistent to get that behavior.

Note that ancestor queries are also strongly consistent.  This is
detailed in the docs:
  
http://code.google.com/appengine/docs/java/datastore/hr/#Data_Storage_Options_Compared


Robert




On Tue, Feb 14, 2012 at 12:51, Leandro Rezende
 wrote:
> im confused now =(
>
> so, even  GetObjectById can return Old Data?
>
> Jeff said "If you access the datastore in ReadPolicy.Consistency.EVENTUAL
> mode...",
>
> Is this something i have to "Set"? to all access become
>  "ReadPolicy.Consistency.EVENTUAL" or its just the way im coding..
>
> Thanks again.
>
>
> 2012/2/14 Jeff Schnitzer 
>>
>> Officially, there is no guaranteed upper bound for queries to become
>> consistent.  In practice it's usually a second or two, but infrastructure
>> failures could in theory extend the period to minutes or hours or (gasp)
>> days.
>>
>> One thing worth noting:  If you access the datastore in
>> ReadPolicy.Consistency.EVENTUAL mode, get-by-id will be eventually
>> consistent just like queries.  And it's *dramatically* faster, ~1/5 latency
>> in my recent experiments.
>>
>> Jeff
>>
>>
>> On Tue, Feb 14, 2012 at 12:15 PM, Mahron  wrote:
>>>
>>> get_by_id or GetObjectbyID are supposed to always be consistent.
>>> Basically all Items returned by query or id are up to date. If it is
>>> not, there is a serious problem.
>>>
>>> "Sometimes the Query returns the value updated, sometimes not." By
>>> that I suppose it does not return the entity at all. If it returns a
>>> version different than after the last Put(), then I am pretty sure
>>> that's a bug.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine+unsubscr...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Zombie tasks eating up my mail quota

[Robert Kluin]

Hey Andrew,
  Are you sure that task was run in your email queue?  Perhaps it was
run in the default queue?

  Do your tasks run on front-ends or backends?  Just asking in case
there is a long-running processon a backend sitting there spinning and
consuming your email quota.


Robert



On Tue, Feb 14, 2012 at 09:23, Andrew Richardson
 wrote:
>
> My app uses the Task Queue to send email messages...and a few days ago, I
> must have gotten a bad task in the queue. One of the tasks has been failing
> and retrying endlessly, eating up all my email quota (I guess they count
> against your quota even with the message fails to send).
>
> Yesterday I paused my "email" queue, deleted a few tasks that looked like
> they could be causing problems, and was hoping to dispatch the rest by hand
> this morning to make sure they went out. But again, as soon as my quotas
> were reset, my app went wild and blew up all the email quota for today.
>
> In the logs I see references to retries for things like
> "X-Appengine-Queuename:email, X-Appengine-Taskname:2162362032997459024" -
> but the email queue was supposedly paused, and that task number was not even
> in the queue! It's probably one of the ones I deleted by hand - is it
> expected for this task to still be present and running? Today I purged the
> email queue and set it to a retry limit of 2...so I'm hoping that helps.
>
> Maybe I'm misunderstanding how Task Queues are supposed to work - I thought
> that pausing it would stop any pending tasks from executing until I resumed
> it? I hope someone can shed some light on this...thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/9G15Fj4GRBQJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Redirect to original href when using federated login

[Robert Kluin]

You might try posting the actual code you're using to generate the
URL, and an actual sample of the complete URL being generated.


This is probably also the type of question they're trying to get
people to ask on SO.



Robert



2012/2/14 Andrei Cosmin Fifiiţă :
> I used url encode to generate a redirectLink that looks something like this:
>
>
> http%3A%2F%2Fvers.myappid.appspot.com%2page.html%3Flocale%3Dro%23m%3myhistorytoken
>
>
> Now, when I call the createLoginURL method i get this:
>
>
> http://vers.myappid.appspot.com/_ah/login_redir?claimid=www.google.com/accounts/o8/id&continue=http://vers.myappid.appspot.com/
>
>
> So now, even the link parameters (charactes after '?') are missing from the
> continue link.
>
> Whan am I doing woring ? Maybe i need to specify some other parameters? (I
> used URLEncoder.encode(originalHref, "ISO-8859-1") in java to generate the
> redirectLink for the google openID provider) and
> userService.createLoginURL(redirectLink, null, e.getValue(), new
> HashSet())
> to get the login URL.
>
>
> On 14 February 2012 08:45, Robert Kluin  wrote:
>>
>> Maybe I'm not getting what you're saying here, but perhaps you should
>> urlencode the continue url.
>>
>>
>>
>>
>>
>>
>> On Mon, Feb 13, 2012 at 11:41, Ice13ill  wrote:
>> > I'm using federated login for my app authentication and I have the
>> > following issue: when users connect with google OpenID the auth
>> > service doesn't redirect to my original href (redirects to my domain
>> > +
>> > the url params but ignores all information after "#").
>> > To create the login link I use "www.google.com/accounts/o8/id"
>> >
>> > When I test in development mode the redirection process works fine.
>> >
>> > Also, until 12-16 months ago, I used the Google accounts API option in
>> > my app setting and I
>> > remember that after login, the redirect link back to my app contained
>> > the characters after the history token ( "#" ).
>> > Is there a workaround for this? or is something I may be doing wrong?
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>> >
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Optimizing the usage of the free quota

[Robert Kluin]

On Mon, Feb 13, 2012 at 22:49, Brandon Wirtz  wrote:
> Ads.
>
> Sell the users information.
>
> Get to critical mass, pocket the money, go out of business before the money
> runs out.

The last option is probably the best bet.


Seriously though, you're developing an app that needs a backend
service, you sell for $1, and you only hope to get 1000 users?  On one
hand I respect the realistic goal/view, but the *first* thing I'd do
is aim a little higher.  (not meant as an insult)



>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Sockets or Apple Push Notification support in Google App Engine

[Robert Kluin]

Do you have a premier support account?  If so you might want to ask
your dev advocate about this.


Robert




On Mon, Feb 13, 2012 at 13:42, Giant Fighting Robot  wrote:
> I have outgrown the 3rd party options for push notifications to Apple
> devices.  I heard rumors that Google was planning to add socket support to
> the app engine which would enable me to roll my own push notifications.  I'm
> starting to think that was Channels, which doesn't appear to do what I
> need.  My questions are:
>
> 1) To double check, does the App Engine allow for apple push notifications
> without going through a 3rd party service?  (maybe it was added in and I
> missed it?)
>
> 2) Assuming no, are there any plans to support it in the near future?  I
> don't need a date, but if it's expected in the coming months I might grin
> and bear it until support comes, or if it's completely orthogonal to GAE's
> plans, I'd love to hear that so I can plan accordingly.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/oPlc4_gMmAMJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] email sent, but not being received

[Robert Kluin]

People commonly see this when their emails are getting flagged by spam
filters.  Many users have found better deliverability with things like
Amazon SES.




On Mon, Feb 13, 2012 at 04:19, snowdown  wrote:
> I'm using google app engine to deploy apps for sending emails. It used
> to work fine, however, the emails sent out this passed weekend didn't
> seem to reach anyone(I didn't change any setting). I did some tests
> with hotmail, gmail etc, didn't receive any.
> However, google obviously charged me for the emails I sent, as seen in
> the billing history.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Redirect to original href when using federated login

[Robert Kluin]

Maybe I'm not getting what you're saying here, but perhaps you should
urlencode the continue url.






On Mon, Feb 13, 2012 at 11:41, Ice13ill  wrote:
> I'm using federated login for my app authentication and I have the
> following issue: when users connect with google OpenID the auth
> service doesn't redirect to my original href (redirects to my domain
> +
> the url params but ignores all information after "#").
> To create the login link I use "www.google.com/accounts/o8/id"
>
> When I test in development mode the redirection process works fine.
>
> Also, until 12-16 months ago, I used the Google accounts API option in
> my app setting and I
> remember that after login, the redirect link back to my app contained
> the characters after the history token ( "#" ).
> Is there a workaround for this? or is something I may be doing wrong?
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] How to handle HTTPS in development?

[Robert Kluin]

Hey Kaan,
  Have you looked at using "secure: always" in your app.yaml, rather
than redirecting yourself?

http://code.google.com/appengine/docs/python/config/appconfig.html#Secure_URLs




Robert





On Sun, Feb 12, 2012 at 21:56, Kaan Soral  wrote:
> HTTP is great, easy to test on SDK, but https is a pain, SDK doesn't support
> two IP's.
>
> I am thinking of building an app that will only use HTTPS, how should I
> proceed?
>
> My current solution is to check if we are on Appengine in code, and if we
> are on Appengine and HTTP, redirect to HTTPS, otherwise SDK would run on
> HTTP
> Very inelegant but it would work
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/MrPPmha8htkJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] What I did wrong? 2000 instances!

[Robert Kluin]

Hi Ronaldo,
  I'd probably go with either Barry or Brandon's suggestion to lower the
max concurrency or disable the process while you're debugging.

  Do you know what is causing the latency to slowly increase?  If it is due
to loading the remote servers, have you investigated if "batching" will
help reduce the load?  Or is it on the App Engine side, perhaps your using
offsets to get to the next entity to process?



Robert




2012/2/12 Ronoaldo José de Lana Pereira 

> Some inputs:
>
>
> Latency:
>
>
> 
> Req/Sec:
>
>
> 
> Instances:
>
>
> 
>
> Our app usualy, with normal traffic, needs about 40 - 60 instances to ru

Re: [google-appengine] Youtube API Logging

[Robert Kluin]

What happens when you open the logs page, then set the level to error?

Do you have exception handling in your code that's masking the exceptions?


Robert




On Sun, Feb 12, 2012 at 06:59, facanferff  wrote:
> Hi, I'm the developer of a plugin supporting Youtube for Showtime (by
> Andreas Oman), in my dashboard I see numbers about the errors,
> uploads, etc. but I can't find a section to see the errors log that
> happened to my users...
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Mail quota Problem

[Robert Kluin]

They've been having a number of issues related to email quotas / usage
display.  You might file an issue with your appid so they can take a
look.


Robert




On Sat, Feb 11, 2012 at 08:35, partnerke...@hotmail.com
 wrote:
> Hello
> Recipients Emailed %100 fills up quickly.
> Estimated cost for the last 6 hours:$0.00* / $150.00 active. But it
> does not send.
>
> The problem is what do you think?
>
> Thank You
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: is multiprocessing or pprocess supported by gae?

[Robert Kluin]

Hi,
  Have you considered using taskqueues?  That will let you spread the
load across multiple instances of your app.
http://code.google.com/appengine/docs/python/taskqueue/

  Python 2.7 supports some threading, but, of course, if you're CPU
bound that won't help in Python (and may actually make it worse).

http://code.google.com/appengine/docs/python/python27/newin27.html#Multithreading



Robert




2012/2/10 saintthor :
> my app takes a lot of cpu time. threading takes more.
>
> On 2月11日, 上午1时57分, Ricardo Bánffy  wrote:
>> If you explain why you need them, maybe someone will be able to help you. I
>> suspect App Engine doesn't support them, but there are other ways to
>> accomplish what they do.
>>
>> On Fri, Feb 10, 2012 at 2:22 PM, saintthor  wrote:
>> > i need them.
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "Google App Engine" group.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> >http://groups.google.com/group/google-appengine?hl=en.
>>
>> --
>> Ricardo Bánffyhttp://www.dieblinkenlights.comhttp://twitter.com/rbanffy
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Indexes

[Robert Kluin]

I also dislike the current workflow.  At the minimum it could easily
be made less painful for the users.  I submitted a ticket with one
proposed improvement.

  http://code.google.com/p/googleappengine/issues/detail?id=6573


Robert




On Fri, Feb 10, 2012 at 20:58, Mike Wesner  wrote:
> Would be great to hear from the App Engine Team about this.  It seems
> like one area that has never had any attention.
>
> Work could be done to:
>
> 1. Speed this process up or at least inform the user of the ETA
> 2. Explain why an index goes into error, or better yet just never let
> them be in error
> 3. Allow the user to fix the index by clicking on something if its in
> error (remove and re-index button?)
>
>
> -mike
>
> On Feb 10, 4:25 pm, Barry Hunter  wrote:
>> If you mean a custom composit index (ie via index.yaml) - then I
>> believe there is a sequential processing queue.
>>
>> The delay is as much about how many tasks (and their size!) are in the
>> queue, rather than the size of your new index.
>>
>> Unless its stuck in some sort of error state - check your logs. In
>> which case might need to do some vacuuming
>>
>> On Fri, Feb 10, 2012 at 6:30 PM, Christian Goudreau
>>
>>
>>
>>
>>
>>
>>
>>  wrote:
>> > Hi Everyone,
>>
>> > Why my indexes takes forever to build while there's only two object? And 
>> > two
>> > field?
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "Google App Engine" group.
>> > To view this discussion on the web visit
>> >https://groups.google.com/d/msg/google-appengine/-/uQ7fzt6wy6IJ.
>> > To post to this group, send email to google-appengine@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Help! We can't pay for App Engine and our apps failing as a result!

[Robert Kluin]

Hi Derek,
  Did you get this corrected?  Have you tried in a different browser?

  If you're still down, you might try filing a billing issue -- but
those often take quite some time.  You might have better luck filing a
prod issue and hoping someone can help you.


http://code.google.com/support/bin/request.py?contact_type=AppEngineBillingSupport

http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue



Robert



On Fri, Feb 10, 2012 at 16:26, Derek Shanahan  wrote:
> We're running into memory issues, and on the Free billing plan. We are
> TRYING to put our CC into Google Checkout to set up a daily limit on
> AppEngine and the whole thing just seems broken.
>
> We put our info in, no Terms of Service is showing, we click to send
> payment to appengine, then nothing happens.
>
> Our users are complaining, please help!
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Task Queue Quota Errors, but I have enough quota

[Robert Kluin]

Hey Brandon,
  No offense, but there's almost nothing correct in this.  I'm also
not sure where some of the terms come from -- I couldn't find
references to them in the Python or Java docs.

  Tasks will basically be run as fast as your settings allow.  There
are three main settings: rate, bucket_size, and
max_concurrent_requests.  The bucket size controls the "token bucket"
behavior.  Rate controls the frequency at which tasks may be selected
from the queue.  The concurrent request limit gives you more fine
grained control of a queue (handy with the new pricing).

  The only limit to how many tasks you can have, that I'm aware of, is
the stored task count/bytes quotas, and possibly the api calls quota.

  Tasks shouldn't ever be lost.  If an app is actually losing tasks
that is a bug in either the app's code or app engine's.

  I'm not sure what timeout you're referring too.  You can specify a
delay before a task will be run (eta or countdown params), and for
pull queues a lease durration.


Robert



On Fri, Feb 10, 2012 at 15:52, Brandon Wirtz  wrote:
>> The bucket and rates are for controlling execution.  They don't limit
> adding
>> tasks to a queue.
>
> Doesn't the Token bucket?
>
> Also there is a max QueSize in MB you can specify in the YAML.  I don't know
> what the limit is.
>
>
> My rough understanding was that
>
> Token Rate was how many task/s you could add.
>
> Process rate was how fast tasks would be processed.
>
> Queue Size was number of tasks that could be pending.
>
> Time Out was the time to expire
>
>
> If your token rate is 5 per S
> And your process is 2 per
> And your Queue size is  5000
> And your time out is 75 minutes
>
> If 18 people wanted to make 10 new tasks in 3 seconds.  The Token bucket
> would empty and only 15 tasks would be created.
>
> If 500 people an hour wanted to create 20 tasks each (1 tasks, 3600
> seconds) yields 2.78 Tasks/s the Token bucket would not go dry. The Process
> bucket would spin up instances enough to process the tasks, and it would try
> to space them out such that it would take 5000 seconds
> But you would lose some of those task because the Task Queue would not have
> only processed all of the tasks before it hit the 5000 task limit (7200-ish
> would complete)
> And because you are limiting tasks to 75 minutes, but it would take 1 hour
> 23 minutes to process the task a portion would get dropped.
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Reduced email quota for billed application

[Robert Kluin]

Hi Piotr,
  Google said yesterday that they were running a job to correct this
issue.  I'd suggest you give it a little more time, then if it isn't
fixed file an issue.


http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue


Robert




On Fri, Feb 10, 2012 at 09:33, Piotr Duda  wrote:
> Hi,
>
> Although I have enabled billing for my application months ago, the
> email quota for it has not been raised. This is major issue for the
> service I'm running since we notify users about different sorts of
> events which occur in the system.  Could anyone from the App Engine
> team sort this out ?
>
> Since I don't want to expose my app_id could you please provide me
> with a way to report this issue in a more private manner ?
>
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to 
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/google-appengine?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Google App Engine community support is moving to Stack Overflow

[Robert Kluin]

Awesome!  You stole what was going to be an evening project this weekend.  ;)

Looking forwards to checking this out.


Robert



On Fri, Feb 10, 2012 at 10:46, Andrin von Rechenberg  wrote:
> Hey guys
>
> Short version:
> Problem solved, just subscribe
> to: google-appengine-stackoverf...@googlegroups.com
> You will receive every new question & answer in your favorite email client.
>
> Long version:
> I like stackoverflow when I'm googling, but I prefer gmail in my workflow
> for being part of the community.
>
> So I spent the afternoon hacking a tool together that crawls the rss feeds
> of StackOverflow every 5 minutes and checks whats new and mails it out.
> This could be done for any StackOverflow tag, I built a generic tool:
> http://stack-over-mail.appspot.com
>
> Of course it runs on AppEngine.
>
> It's running on free quota, so until someone sponsors it, please dont
> subscribe to super heavy tags like "java"...
>
> I'll open-source the source code soonish.
>
> Cheers,
> -Andrin
>
>
> On Fri, Feb 10, 2012 at 12:14 PM, MLTrim 
> wrote:
>>
>> I'm a Stack Overflow user and I really appreciate this move; thank you
>> for this.
>> For the reluctant of this change, fiddle with Stack Overflow for a
>> couple of weeks and you'll be addicted.
>> Michele
>> [Systempuntoout]
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Understanding Data Writes

[Robert Kluin]

Hey Richard,
  I think a factor could well be the small number of entities you're
removing.  I'm sure there is a good bit of overhead for that small a
batch, but I don't have an exact figure for you (without looking over
their code or doing some tests).

  For 1300 entities, I would try doing:
db.delete(ThatKind.all(keys_only=True).fetch(2000)).  That should take
only a second to run.  If you don't want to bother with uploading
code, set the datastore viewer to show 200 entities and manually
remove them (there's a select all button).  Frankly by doing just a
little more than "db.delete(ThatKind.all(keys_only=True).fetch(500))"
inside a loop within a task you can very rapidly delete quite a lot of
data.

  cursor = None
  while True:
query = ThatKind.all(keys_only=True)
if cursor:
  query.with_cursor(cursor)
keys = query.fetch(500)
cursor = query.cursor()
db.delete(keys)
if len(keys) < 500:
  break

  So, the first thing to note is that I've never been able to
successfully estimate the space exactly.  I can usually get close by
using the numbers in the article, then adding somewhere between a 15%
and 30% "fuzz" factor.  I assume that is the pb overhead, etc
Your estimate will get you in the ballpark though.  The basic idea is
entity count *  (len(your appid) + len(namespace) + len(kind name) +
len(property name) + len(avg property value) + len(avg str(entity
key))).  There's also whatever field separators they're using, so
probably another 5 or so bytes.  As I said, I've got spreadsheets I
use to estimate all this, but it isn't exact.


Robert



On Thu, Feb 9, 2012 at 04:41, Richard Arrano  wrote:
> Hi Robert,
> Thanks for the quick response! I gathered that the admin would be
> incurring some overhead, but does it seem reasonable that it could
> account for my estimate having been off by nearly a factor of 10? It
> seems like in that case, it would be far cheaper to just write a
> custom entity delete task that does a keys only query and calls
> db.delete on them.
>
> Thanks for the article, I think I understand. To clarify, in the
> example I mentioned, would adding a non-composite EntitiesByProperty
> ASC type index then require 12,000 * (size of EntitiesByProperty ASC
> i.e. 3 str + key + property value) on top of the original storage
> space?
>
> Thanks,
> Rick
>
> On Feb 8, 11:18 pm, Robert Kluin  wrote:
>> Hi Richard,
>>   The datastore admin also incurs some overhead.  At the minimum it
>> will be querying to find the keys that need deleted, so you'll have
>> *at least* 1 additional small operation per entity deleted.  In
>> addition you'll have overhead from the queries, and the shards getting
>> / writing their status entities -- so several more datastore
>> operations per task that was run.  All those add up pretty fast.
>>
>>   You can see how many write operations an entity needs in the SDK's
>> datastore view.  There's not a really good way to easily determine the
>> storage used by an index, I've got a feature request to at least
>> provide details on what existing indexes are using:
>>  http://code.google.com/p/googleappengine/issues/detail?id=2740
>>
>>   There's also an article where this is discussed, note that the
>> article is a little out of date.  It doesn't account for namespaces,
>> so you'll need to factor those in if you're using them.  I usually do
>> my estimates in a spreadsheet.
>>    http://code.google.com/appengine/articles/storage_breakdown.html
>>
>> Robert
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Feb 9, 2012 at 00:44, Richard Arrano  wrote:
>> > Hello,
>> > I'm having some trouble understanding the billing figures for when I
>> > perform data writes. I had 1300 entities with 1 property indexed and I
>> > kicked off a job via the Datastore Admin to delete them all. Given
>> > that:
>>
>> > Entity Delete (per entity)      2 Writes + 2 Writes per indexed property
>> > value + 1 Write per composite index value
>>
>> > It seems like in that case, for each entity there would be 2 writes +
>> > 2 writes for the 1 indexed property = 4 writes per entity, so 4 * 1300
>> > = 5200 writes used, or ~10% of the daily quota for writes consumed.
>> > However, within seconds, the data was indeed deleted but 100% of my
>> > quota had been consumed(0% had been consumed prior). How was I somehow
>> > off by a factor of 10?
>>
>> > On a related note, is there any tool out there for estimating how much
>> > space will be consumed by a certain object? I.e. I have a huge am

Re: [google-appengine] check number of users (sessions) using my sight

[Robert Kluin]

Unless those are actively cleaned up, the count probably won't be meaningful.



Robert






On Thu, Feb 9, 2012 at 19:53, Ikai Lan (Google)  wrote:
> You should be able to look at the number of sessions in the datastore (this
> uses the _ah_session Kind), but I don't know how useful this information
> will be for you. What information are you trying to find?
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com
>
>
>
> On Thu, Feb 9, 2012 at 11:54 AM, tubin gen  wrote:
>>
>> Is there a way to check number of users or httpsession  using my site
>> through admin console ?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.