date:20111206



And exactly which front-end instance class have we all currently been using? I 
hope the lowest one-- I don't want to suffer a performance hit and increased 
expenses.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine-java/-/y0mS6iOJyVkJ.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] unable to deploy applications to appengine

2011-12-06 Thread Kesava Neeli

Hi,

We are seeing 500 Internal Server errors on appengine while deploying 
applications. Anyone seeing similar issues? This is at 6:30 PM PST

Thanks
Neeli 

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine-java/-/DZyb4ZwqizAJ.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

[appengine-java] Re: unable to deploy applications to appengine

2011-12-06 Thread Kesava Neeli

Worked fine at 10:00 PM PST

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine-java/-/KDRyiFN60H4J.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [appengine-java] An alternative to Objectify?

2011-12-06 Thread Raphael André Bauer

On Tue, Dec 6, 2011 at 10:20 PM, David Chandler drfibona...@google.com wrote:
 Objectify works great with RequestFactory:

 http://code.google.com/p/listwidget/

 Do you have a specific concern about it?

Btw - JPA2 and Objectify are similar in terms of technology. So you
can always use RequestFactory and Objectify or RequestFactory and
JPA2.

Objectify does a great job providing a great persistence layer
exclusively for App Engine. JPA2 is more general and will work (in
theory) for other Datastores, too.

Best,


Raphael


 /dmc


 On Tue, Dec 6, 2011 at 2:51 AM, Alexander Orlov alexander.or...@loxal.net
 wrote:

 Is there a good/better alternative to Objectify? I want to use a Datastore
 abstraction layer like JPA2 together with GWT's RequestFactory using its
 ServiceLocator pattern. Is the usage of Objectify even advisable in this
 case or should I use the Datastore directly (which I've done 2y ago... and
 it wasn't that scary).

 -Alex

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine-java/-/VwaHVkYLKNMJ.
 To post to this group, send email to
 google-appengine-java@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.




 --
 David Chandler
 Developer Programs Engineer
 w: http://code.google.com/
 b: http://turbomanage.wordpress.com/
 t: @googledevtools

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine for Java group.
 To post to this group, send email to google-appengine-java@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine-java+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine-java?hl=en.



-- 
inc: http://ars-machina.raphaelbauer.com
tech: http://ars-codia.raphaelbauer.com
web: http://raphaelbauer.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine-java?hl=en.

Re: [google-appengine] Re: 1.6.1 Pre-release SDKs are live

2011-12-06 Thread Timofey Koolin

Will OCR work with russian language?

2011/12/6 Brian Quinlan bquin...@google.com:
 On Tue, Dec 6, 2011 at 6:05 PM, Westmark fredrik.westm...@gmail.com wrote:
 I would be happy if you could specify the versions of included third-
 party libraries, e.g. ndb and webapp2.

 ndb is being integrated into the google namespace so it isn't really
 third-party anymore. You can specify the version of other
 third-party libraries.

 See:
 http://code.google.com/appengine/docs/python/python27/using27.html#Configuring_Libraries

 Cheers,
 Brian

 Cheers / Fredrik

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.


 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.




-- 
Blog: www.rekby.ru

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Problem with GAE Datastore Viewer

2011-12-06 Thread bartprokop

Hi Ikai,

Thanks for your reply. Anyway, the problem is a bit deeper. When I set
the property to null, my App Engine Data Vievwer panel was fixed. So
null is valid value for my field. Before that, I accessed the field
programically (JAVA through JDO) and get very strange value for that
field:

com.google.appengine.api.datastore.PostalAddress@0 - zero is probably
bad value here.

I suppose that it was a bug in JDO when persisting the entity do
DataStore. After fixing property (setting it to null), it seems to
work.

On Dec 2, 7:16 pm, Ikai Lan (Google) ika...@google.com wrote:
 HI bart,

 Oy, I know what is causing this. The datastore admin is written using
 Python which has validations for fields like Address or GeoPt. The Java
 code does not, so it's possible to set this to null, which would cause this
 to error out.

 In general I advise people to use StringProperty as much as possible. To
 fix this, you have two choices:

 1. Change the Address property to be non-null programmatically
 2. Change the property on all entities to be a String property. You'll
 probably use App Engine mapreduce for this.

 I'm going to make a list of polish bugs and feature requests that I'm
 going to champion in an upcoming release. I'm wondering what the
 implications of a quick and dirty fix to remove the validations are.
 Unfortunately, if history is any kind of a teacher, the fix is NEVER that
 easy.

 --
 Ikai Lan
 Developer Programs Engineer, Google App Engine
 plus.ikailan.com | twitter.com/ikai







 On Fri, Dec 2, 2011 at 12:55 AM, bartprokop prokop.b...@gmail.com wrote:
  Hi, My application is developed in JAVA, utilizing JDO.
  Unfortunatelly, I started today getting starnge errors - 500 when
  trying to run GQL queries on my data:

  SELECT * FROM Przedszkole where __key__ =
  Key('agtzfmUtZHppZWNrb3ITCxILUHJ6ZWRzemtvbGUYgooIDA')

  results with Server error 500 and strange message: address must not
  be empty.

  My application ishttp://e-dziecko.appspot.com.

  Any hint on this?

  More funny, access object programatically seems to work without any
  interruption.

  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: is GAE admin panel access covered by SLA?

That's ridiculous! I wouldn't complaining if it was preview release!

The admin console if the first point where your users make assumptions
about your platform, if site is not responding first you blame your code,
but if admin panel fails that looks very unreliable and bad for your
business.

On 6 December 2011 01:09, Gregory D'alesandre gr...@google.com wrote:

 The Admin Console has no SLA.  We are working on migrating the Admin
 Console app to HRD which will improve its reliability dramatically at that
 point we will re-evaluate whether we can do an SLA for it.

 Greg D'Alesandre
 Senior Product Manager, Google App Engine


 On Mon, Dec 5, 2011 at 5:00 PM, JH ja...@mhztech.com wrote:

 +1.  Admin console is extremely unreliable.  It seems every time I use
 it response times are extremely slow or result in 500's.

 On Dec 5, 4:47 pm, Andrius A andriu...@gmail.com wrote:
  It is getting annoying when suddenly GAE admin panel starts behaving
 slowly
  and throws errors and its happening more often recently. I wonder is it
  covered by SLA for premier accounts?
  Is Google aware of it and can comment wether we should not expect it to
  work reliably 100%?
 
  Thanks,
  Andrius

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Unhappy GAE Developers Petition

Well, it appears I am the only one here unhappy with the issues who finds
time to write them here.  I cant believe GAE is out of preview and we are
paying 10x higher prices and confirmed by Google that GAE admin panel is
not reliable and of course they are working with that, just don't
undertstand why it took so long and now they blame MS everywhere they can.

To make the things worse for you Google some of the developers here were
contacted by a few major online magazines and Gartner, they all are
concered with the issues we were moaning here all the time so I really hope
upcomming articles will make you rethink.


On 24 November 2011 22:55, Andrius A andriu...@gmail.com wrote:

 Hi Brian,

 I am not happy because 2.7 is still experimental but pricing is increased,
 and I will be even more unhappy if by 1st December with fronted price
 increase python 2.7 will still be experimental.

 and to add to this there are more production issues I was writing but
 unfortunately haven't received an answer. Here is the list:

 1) Broken backends in SDK:
 http://groups.google.com/group/google-appengine-python/browse_thread/...http://www.google.com/url?sa=Dq=http://groups.google.com/group/google-appengine-python/browse_thread/thread/cf37410a6cf45283/425548774212402f%3Fq%3D%23425548774212402fusg=AFQjCNHiAuNVzUPyZFKcbjg_7YF83qfjDA


 2) Unexplained random frontend response latency:
 http://groups.google.com/group/google-appengine/browse_thread/thread/...http://www.google.com/url?sa=Dq=http://groups.google.com/group/google-appengine/browse_thread/thread/ad3c7473b34152ba/d187cb8acc295066%3Fq%3D%23d187cb8acc295066usg=AFQjCNGAbz_N7V41eK_bJOVpVamfBrY0xg


 3) Memcache expiry time setting bug:
 http://groups.google.com/group/google-appengine/browse_thread/thread/...http://www.google.com/url?sa=Dq=http://groups.google.com/group/google-appengine/browse_thread/thread/d0dca5913d33f3d1/bfd9117ad45ba516%3Fq%3D%23bfd9117ad45ba516usg=AFQjCNGTzoND8L5rybwCrVGlcUKL8ml_lw


 4) Channel API not working in backends:
 http://code.google.com/p/googleappengine/issues/detail?id=5123http://www.google.com/url?sa=Dq=http://code.google.com/p/googleappengine/issues/detail%3Fid%3D5123usg=AFQjCNGf37hfVjHzH50xrXasmAfLdW1v-A
  (Issue
 raised in May 27)
 5) Cant delete tasks from Backends:

 http://groups.google.com/group/google-appengine-python/browse_thread/thread/f1f674891cb5a2c1/9895847d1525d9e1#9895847d1525d9e1

 Thanks,
 Andrius

 On 24 November 2011 20:33, Brian Quinlan bquin...@google.com wrote:

 Hi Andrius,

 On Thu, Nov 24, 2011 at 11:40 PM, Andrius A andriu...@gmail.com wrote:
  Yes, we all do mistakes and GAE platform is great. But this thread is
 not
  about this discussion. So if you have any outstanding issues fill in the
  form!
 
  Personally I can't understand how you can be happy with price increase
 and
  production issues with python 2.7

 As an experimental release, we were expecting significant issues with
 the Python 2.7 runtime (i.e. instability, missing features,
 performance problems) and I've actually been pleasantly surprised at
 how few have been found (thanks to everyone for helping us test!).

 Are you unhappy with Python 2.7 production issues because we didn't
 adequately explain that you might experience problems when using it?
 Or is there some other reason?

 Cheers,
 Brian

  I totally believe that for hosting just html in app engine is great but
 if
  you try using more apis such as backends, channels, task queues you will
  come up to lots of problems.
 
  On Nov 24, 2011 12:10 PM, Simon Knott knott.si...@gmail.com wrote:
 
  Hi,
  In my experience on the GAE forums, a lot of unhappy developers I see
  are actually the people who don't understand the GAE architecture and
 have
  poorly designed applications for the environment.  Their applications
 are
  subsequently slow and/or expensive to run, for example because they
 haven't
  denormalised their data structure or they've indexed every property
 under
  the sun, and they then lay the blame on Google.  I personally applaud
 the
  development team who provide adhoc support on these forums, as they
 have far
  more patience than I have!
  I'm not saying that there couldn't be some improvements to the support
  channels for genuine issues with the service and I would love to have
 a more
  tiered structure, as someone has suggested in the last day, so that
 smaller
  companies could get support for less money, within a larger SLA.  It
 will be
  interesting to see who responds to your questionnaire and the
 complaints
  that are aired.
 
  Cheers,
  Simon
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To view this discussion on the web visit
  https://groups.google.com/d/msg/google-appengine/-/WRUFpsUp6IcJ.
  To post to this group, send email to google-appengine@googlegroups.com
 .
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options,

Re: [google-appengine] Re: Unhappy GAE Developers Petition

2011-12-06 Thread Simon Knott

Andrius,

I'm curious as to what changes you've made to your application to fit in 
with the new pricing model?  

Whilst a lot of people said they were facing 10x price increase (some were 
100x) when the model was first announced, a lot of those people have since 
come back and stated how they've managed to reduce that to 1.5-2x by 
re-writing their code.

Cheers,
Simon

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/A7uwTwRi69wJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: 1.6.1 Pre-release SDKs are live

2011-12-06 Thread Amine Azariz

Good work guys !
The Conversion API seems very cool.
Full-text-search ? Not yet public ?

On 5 déc, 23:32, Marzia Niccolai marce+appeng...@google.com wrote:
 Hi,

 We've uploaded the pre-release SDKs, get 'em while they are hot.

 As a reminder, the features listed in the pre-release SDKs notes are not
 available in production at this time, and documentation for them will be
 available when the release is officially announced. If you find any
 backward incompatible changes with the prerelease, please follow up on this
 thread with a link to the issue in our issue tracker.

 Thanks,
 Marzia

 Python Version 1.6.1===- You can now
 configure Frontend Instance Classes from your Admin Console's
 application settings page. Three classes are available, with
 increasing  memory, CPU limits, and associated cost. By default, all
 applications use the  basic frontend instance setting of 128MB memory
 and 600MHz CPU.- We've added new functionality to the Log API that
 will allow you to read your  application's logs programmatically.- We
 are releasing an experimental Conversion API that will allow you to
 convert  between document types including .doc, .html, .pdf, images
 using OCR, and  more.- The High Replication Datastore migration
 utility is now available as a GA  feature, and is no longer
 experimental.- The ext.db API query functions (run, fetch, count, get)
 now accept the keyword  arguments: deadline, read_policy,
 prefetch_size, batch_size, limit, offset,  start_cursor, end_cursor,
 keys_only. It is recommended that developers use  Query.run() with a
 limit or batch_size instead of Query.fetch() when iterating  over
 results.- The ext.db API model functions (get, put, delete,
 allocate_ids) now directly  accept the keyword arguments: deadline,
 read_policy- The Blobstore API now supports the multiple=true
 attribute to the HTML input  tag.- Fixed an issue logging unicode
 objects in the SDK.- Fixed an issue with the dev_appserver where HTTP
 HEAD always returned  content-length: 
 0.http://code.google.com/p/googleappengine/issues/detail?id=2000
 http://www.google.com/url?q=http://code.google.com/p/googleappengine/...-
 Fixed an issue in the SDK where importing Crypto.Util.Counter caused
 an  ImportError.http://code.google.com/p/googleappengine/issues/detail?id=4168
 http://www.google.com/url?q=http://code.google.com/p/googleappengine/...-
 Fixed an issue where the SDK didn't work with 
 virtualenv.http://code.google.com/p/googleappengine/issues/detail?id=4339
 http://www.google.com/url?q=http://code.google.com/p/googleappengine/...-
 Fixed an issue that was causing slow serving of requests in the SDK.
  http://code.google.com/p/googleappengine/issues/detail?id=6005
 http://www.google.com/url?q=http://code.google.com/p/googleappengine/...-
 Fixed an issue where the Channel API didn't work in the SDK with
 Python 2.7.    http://code.google.com/p/googleappengine/issues/detail?id=6267
 http://www.google.com/url?q=http://code.google.com/p/googleappengine/...-
 Fixed an httplib compatibility issue between Python 2.5 and Python 2.7
 in the  SDK.http://code.google.com/p/googleappengine/issues/detail?id=6271
 http://www.google.com/url?q=http://code.google.com/p/googleappengine/...-
 Fixed an error in the SDK on first page load for a 
 handler.http://code.google.com/p/googleappengine/issues/detail?id=6302
 http://www.google.com/url?q=http://code.google.com/p/googleappengine/...-
 Fixed an issue in the SDK that caused slowness due to app's logs being
 written  to the datastore. Set the --persist_logs flag in the SDK if
 you are using the  Logservice API to read 
 logs.http://code.google.com/p/googleappengine/issues/detail?id=6355
 http://www.google.com/url?q=http://code.google.com/p/googleappengine/...

 Java Version 1.6.1=- You can now configure Frontend
 Instance Classes from your Admin Console's  application settings page.
 Three classes are available, with increasing  memory, CPU limits, and
 associated cost. By default, all applications use the  basic frontend
 instance setting of 128MB memory and 600MHz CPU.- We've added new
 functionality to the Log API that will allow you to read your
 application's logs programmatically.- We are releasing an experimental
 Conversion API that will allow you to convert  between document types
 including .doc, .html, .pdf, images using OCR, and  more.- The High
 Replication Datastore migration utility is now available as a GA
 feature, and is no longer experimental.- The dev appserver now emits a
 warning when loading any classes that fall under
 com.google.appengine.repackaged.*.- Version 2.0.0 of the DataNucleus
 App Engine plugin is available as an  experimental download 
 fromhttp://code.google.com/p/datanucleus-appengine/downloads/list
 http://www.google.com/url?q=http://code.google.com/p/datanucleus-appe
 This release  adds support for JPA2 and JDO3.- The Blobstore API now
 supports the multiple=true attribute to the HTML input

Re: [google-appengine] DNS by IP Range, Where does Google Get it's DNS?

2011-12-06 Thread Joshua Smith

Probably from 8.8.8.8 and 8.8.4.4 dontchathink?

I've assumed that part of the reason they set these up is to figure out what 
domains are most popular.

On Dec 5, 2011, at 10:49 PM, Brandon Wirtz wrote:

 I realize this is a question for the search team, but they tell me it is 
 confidential, which considering the stuff they tell me anyway I think means 
 they don’t know.
  
 Can I set DNS that is provided to Google/ Google Bot to be different than 
 what it is for the rest of the world? I would like to be able to specify to 
 Google that my app is NOT hosted on GAE to make webmasters tools work again.  
 We got crawled on a domain for almost 1M page requests on a domain that 
 doesn’t have any content listed as indexable, but that didn’t stop the Google 
 bot from checking that every last page was set to no-index.
  
 Alas the cost of serving pages to the Google bot offsets any savings gained 
 by running in the cloud and so if I don’t find a solution several domains 
 will have to be relocated to CDNinaBox for Amazon instead.
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Unhappy GAE Developers Petition

Did you Simon reduced the costs by 1.5-2x? I dont know what are you
building but if I had a blog site running in GAE I wouldn't be complaining.

I think I am pushing GAE to the limits, my sites do a lot of ajax requests
with some processing on the server (1 request per user each second) and I
have thousands of users, you would say that I should use channels? Yes, I
thought so, but can't since there is no way to broadcast the same message
to all users and looping through each channel will delay sending message!
You would then say use backends to off load sending message through
backends? Ohh yes, but backends do not support channels! So backends are
really good, you can have them running on background and doing time
consuming work which actually saves your money! But wait, they are not
reliable, they restart few times per day and this is what google says in
documentation, so you need to build some mechanism on top to make sure when
backends restart your app do not collapse!
So far I have moved all processing to backends and made all data to be
passed to user requests via memcache and believe me this is where it gets
more complicated. You later realise its nearly impossible to have backends
and frontedns working in local environtment, so develpment process and
debugging is really really painful. Also backends in production restart few
times per day so reliability is very low, so on top you build  few master
backends who look after each other and launch other backends when needed
through task queue, and let this make more complicated, we know that task
queue can execute in some occasions the same task few times (which is more
frequent to happen within backends) and then you rely on memcache, and lets
make it a bit more complicated - you know that memcache can be  flushed any
time, so then you have datastore which can also time out and then to be it
the most complicated to handle - request can occasionaly  fail to load. Do
you see the point? You have a so many API's, we are being charged for them
but they are too unreliable, thats totoally not right. I understand that
nothing is  reliable, anything could happen, but at least backends should
not restart few times per day, datastore shoudn't time out every hour,
requests shouldn't fail every day! and finally I shouldn't be paying for
something called experimental!

On 6 December 2011 11:55, Simon Knott knott.si...@gmail.com wrote:

 Andrius,

 I'm curious as to what changes you've made to your application to fit in
 with the new pricing model?

 Whilst a lot of people said they were facing 10x price increase (some were
 100x) when the model was first announced, a lot of those people have since
 come back and stated how they've managed to reduce that to 1.5-2x by
 re-writing their code.

 Cheers,
 Simon

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/A7uwTwRi69wJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Unhappy GAE Developers Petition

2011-12-06 Thread Jeff Schnitzer

It sounds like you would be pushing any architecture to the limits.  A few
thousand requests per second with shared state is a nontrivial problem,
especially if you have a poor read/write ratio.  If you have a long-polling
scenario you may have chosen the wrong platform; this is something that
node.js or twisted python or equivalent asynchronous architectures excel
at.  But then expect to do a lot of work setting up and maintaining a fleet
of servers; it isn't a one-man job anymore.

What percentage of your (present) cost is instance hours vs datastore
operations?

 and finally I shouldn't be paying for something called experimental!

Maybe you shouldn't be *using* something called experimental.  You're
utilizing GAE in a way that is not efficient.  Possibly your application is
one that cannot be run efficiently on GAE; it sounds like an async
appserver would be more appropriate.  But maybe it can be done.  Perviously
Google hid the real cost of your app from you; now you're getting charged
based on a closer approximation of the real cost of service.

If you want help fitting your application to GAE's architecture then you
need to post a lot more details about what you are doing.  This post was
the first time in your countless messages that you included enough
information for any of us to even get a glimpse.  Seriously, we (the
community) are willing to help, but you need to actually want help.

Jeff

On Tue, Dec 6, 2011 at 9:08 AM, Andrius A andriu...@gmail.com wrote:

 Did you Simon reduced the costs by 1.5-2x? I dont know what are you
 building but if I had a blog site running in GAE I wouldn't be complaining.

 I think I am pushing GAE to the limits, my sites do a lot of ajax requests
 with some processing on the server (1 request per user each second) and I
 have thousands of users, you would say that I should use channels? Yes, I
 thought so, but can't since there is no way to broadcast the same message
 to all users and looping through each channel will delay sending message!
 You would then say use backends to off load sending message through
 backends? Ohh yes, but backends do not support channels! So backends are
 really good, you can have them running on background and doing time
 consuming work which actually saves your money! But wait, they are not
 reliable, they restart few times per day and this is what google says in
 documentation, so you need to build some mechanism on top to make sure when
 backends restart your app do not collapse!
 So far I have moved all processing to backends and made all data to be
 passed to user requests via memcache and believe me this is where it gets
 more complicated. You later realise its nearly impossible to have backends
 and frontedns working in local environtment, so develpment process and
 debugging is really really painful. Also backends in production restart few
 times per day so reliability is very low, so on top you build  few master
 backends who look after each other and launch other backends when needed
 through task queue, and let this make more complicated, we know that task
 queue can execute in some occasions the same task few times (which is more
 frequent to happen within backends) and then you rely on memcache, and lets
 make it a bit more complicated - you know that memcache can be  flushed any
 time, so then you have datastore which can also time out and then to be it
 the most complicated to handle - request can occasionaly  fail to load. Do
 you see the point? You have a so many API's, we are being charged for them
 but they are too unreliable, thats totoally not right. I understand that
 nothing is  reliable, anything could happen, but at least backends should
 not restart few times per day, datastore shoudn't time out every hour,
 requests shouldn't fail every day! and finally I shouldn't be paying for
 something called experimental!


 On 6 December 2011 11:55, Simon Knott knott.si...@gmail.com wrote:

 Andrius,

 I'm curious as to what changes you've made to your application to fit in
 with the new pricing model?

 Whilst a lot of people said they were facing 10x price increase (some
 were 100x) when the model was first announced, a lot of those people have
 since come back and stated how they've managed to reduce that to 1.5-2x by
 re-writing their code.

 Cheers,
 Simon

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/A7uwTwRi69wJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To

[google-appengine] Batch Get vs. Query and beware Appstats

2011-12-06 Thread Martin

I've been trying to track down why I'm seeing huge latencies in one of
my apps. I tracked it down to a large (~100 entity) batch get call. My
understanding was the batch get should be comparable to a query, which
is seems confirmed here

http://groups.google.com/group/google-appengine/browse_thread/thread/84ef15d24d3749fb/7ea0a5a22ef4b316

In my testing batch gets are about twice as long as a simple query.
With Appstats turned on it's taking 8x!

Query Fetch ~500ms (w/o appstats) ~500ms (w appstats)
Batch Get~1000ms (w/o appstats) ~4000ms (w appstats)

Does this seem about correct? My impression leaving appstats in the
production version was minimal overhead. I've included a simple
version below. This is python 2.7 and HR datastore btw.

from google.appengine.ext import db
import webapp2 as webapp

import time

class UserAccount(db.Model):
name = db.StringProperty(indexed=False)


class MakeDataHandler(webapp.RequestHandler):
def get(self):
db.put([UserAccount(key_name='TST'+str(i), name=Testing) for
i in range(100)])

class TestSpeedHandler2(webapp.RequestHandler):
def get(self):
start_time = time.time()

db.get([db.Key().from_path(UserAccount,str(f)) for f in
range(100)])

self.response.write(str(int((time.time() - start_time)*1000)))


class TestSpeedHandler3(webapp.RequestHandler):
def get(self):
start_time = time.time()

db.Query(UserAccount).fetch(100)

self.response.write(str(int((time.time() - start_time)*1000)))


application = webapp.WSGIApplication([(r'/makedata',MakeDataHandler),
  (r'/
testspeed2',TestSpeedHandler2),
  (r'/
testspeed3',TestSpeedHandler3)],
  debug=False)

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Max Idle, Max Pending Latency

Brandon,
This is very helpful (maybe a little funny too, but that might depend on
who the reader is :)).  I have questions Re: ***If you don’t have
concurrency that means you need to handle them in serial, unless you have
more than one instance*:  I am one of those cheap people who insist on
keeping $MII to 1, even though, during the day, my num-active-instances
hovers around 10, and spikes to ~25 occasionally.  I see this as I'm only
paying for one instance, but I'm getting 25 for free.  However, I
understand (and it is also apparent from your calculations below) that I'm
paying somehow, because of increased latencies, and eventual timeouts.  I'm
counting on the fact though, that my app startup time is around 200ms (not
15 seconds as in your example), and I have set $MPL to 50ms.  I expect GAE
to spawn multiple new instances indefinitely to service spikes in my load.
 Each new instance will take 200ms to spawn, and assuming a 15 second
timeout for my client, I expect I can handle 75 concurrent requests before
my client starts to see timeout errors.  Do you agree?  I'm just looking
for some confirmation for my calculations.  If course, since I'm lazy too
(and maybe stubborn as well), I have stayed married to python2.5, and this
means 75 concurrent requests will get served by 75 GAE front-end instances.
 Do you think GAE will spawn 75 instances, given my $MII=1, and $MPL=50ms,
in response to a 75 request spike in my load?

Thanks in advance.


On Mon, Dec 5, 2011 at 7:10 PM, Brandon Wirtz drak...@digerat.com wrote:

 I’m too lazy to build a tool to calculate this for you. 

 ** **

 The Formula for non-concurrent tasks is pretty simple.   

 ** **

 How many simultaneous requests do you need to handle? (Hint Chrome will
 request 8 assets at a time) If you don’t know I’m telling you it is 10.***
 *

 ** **

 Great. So you need to handle 10 requests. If you don’t have concurrency
 that means you need to handle them in serial, unless you have more than one
 instance.

 ** **

 How Many instances do you have? 1 Because if you are asking about this
 topic I assume you care enough to set 1 for always on, but you set idle to
 1 because you are cheap.

 ** **

 How Long Does your App take to spin up? 15 seconds. Mine doesn’t take this
 long, but yours does because I needed to pick a number that accounts for
 all you people who get the errors when your app spins up.

 ** **

 How long does the average request take to fill? 2 seconds. I pulled this
 out of my ass, you should be a good person and shoot for 1 second but
 things happen, yours take 2.

 ** **

 OK. S we now have all the numbers we need

 ** **

 ** **

 $CI = Current Instances

 $MII = Max Idle Instances

 $MPL = Max Pending Latency

 $ART = Avertage Request time

 $SR = Simultaneous Requests

 $

 ** **

 $SR[8] / $CI[1]  * $ART[2] = Seconds to fill requests. In this case 16  If
 your app was idle before the first request came in chances are that the
 user timed out the 8th request as it will generally only wait 15. You
 Errored.

 ** **

 So we kick on Max pending Latency of 200Ms Why that number it sounded nice
 a lot of you pick this number.

 $SR[8] / $CI[1]  * $ART[2]  - 16 Seconds of stuff to do, we need 8
 instances to try and get it all done because with 200 ms for a pending, no
 task can be pending it takes too long…

 ** **

 Spin up $MII [1] Instances

 We now have 1 instance for the first 5 seconds and 2 after it gets spun
 up, so

 Instance 1 fills Request 1,2,3

 Instance 2 fills request 4

 5,6,7,8 are still pending, we should spin up $MII[more instances because
 everyone is busy and more than 200ms out

 Instance 3 fills 6

 Instance 1 fills 7

 Instance 2 Fills 8

 ** **

 But the browser timed out Request 4 because it was 15 seconds before the
 instance started to spin up, it took 15 seconds to spin up instance 3 so
 that request failed, as did request 8.

 ** **

 ** **

 ** **

 ** **

 ** **

 ** **

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Unhappy GAE Developers Petition

We are running auction sites in GAE, they have timer and with each user bid
price increases. It deals with multiple bids by using sharded counters in
MS datastore, because its much more faster and consistent. Auction data
processing is done by backends and saved into memcache which is read by
frontends via ajax requests.

About 60-75% is paid for instances, before new pricing this kind of site in
GAE was cheap to host.

Big pain here is various issues with GAE inside, especially  requests
failing to load, often datastore read deadlines, unexplained requests
latency within GAE, all backends suddenly restarting in the same time..
How can you build something reliable in GAE by using these api's all
together? backends + taskqueue + memcache + datastore? Try to fit in your
mind, because I can't - any of them can fail any time, tasks can be
executed multiple times in the queue, memcache flushes any time and
datastore reads die?

And again, errors within GAE are too frequent, services die too often,
requests suddenly break or become too slow, admin console works like a
snail and crashes too often.


On 6 December 2011 14:59, Jeff Schnitzer j...@infohazard.org wrote:

 It sounds like you would be pushing any architecture to the limits.  A few
 thousand requests per second with shared state is a nontrivial problem,
 especially if you have a poor read/write ratio.  If you have a long-polling
 scenario you may have chosen the wrong platform; this is something that
 node.js or twisted python or equivalent asynchronous architectures excel
 at.  But then expect to do a lot of work setting up and maintaining a fleet
 of servers; it isn't a one-man job anymore.

 What percentage of your (present) cost is instance hours vs datastore
 operations?

  and finally I shouldn't be paying for something called experimental!

 Maybe you shouldn't be *using* something called experimental.  You're
 utilizing GAE in a way that is not efficient.  Possibly your application is
 one that cannot be run efficiently on GAE; it sounds like an async
 appserver would be more appropriate.  But maybe it can be done.  Perviously
 Google hid the real cost of your app from you; now you're getting charged
 based on a closer approximation of the real cost of service.

 If you want help fitting your application to GAE's architecture then you
 need to post a lot more details about what you are doing.  This post was
 the first time in your countless messages that you included enough
 information for any of us to even get a glimpse.  Seriously, we (the
 community) are willing to help, but you need to actually want help.

 Jeff

 On Tue, Dec 6, 2011 at 9:08 AM, Andrius A andriu...@gmail.com wrote:

 Did you Simon reduced the costs by 1.5-2x? I dont know what are you
 building but if I had a blog site running in GAE I wouldn't be complaining.

 I think I am pushing GAE to the limits, my sites do a lot of ajax
 requests with some processing on the server (1 request per user each
 second) and I have thousands of users, you would say that I should use
 channels? Yes, I thought so, but can't since there is no way to broadcast
 the same message to all users and looping through each channel will delay
 sending message! You would then say use backends to off load sending
 message through backends? Ohh yes, but backends do not support channels! So
 backends are really good, you can have them running on background and doing
 time consuming work which actually saves your money! But wait, they are not
 reliable, they restart few times per day and this is what google says in
 documentation, so you need to build some mechanism on top to make sure when
 backends restart your app do not collapse!
 So far I have moved all processing to backends and made all data to be
 passed to user requests via memcache and believe me this is where it gets
 more complicated. You later realise its nearly impossible to have backends
 and frontedns working in local environtment, so develpment process and
 debugging is really really painful. Also backends in production restart few
 times per day so reliability is very low, so on top you build  few master
 backends who look after each other and launch other backends when needed
 through task queue, and let this make more complicated, we know that task
 queue can execute in some occasions the same task few times (which is more
 frequent to happen within backends) and then you rely on memcache, and lets
 make it a bit more complicated - you know that memcache can be  flushed any
 time, so then you have datastore which can also time out and then to be it
 the most complicated to handle - request can occasionaly  fail to load. Do
 you see the point? You have a so many API's, we are being charged for them
 but they are too unreliable, thats totoally not right. I understand that
 nothing is  reliable, anything could happen, but at least backends should
 not restart few times per day, datastore shoudn't time out every hour,
 requests shouldn't fail

[google-appengine] Re: DeadlineExceeded errors haunting every now and then

Here are more examples. Can anyone from Google bother to respond? I do not 
understand sometimes why I am wasting my money, time and efforts with GAE. 
Makes me feel helpless. I am now going to start looking for alternative 
hosting.

http://gyazo.com/5a1445b8290e0f5dfd7c54233b27c4b1

@voscausa These errors come every once in a while, so could not be my code. 
Even if it is my code, I expect service from GAE folks. If they cannot 
provide service, then they should stop building software for end customers. 
My respect for Google has just gone down a notch.

Sarang

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/-JIyoVUfM6IJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: DeadlineExceeded errors haunting every now and then

Just wanted to point that I opened a ticket on Nov 18th for this issue, and 
there is still no reply or anything else for that matter from Google.

http://code.google.com/p/googleappengine/issues/list?can=4cursor=googleappengine%3A6359ts=1323189503updated=6359

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/t_yVXO6JPowJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Unhappy GAE Developers Petition

2011-12-06 Thread Niklas Rosencrantz

About the cost I completely agree with Simon Knott that bad system design 
can cause creeping costs you eg. having a cron job every 5 seconds for what 
can be batched in a daily cron job would significantly increase your costs 
at no particular gain in efficiency so I too think that some of us who saw 
creeping costs and creeping CPU used the GAE environment in a wrong way and 
we had to unload some of our scripts such as cron jobs every 5 seconds not 
thinking about number of reads/writes/creations. 

Which was good since the pricing force us the write program 
cost-efficiently - otherwise we can't afford running the apps.

So I think focus on a particular technical problem and solve it rather than 
bringing everything up under unhappiness. 
Work for a specific issue, debug it, make a proposed solution while you 
wait for the solution and then you'll have 2 solutions. 

I found I could solve all problems with app engine though in a python way 
sometimes there seems to have been one way only to solve it well (which was 
webapp2 + wtforms + jinja2 for me.)

But I like that you bring this up since I too have been frustrated with 
lack of real system functions such as eg. backup where phpMyAdmin you 
just push the button and you get a backup in zipped SQL of your whole 
database and no need to worry. App engine severely lacks a good backup 
system, we have to write our own backup system and this is not application 
development, it's developer tool development which is OK and fine solving 
it once per environment. Solving same problem for every app (backup, file 
system, static files, upgrading to python 2.7 with all its migrations..) 
should be done once for the whole platform like making one good backup 
function that works for the whole platform or just an XML export of the 
datastore if they can't other format.

My bottom line is that I came here to do application development and found 
myself implementing development tools such as backup and file system. 

That said, Google App Engine has enough advantages to still be the best 
platform and the best development environment. I just wish it was a real 
system that came with an export / import function and backwards 
compatibility and lots of ready-made builtin solutions for CMS with 
plug-ins etc like regular LAMP has. Regular LAMp has, to be fair, at least 
5 years head start of app engine so to be fair we have to wait some time 
for app engine to mature and all problems will have been solved. In 5 years 
time, I believe app engine will still exist and that nobody more worries 
about their python 2.6 to 2.7 migration.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/XZyCMPmm7eMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: DeadlineExceeded errors haunting every now and then

2011-12-06 Thread Johan Euphrosine (Google)

Hi Sarang,

Sorry for the late followup, but please keep in mind that there is no 
guaranteed response time for community support on the public issue tracker 
and the groups.

Your issue is likely to be due to temporary M/S latency, I 
would recommend switching to the High Replication Datastore using the new 
migration tool which is way more far reliable.

Hope that helps.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/cJ7dp3FiLaQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: DeadlineExceeded errors haunting every now and then

While I would really like to take your word on this, a simple google 
search completely contradicts what you are saying:

https://groups.google.com/forum/#!topic/google-appengine/ZXX_SgaDu2s

I am paying unnecessarily 8 cents/hr for DeadLineExceeded errors and I do 
not intend to pay even more using HRD and getting to the same place. I also 
read that blobstore data would be gone.. this is crazy. How can apps 
migrate then?

Also, regarding response time, if Google can deduct my credit card on time, 
then I also expect you to respond on time. 20 days is not acceptable at all 
when so many of your users are experiencing the same issues and your 
status shows everything is OK!

Sarang

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/UrfiFXjdiTcJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: DeadlineExceededError on HRD and sky-high latency (python27+threadsafe)

Glad to see this post as I am also suffering at the hands of GAE for the 
last month now. Every now and then DeadlineExceeded errors shoot up giving 
response times of over a minute!

See my thread:
https://groups.google.com/forum/?hl=en#!topic/google-appengine/NpqadsOUyZ4

Sarang

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/-JsYHq24EyMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: DeadlineExceeded errors haunting every now and then

Hi Johan,
I still can't get down to the bottom of why HRD is always the answer to
high latencies.  M/S versus HRD should be an entirely application-dependent
decision.  API timeouts can be used to gracefully degrade during a
downtime, but that's a conscious choice on part of the application's
author.  So, why the push towards HRD?  Also, if you keep pushing HRD,
costs are a better incentive than words :)  Basically, I'm saying that if
Google has a strong technical reason to move users towards HRD (such as:
why bother maintaining two options), then lower HRD costs to reflect that.


On Tue, Dec 6, 2011 at 11:37 AM, Sarang sar...@mycontactid.net wrote:

 While I would really like to take your word on this, a simple google
 search completely contradicts what you are saying:

 https://groups.google.com/forum/#!topic/google-appengine/ZXX_SgaDu2s

 I am paying unnecessarily 8 cents/hr for DeadLineExceeded errors and I do
 not intend to pay even more using HRD and getting to the same place. I also
 read that blobstore data would be gone.. this is crazy. How can apps
 migrate then?

 Also, regarding response time, if Google can deduct my credit card on
 time, then I also expect you to respond on time. 20 days is not acceptable
 at all when so many of your users are experiencing the same issues and your
 status shows everything is OK!

 Sarang

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/UrfiFXjdiTcJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: 1.6.1 Pre-release SDKs are live

2011-12-06 Thread sebastián serrano

Great Work! Is really nice feature to see the frontend class configuration 
feature, I have been thinking for a while it would be nice to have.

Cheers, Sebastian
www.devsar.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/xhQPJmAiFGQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Help!Can't create the second application

2011-12-06 Thread Amy Unruh

It sounds like you're logged in using a different account from the one that
you used to set up and create your first GAE application.  Make sure you're
logged in using the same account, then you should be able to add
applications by visiting https://appengine.google.com/ .

On Mon, Dec 5, 2011 at 9:01 AM, linxx oxx...@gmail.com wrote:

 I already have one app-engine application
 Just wanted to create the second one
 It says that:

 The phone number has been sent too many messages or has already been
 used to confirm an account.

 Of course, this phone was used to confirm an account.
 What now?
 Need hep,thanks.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] [python] Handling cookies with urlfetch service

2011-12-06 Thread ML

Hey,

I need to manage Google Alerts in my app, so I found this lib
http://pypi.python.org/pypi/galerts
It works outside of GAE env nicely, but does not want to on Appengine.

So then I discovered that any urllib library on GAE does not support
cookies and redirects out of the box, so I used this solution
http://everydayscripting.blogspot.com/2009/08/google-app-engine-cookie-handling-with.html
But it's keep showing me login page all time.

Any clues?
Did something change in GAE while handling cookies?

Regards, Matt

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] more than 12M operations in less than 10 hrs

2011-12-06 Thread cloudpre

Hi - my app is taking more than 12M reads in less than 10 hours- is
there a tuner to check which calls are exhibiting this weird behavior?

I mean how do we tune this?

Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Getting ApiDeadlineExceededException on file.create()

2011-12-06 Thread tempy

I've been getting these errors on every single call. App id: hrkleio

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Unhappy GAE Developers Petition

2011-12-06 Thread Jeff Schnitzer

On Tue, Dec 6, 2011 at 1:00 PM, Niklas Rosencrantz nikla...@gmail.comwrote:


 eg. backup where phpMyAdmin you just push the button and you get a backup
 in zipped SQL of your whole database and no need to worry.


I found backing up a production mysql system with significant data size to
be a huge pain in the ass.  Backups invariably ended up locking large
sections of the database and freezing the frontends for unacceptable
lengths of time.  The only solution was to set up a slave and run all
backups off of the slave.

InnoDB's awful locking policy is why I will never run MySQL in production
ever again.  If I need an RDBMS in production, I'll use Postgres.

This brings up an interesting question.  Without a full-database MVCC
system, how do you backup the whole database?  Especially with multigroup
transactions, there's no way to guarantee an isolated snapshot.  Depending
on how your application works, getting a consistent backup might be
impossible.

Jeff

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Unhappy GAE Developers Petition

2011-12-06 Thread Barry Hunter

On Tue, Dec 6, 2011 at 7:00 PM, Jeff Schnitzer j...@infohazard.org wrote:
 On Tue, Dec 6, 2011 at 1:00 PM, Niklas Rosencrantz nikla...@gmail.com
 wrote:


 eg. backup where phpMyAdmin you just push the button and you get a backup
 in zipped SQL of your whole database and no need to worry.

The idea of downloading a multi-gigabyte backup with phpMyAdmin, is
kinda funny.



 I found backing up a production mysql system with significant data size to
 be a huge pain in the ass.  Backups invariably ended up locking large
 sections of the database and freezing the frontends for unacceptable lengths
 of time.  The only solution was to set up a slave and run all backups off of
 the slave.

 InnoDB's awful locking policy is why I will never run MySQL in production
 ever again.  If I need an RDBMS in production, I'll use Postgres.

 This brings up an interesting question.  Without a full-database MVCC
 system, how do you backup the whole database?  Especially with multigroup
 transactions, there's no way to guarantee an isolated snapshot.  Depending
 on how your application works, getting a consistent backup might be
 impossible.

... without a readonly period - ie downtime.

Even youtube, goes readonly sometimes... (like right now)
http://www.google.com/support/youtube/bin/answer.py?answer=1751921topic=16550


 Jeff

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: DeadlineExceeded errors haunting every now and then

2011-12-06 Thread sergio.jar...@gmail.com

I'm seeing this on HRD!
Please see:

http://groups.google.com/group/google-appengine/browse_thread/thread/6575ff4a0683bb6b

On Dec 6, 5:32 pm, Johan Euphrosine (Google) pro...@google.com
wrote:
 Hi Sarang,

 Sorry for the late followup, but please keep in mind that there is no
 guaranteed response time for community support on the public issue tracker
 and the groups.

 Your issue is likely to be due to temporary M/S latency, I
 would recommend switching to the High Replication Datastore using the new
 migration tool which is way more far reliable.

 Hope that helps.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] How to catch Out of Memory errors?

2011-12-06 Thread André Pankraz

Sometimes I wonder if you really use GAE too ;) Or are Java and Python this 
different?


All ServletResponses and Log-Events are (for me) cached (and flushed only 
at the end of the request).

E.g.: If you have a 1 hour backend job no log entry appears in the Admin 
console till the request is complete...HttpResponse.out.write/flush will 
return nothing either till the response is complete. You can flush all you 
want...nothing happens.


If you get an OutOfMem instance kill or a (for me) famous Connection to 
client lost. no log entries will appear in the console, no output will be 
written to the servlet response stream.
This is especially funny because im sure that the Blob-creation API in Java 
has severe Memory leaks.


How do we debug this stuff? I miss jvisialvm or hsperf-dumps or or or...



I know Brandon...we all should only write 50 ms Proxy-Caches or we are 
wrong here. ;)

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/1ZeWxwnrKEQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Significant performance degradation on the SDK server with 1.5.3

2011-12-06 Thread PK

Wow!! I just benchmarked the same test case with 1.6.1 prerelease without 
any workarounds and am happy to report that that performance issues have 
been addressed:

real *0m35.221s*
user 0m16.330s
sys 0m2.460s

Thanks for taking care of the issues!!!

PK
www.gae123.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/NXZ_URDnJy8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: 1.6.1 Pre-release SDKs are live

2011-12-06 Thread PK

Thanks for the new features. I am also very happy to report that on 
evaluation:

1. Passed my regression test suite
2. Has addressed the SDK performance issues that I reported in this 
threadhttps://groups.google.com/d/topic/google-appengine/CBIsXJQn-f0/discussion.
 
Thanks for taking the issues seriously and addressing them in the next 
immediate opportunity.

PK
www.gae123.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/j0JPmIXcI3cJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Performance of the SDK development server has become unbearable

2011-12-06 Thread PK

Very happy to report that 1.6.1 prerelease has addressed the performance 
regressions reported here.  Many thanks to everybody who starred the issues 
and to Robert and the rest of the Google team who followed up.

PK
http://www.gae123.com

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/zbXjoh77m9YJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: DeadlineExceeded errors haunting every now and then

2011-12-06 Thread Johan Euphrosine

Hi Saraang,

DeadlineExceededError just means that the response has taken more time
that the authorized deadline (60s), this could be due to several
reason ranging from slow application code loading, slow application
code execution, slow datastore queries to M/S latency issues.

I only suggered the latter because it is the most common, and
migrating to HRD would eliminate this possible cause.

If you are looking more operational support with a response SLA, we
are offering this as part of the premier accounts:
http://www.google.com/enterprise/cloud/appengine/pricing.html

Hope that helps.

On Tue, Dec 6, 2011 at 9:37 AM, Sarang sar...@mycontactid.net wrote:
 While I would really like to take your word on this, a simple google
 search completely contradicts what you are saying:

 https://groups.google.com/forum/#!topic/google-appengine/ZXX_SgaDu2s

 I am paying unnecessarily 8 cents/hr for DeadLineExceeded errors and I do
 not intend to pay even more using HRD and getting to the same place. I also
 read that blobstore data would be gone.. this is crazy. How can apps migrate
 then?

 Also, regarding response time, if Google can deduct my credit card on time,
 then I also expect you to respond on time. 20 days is not acceptable at all
 when so many of your users are experiencing the same issues and your
 status shows everything is OK!

 Sarang

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/UrfiFXjdiTcJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Unhappy GAE Developers Petition

2011-12-06 Thread Kaan Soral

Can't agree with you more, you have my full support

I can't believe this last month either, I paid 1k$-2k$ extra just because 
1.6.1 didn't roll out and scheduler works for Google Accounting rather than 
us

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/B5AVjPxTC0YJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] Max Idle, Max Pending Latency

Likely you have to ask the GAE guys, but what I have seen is that Max Idle
is the maximum number of Instance that will be spun up at once.

If you take 200ms to spin up an instance and got 75 requests,  you'd take
75/205 (they takes a few ms to check) to spin up all the instances  so I'm
going to make the math easy  5 new instances a second, 75 requests, 15
seconds for browser to time out. you would cut it pretty close, but
shouldn't see too many time out.

 

That assumes everything is happy. Remember that when GAE is being slow
spinups sometimes take 50% longer and requests take 50% longer to serve. 

 

But you nailed the point.   You can Pay for Speed and reliability, or you
can be cheap and have things be a bit slower and less reliable.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Rishi Arora
Sent: Tuesday, December 06, 2011 7:46 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Max Idle, Max Pending Latency

 

Brandon,

This is very helpful (maybe a little funny too, but that might depend on who
the reader is :)).  I have questions Re: If you don't have concurrency that
means you need to handle them in serial, unless you have more than one
instance:  I am one of those cheap people who insist on keeping $MII to 1,
even though, during the day, my num-active-instances hovers around 10, and
spikes to ~25 occasionally.  I see this as I'm only paying for one
instance, but I'm getting 25 for free.  However, I understand (and it is
also apparent from your calculations below) that I'm paying somehow, because
of increased latencies, and eventual timeouts.  I'm counting on the fact
though, that my app startup time is around 200ms (not 15 seconds as in your
example), and I have set $MPL to 50ms.  I expect GAE to spawn multiple new
instances indefinitely to service spikes in my load.  Each new instance will
take 200ms to spawn, and assuming a 15 second timeout for my client, I
expect I can handle 75 concurrent requests before my client starts to see
timeout errors.  Do you agree?  I'm just looking for some confirmation for
my calculations.  If course, since I'm lazy too (and maybe stubborn as
well), I have stayed married to python2.5, and this means 75 concurrent
requests will get served by 75 GAE front-end instances.  Do you think GAE
will spawn 75 instances, given my $MII=1, and $MPL=50ms, in response to a 75
request spike in my load?

 

Thanks in advance.

 

On Mon, Dec 5, 2011 at 7:10 PM, Brandon Wirtz drak...@digerat.com wrote:

I'm too lazy to build a tool to calculate this for you. 

 

The Formula for non-concurrent tasks is pretty simple.   

 

How many simultaneous requests do you need to handle? (Hint Chrome will
request 8 assets at a time) If you don't know I'm telling you it is 10.

 

Great. So you need to handle 10 requests. If you don't have concurrency that
means you need to handle them in serial, unless you have more than one
instance.

 

How Many instances do you have? 1 Because if you are asking about this topic
I assume you care enough to set 1 for always on, but you set idle to 1
because you are cheap.

 

How Long Does your App take to spin up? 15 seconds. Mine doesn't take this
long, but yours does because I needed to pick a number that accounts for all
you people who get the errors when your app spins up.

 

How long does the average request take to fill? 2 seconds. I pulled this out
of my ass, you should be a good person and shoot for 1 second but things
happen, yours take 2.

 

OK. S we now have all the numbers we need

 

 

$CI = Current Instances

$MII = Max Idle Instances

$MPL = Max Pending Latency

$ART = Avertage Request time

$SR = Simultaneous Requests

$

 

$SR[8] / $CI[1]  * $ART[2] = Seconds to fill requests. In this case 16  If
your app was idle before the first request came in chances are that the user
timed out the 8th request as it will generally only wait 15. You Errored.

 

So we kick on Max pending Latency of 200Ms Why that number it sounded nice a
lot of you pick this number.

$SR[8] / $CI[1]  * $ART[2]  - 16 Seconds of stuff to do, we need 8
instances to try and get it all done because with 200 ms for a pending, no
task can be pending it takes too long.

 

Spin up $MII [1] Instances

We now have 1 instance for the first 5 seconds and 2 after it gets spun up,
so

Instance 1 fills Request 1,2,3

Instance 2 fills request 4

5,6,7,8 are still pending, we should spin up $MII[more instances because
everyone is busy and more than 200ms out

Instance 3 fills 6

Instance 1 fills 7

Instance 2 Fills 8

 

But the browser timed out Request 4 because it was 15 seconds before the
instance started to spin up, it took 15 seconds to spin up instance 3 so
that request failed, as did request 8.

 

 

 

 

 

 

-- 
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To

RE: [google-appengine] How to catch Out of Memory errors?

I don’t actually use GAE.  I troll the forums with mis-information so that
everyone gets it in their head how great the service is, then when they have
invested months in code that doesn’t do what they expect I buy their
forclosed home.  It’s great, Google gets more business, I get cheap
property. We all live happily ever after.

 

I’ll take a look at if Java does the same thing.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of André Pankraz
Sent: Tuesday, December 06, 2011 12:03 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] How to catch Out of Memory errors?

 

Sometimes I wonder if you really use GAE too ;) Or are Java and Python this
different?


All ServletResponses and Log-Events are (for me) cached (and flushed only at
the end of the request).

E.g.: If you have a 1 hour backend job no log entry appears in the Admin
console till the request is complete...HttpResponse.out.write/flush will
return nothing either till the response is complete. You can flush all you
want...nothing happens.


If you get an OutOfMem instance kill or a (for me) famous Connection to
client lost. no log entries will appear in the console, no output will be
written to the servlet response stream.
This is especially funny because im sure that the Blob-creation API in Java
has severe Memory leaks.


How do we debug this stuff? I miss jvisialvm or hsperf-dumps or or or...



I know Brandon...we all should only write 50 ms Proxy-Caches or we are wrong
here. ;)

-- 
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/1ZeWxwnrKEQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: DeadlineExceeded errors haunting every now and then

Guys, we have a form where we collect issues you are experiencing and will
pass it to Google to get an attention and hopefully have things getting
better. So please fill in:
https://docs.google.com/spreadsheet/viewform?hl=en_USformkey=dC1DMmpPSU1WZnk0d1FMa3JmNXIwaGc6MQ#gid=0

Johan, thank you for your attention, but I don't understand what you mean
by slow code loading and slow execution. Code would normally load and
execute fast, but in some cases it will take tens of seconds to load and/or
throw deadline exceeded errors at the high rate - how can you explain that?
Is this going to happen all the time and we should not complain?

If this is the case only with MS you should publicly acknowledge that by
clearly stating that in documentation, at the moment we are confused! MS
works for us because it's much faster and cheaper and we are ok with
scheduled maintenance.

Thank you,
Andrius

On 6 December 2011 20:48, Johan Euphrosine pro...@google.com wrote:

Hi Saraang,

DeadlineExceededError just means that the response has taken more time
that the authorized deadline (60s), this could be due to several
reason ranging from slow application code loading, slow application
code execution, slow datastore queries to M/S latency issues.

I only suggered the latter because it is the most common, and
migrating to HRD would eliminate this possible cause.

If you are looking more operational support with a response SLA, we
are offering this as part of the premier accounts:
http://www.google.com/enterprise/cloud/appengine/pricing.html

Hope that helps.

On Tue, Dec 6, 2011 at 9:37 AM, Sarang sar...@mycontactid.net wrote:
While I would really like to take your word on this, a simple google
search completely contradicts what you are saying:

https://groups.google.com/forum/#!topic/google-appengine/ZXX_SgaDu2s

I am paying unnecessarily 8 cents/hr for DeadLineExceeded errors and I do
not intend to pay even more using HRD and getting to the same place. I
also
read that blobstore data would be gone.. this is crazy. How can apps
migrate
then?

Also, regarding response time, if Google can deduct my credit card on
time,
then I also expect you to respond on time. 20 days is not acceptable at
all
when so many of your users are experiencing the same issues and your
status shows everything is OK!

Sarang

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/UrfiFXjdiTcJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Max Idle, Max Pending Latency

Ah excellent!  Thank you for confirming.  I'm glad I now understand what
I'm sacrificing by limiting $MII to 1.  I'll milk the system by continuing
to decrease $MPL (too bad it can't be negative :)), and when my load
increases to a point where minimum possible $MPL isn't good enough, i'll
bump up $MII to 2.  Hopefully my revenue can support that when that happens.

On Tue, Dec 6, 2011 at 3:29 PM, Brandon Wirtz drak...@digerat.com wrote:

 Likely you have to ask the GAE guys, but what I have seen is that Max Idle
 is the maximum number of Instance that will be spun up at once.

 If you take 200ms to spin up an instance and got 75 requests,  you’d take
 75/205 (they takes a few ms to check) to spin up all the instances  so I’m
 going to make the math easy  5 new instances a second, 75 requests, 15
 seconds for browser to time out… you would cut it pretty close, but
 shouldn’t see too many time out.

 ** **

 That assumes everything is happy… Remember that when GAE is being “slow”
 spinups sometimes take 50% longer and requests take 50% longer to serve… *
 ***

 ** **

 But you nailed the point.   You can Pay for Speed and reliability, or you
 can be cheap and have things be a bit slower and less reliable.

 ** **

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Rishi Arora
 *Sent:* Tuesday, December 06, 2011 7:46 AM
 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Max Idle, Max Pending Latency

 ** **

 Brandon,

 This is very helpful (maybe a little funny too, but that might depend on
 who the reader is :)).  I have questions Re: ***If you don’t have
 concurrency that means you need to handle them in serial, unless you have
 more than one instance*:  I am one of those cheap people who insist on
 keeping $MII to 1, even though, during the day, my num-active-instances
 hovers around 10, and spikes to ~25 occasionally.  I see this as I'm only
 paying for one instance, but I'm getting 25 for free.  However, I
 understand (and it is also apparent from your calculations below) that I'm
 paying somehow, because of increased latencies, and eventual timeouts.  I'm
 counting on the fact though, that my app startup time is around 200ms (not
 15 seconds as in your example), and I have set $MPL to 50ms.  I expect GAE
 to spawn multiple new instances indefinitely to service spikes in my load.
  Each new instance will take 200ms to spawn, and assuming a 15 second
 timeout for my client, I expect I can handle 75 concurrent requests before
 my client starts to see timeout errors.  Do you agree?  I'm just looking
 for some confirmation for my calculations.  If course, since I'm lazy too
 (and maybe stubborn as well), I have stayed married to python2.5, and this
 means 75 concurrent requests will get served by 75 GAE front-end instances.
  Do you think GAE will spawn 75 instances, given my $MII=1, and $MPL=50ms,
 in response to a 75 request spike in my load?

 ** **

 Thanks in advance.

 ** **

 On Mon, Dec 5, 2011 at 7:10 PM, Brandon Wirtz drak...@digerat.com wrote:
 

 I’m too lazy to build a tool to calculate this for you. 

  

 The Formula for non-concurrent tasks is pretty simple.   

  

 How many simultaneous requests do you need to handle? (Hint Chrome will
 request 8 assets at a time) If you don’t know I’m telling you it is 10.***
 *

  

 Great. So you need to handle 10 requests. If you don’t have concurrency
 that means you need to handle them in serial, unless you have more than one
 instance.

  

 How Many instances do you have? 1 Because if you are asking about this
 topic I assume you care enough to set 1 for always on, but you set idle to
 1 because you are cheap.

  

 How Long Does your App take to spin up? 15 seconds. Mine doesn’t take this
 long, but yours does because I needed to pick a number that accounts for
 all you people who get the errors when your app spins up.

  

 How long does the average request take to fill? 2 seconds. I pulled this
 out of my ass, you should be a good person and shoot for 1 second but
 things happen, yours take 2.

  

 OK. S we now have all the numbers we need

  

  

 $CI = Current Instances

 $MII = Max Idle Instances

 $MPL = Max Pending Latency

 $ART = Avertage Request time

 $SR = Simultaneous Requests

 $

  

 $SR[8] / $CI[1]  * $ART[2] = Seconds to fill requests. In this case 16  If
 your app was idle before the first request came in chances are that the
 user timed out the 8th request as it will generally only wait 15. You
 Errored.

  

 So we kick on Max pending Latency of 200Ms Why that number it sounded nice
 a lot of you pick this number.

 $SR[8] / $CI[1]  * $ART[2]  - 16 Seconds of stuff to do, we need 8
 instances to try and get it all done because with 200 ms for a pending, no
 task can be pending it takes too long…

Re: [google-appengine] Re: The Performance of the SDK development server has become unbearable

I would like to add to that as well: developing backends in local
environment is almost impossible because it is really slow running and
there is no documentation how to do that using SDK?

On 6 December 2011 20:19, PK p...@gae123.com wrote:

 Very happy to report that 1.6.1 prerelease has addressed the performance
 regressions reported here.  Many thanks to everybody who starred the issues
 and to Robert and the rest of the Google team who followed up.

 PK
 http://www.gae123.com

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/zbXjoh77m9YJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: DeadlineExceeded errors haunting every now and then

Andrius,
I agree with your point about MS too. It sounds really suspicious that
Google keeps pushing HRD. From the documentation, MS and HRD have clearly
stated cost-benefit trade-offs - but its too bad that in practice that
isn't the case. It should be the app author's choice. And if Google has a
hidden agenda to push HRD, then they should incentivize us by lowering its
cost, since its more of an advantage to them than to people like us.

Rishi

On Tue, Dec 6, 2011 at 4:03 PM, Andrius A andriu...@gmail.com wrote:

Guys, we have a form where we collect issues you are experiencing and will
pass it to Google to get an attention and hopefully have things getting
better. So please fill in:

https://docs.google.com/spreadsheet/viewform?hl=en_USformkey=dC1DMmpPSU1WZnk0d1FMa3JmNXIwaGc6MQ#gid=0

Thank you,
Andrius

On 6 December 2011 20:48, Johan Euphrosine pro...@google.com wrote:

Hi Saraang,

I only suggered the latter because it is the most common, and
migrating to HRD would eliminate this possible cause.

If you are looking more operational support with a response SLA, we
are offering this as part of the premier accounts:
http://www.google.com/enterprise/cloud/appengine/pricing.html

Hope that helps.

On Tue, Dec 6, 2011 at 9:37 AM, Sarang sar...@mycontactid.net wrote:
While I would really like to take your word on this, a simple google
search completely contradicts what you are saying:

https://groups.google.com/forum/#!topic/google-appengine/ZXX_SgaDu2s

I am paying unnecessarily 8 cents/hr for DeadLineExceeded errors and I
do
not intend to pay even more using HRD and getting to the same place. I
also
read that blobstore data would be gone.. this is crazy. How can apps
migrate
then?

Sarang

--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/UrfiFXjdiTcJ.

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

Re: [google-appengine] Re: DeadlineExceededError on HRD and sky-high latency (python27+threadsafe)

You are not the only one, I don't like the fact that google puts a blame on
MS, there is something wrong with GAE constantly going on and they should
admit that and explain us in more details what's happening.
We have a form for issue like this to be noticed, please fill in to get
more attention:
https://docs.google.com/spreadsheet/viewform?hl=en_USformkey=dC1DMmpPSU1WZnk0d1FMa3JmNXIwaGc6MQ#gid=0

I really believe into GAE and still think this is a great a platform, but
it has issues which need be admitted and taken care of.

On 6 December 2011 17:41, Sarang sar...@mycontactid.net wrote:

Glad to see this post as I am also suffering at the hands of GAE for the
last month now. Every now and then DeadlineExceeded errors shoot up giving
response times of over a minute!

See my thread:
https://groups.google.com/forum/?hl=en#!topic/google-appengine/NpqadsOUyZ4

Sarang

Re: [google-appengine] DeadlineExceededError on HRD and sky-high latency (python27+threadsafe)

On Tue, Dec 6, 2011 at 6:14 AM, sergio.jar...@gmail.com
sergio.jar...@gmail.com wrote:
 Hi!
 I've successfully migrated from master-slave to hdd with python 27 and
 threadsafe however I've been getting sporadically a few deadline
 exceeded errors (bellow stack trace excerpt).

 I also have configured min idle requests to automatic and maximum to 1
 and pending latency to 800ms, I've observed on occasion  latencies of
 up to 63000(!!!)ms and without app engine spinning extra instances to
 serve requests.

 Any thoughts? I can provide app-id off-list if someone from google
 wants to have a look.

You are likely encountering this issue:
http://code.google.com/p/googleappengine/issues/detail?id=6323

Could you try setting threadsafe to no and seeing if that fixes the problem?

Cheers,
Brian

 Thanks!

 File /base/python27_runtime/python27_lib/versions/1/google/appengine/
 ext/db/__init__.py, line 3588, in __get__
    instance = get(reference_id)
  File /base/python27_runtime/python27_lib/versions/1/google/
 appengine/ext/db/__init__.py, line 1515, in get
    return get_async(keys, **kwargs).get_result()
  File /base/python27_runtime/python27_lib/versions/1/google/
 appengine/api/apiproxy_stub_map.py, line 592, in get_result
    return self.__get_result_hook(self)
  File /base/python27_runtime/python27_lib/versions/1/google/
 appengine/datastore/datastore_rpc.py, line 1437, in __get_hook
    entities = rpc.user_data(entities)
  File /base/python27_runtime/python27_lib/versions/1/google/
 appengine/api/datastore.py, line 623, in local_extra_hook
    return extra_hook(result)
  File /base/python27_runtime/python27_lib/versions/1/google/
 appengine/ext/db/__init__.py, line 1485, in extra_hook
    model = cls1.from_entity(entity)
  File /base/python27_runtime/python27_lib/versions/1/google/
 appengine/ext/db/__init__.py, line 1419, in from_entity
    return cls(None, _from_entity=entity, **entity_values)
  File /base/python27_runtime/python27_lib/versions/1/google/
 appengine/ext/db/__init__.py, line 1800, in __init__
    super(Expando, self).__init__(parent, key_name, _app, **kwds)
  File /base/python27_runtime/python27_lib/versions/1/google/
 appengine/ext/db/__init__.py, line 945, in __init__
    prop.__set__(self, value)
  File /base/python27_runtime/python27_lib/versions/1/google/
 appengine/ext/db/__init__.py, line 600, in __set__
    setattr(model_instance, self._attr_name(), value)
  File /base/python27_runtime/python27_lib/versions/1/google/
 appengine/ext/db/__init__.py, line 1826, in __setattr__
    check_reserved_word(key)
  File /base/python27_runtime/python27_lib/versions/1/google/
 appengine/ext/db/__init__.py, line 300, in check_reserved_word
    if datastore_types.RESERVED_PROPERTY_NAME.match(attr_name):
 DeadlineExceededError

 --
 You received this message because you are subscribed to the Google Groups 
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to 
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: DeadlineExceededError on HRD and sky-high latency (python27+threadsafe)

Hey Andrius,

On Wed, Dec 7, 2011 at 9:14 AM, Andrius A andriu...@gmail.com wrote:
You are not the only one, I don't like the fact that google puts a blame on
MS, there is something wrong with GAE constantly going on and they should
admit that and explain us in more details what's happening.

There is a known bug where the combination of Python 2.7 and enabling
concurrent requests causes large latency increases. The Python 2.7
runtime is still experimental and bugs like this are expected.

There are also known issues where accessing the MS datastore can
result in highly variable latencies and possibly errors. This is the
result of the design of the MS datastore and is why we promote
migrating to the HR datastore.

The App Engine Administration Console uses the MS datastore (for a
variety of esoteric reasons) so its latency spikes will often
correlate with MS latency spikes.

We have a form for issue like this to be noticed, please fill in to get more
attention:
https://docs.google.com/spreadsheet/viewform?hl=en_USformkey=dC1DMmpPSU1WZnk0d1FMa3JmNXIwaGc6MQ#gid=0

Can I also suggest that people file bugs using the public issue tracker at:
http://code.google.com/p/googleappengine/issues ?

Also, if you are using an experimental feature, it would be great if
you do a quick search to see if the issue has already been filed and
make a note or star it yourself it is has. The more information we
have the easier it is to prioritize and fix bugs.

I really believe into GAE and still think this is a great a platform, but it
has issues which need be admitted and taken care of.

No one is more aware of App Engine's imperfections than the App Engine
team and we have a public issue tracker so that people can make us
aware of their particular issues and see what problems other people
are having.

Cheers,
Brian

On 6 December 2011 17:41, Sarang sar...@mycontactid.net wrote:

Glad to see this post as I am also suffering at the hands of GAE for the
last month now. Every now and then DeadlineExceeded errors shoot up giving
response times of over a minute!

See my thread:
https://groups.google.com/forum/?hl=en#!topic/google-appengine/NpqadsOUyZ4

Sarang

Re: [google-appengine] Re: DeadlineExceeded errors haunting every now and then

2011-12-06 Thread Gregory D'alesandre

Hello All,

The reason that Johan is encouraging you to migrate to HRD is because it is
more reliable. M/S is wholly dependent on a single piece of storage
infrastructure, if that happens to be slow your app will likely slow down
and you can then end up getting DeadlineExceeded errors. When you ask
about fixing the latency problems with M/S, we have done so by providing
an option that has a much more consistent latency profile (called HRD).
Beyond that there just isn't that much we can do to fix one-off latency
issues, they are typically not systemwide issues which is why a few people
at a time are seeing them.

Sergio's comment that he is seeing this on HRD is most likely because he is
using Python 2.7 which is still experimental (meaning you will periodically
run into issues).

To be clear, HRD and M/S are the same price, there is no price
differential. I'm not sure why you think it is suspicious that we are
encouraging everyone to move to a more stable product. You are completely
correct that, today, it is your choice as to whether you want to use M/S or
HRD. But when you are choosing M/S you are choosing an inherently less
stable option and this means that when you run into issues you are taking
on that responsibility as well. To Andrius' point, you should feel free to
complain but the basic response will be: if you want more consistent
latency profiles, you should move to HRD. We have publicly stated this by
not providing any SLA for M/S but only for HRD.

I hope that helps clarify,

Greg D'Alesandre
Senior Product Manager, Google App Engine

On Tue, Dec 6, 2011 at 2:12 PM, Rishi Arora rishi.ar...@ship-rack.comwrote:

Rishi

On Tue, Dec 6, 2011 at 4:03 PM, Andrius A andriu...@gmail.com wrote:

Guys, we have a form where we collect issues you are experiencing and
will pass it to Google to get an attention and hopefully have things
getting better. So please fill in:

https://docs.google.com/spreadsheet/viewform?hl=en_USformkey=dC1DMmpPSU1WZnk0d1FMa3JmNXIwaGc6MQ#gid=0

Thank you,
Andrius

On 6 December 2011 20:48, Johan Euphrosine pro...@google.com wrote:

Hi Saraang,

I only suggered the latter because it is the most common, and
migrating to HRD would eliminate this possible cause.

If you are looking more operational support with a response SLA, we
are offering this as part of the premier accounts:
http://www.google.com/enterprise/cloud/appengine/pricing.html

Hope that helps.

On Tue, Dec 6, 2011 at 9:37 AM, Sarang sar...@mycontactid.net wrote:
While I would really like to take your word on this, a simple google
search completely contradicts what you are saying:

https://groups.google.com/forum/#!topic/google-appengine/ZXX_SgaDu2s

I am paying unnecessarily 8 cents/hr for DeadLineExceeded errors and I
do
not intend to pay even more using HRD and getting to the same place. I
also
read that blobstore data would be gone.. this is crazy. How can apps
migrate
then?

Also, regarding response time, if Google can deduct my credit card on
time,
then I also expect you to respond on time. 20 days is not acceptable
at all
when so many of your users are experiencing the same issues and your
status shows everything is OK!

Sarang

--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/UrfiFXjdiTcJ.

To post to this group, send email to google-appengine@googlegroups.com
.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit

[google-appengine] Blobstore outage?

2011-12-06 Thread rasmus

We have seen many blobstore errors today:

DeadlineExceededError: The API call file.Create() took too long to respond

for 2 periods: around 18:15 - 18:30 GMT
and briefly around 22:00 GMT.

I have been switching between 2 versions to fix this: 
(1) stores 1-3 blobs for *every* request, including many 500 byte ones, and
(2) stores only 1kb blobs, on maybe 1 in 5+ requests

Version (1) had periods when most blobstore requests failed (then we 
respond 500, then our customers retry fairly eagerly, so it piles up)
but it also worked for a few hours,
Version (2) seems nearly problem-free - but it was of course running at 
different times from (1), and there were occasional rejects.


This made me wonder:
Is it possible for our app to hammer down Blobstore by storing many small 
blobs?

Our volumes are not huge, average 10-100 requests / minute, with peaks of 
100+ requests near-simultaneously.
But in each case the outage resolved immediately when I switched from 
version (1) to v (2)

Thanks for any tips/confirmation.

rasmus


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/wPN-R3p3E-UJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: DeadlineExceeded errors haunting every now and then

2011-12-06 Thread Kenneth

Hi Greg,

Can you give us some indication if the migration tool is ever going to include 
blobstore migration? That's the only thing holding me back. I know I can do it 
myself but I'd rather not. I see now that it has gone ga, have you stuck a fork 
in it and called it done? 

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/f7KzNP-pCuwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: DeadlineExceeded errors haunting every now and then

2011-12-06 Thread Gregory D'alesandre

Hi Kenneth,

We are planning to add blobstore migration at some point but it won't
likely be until the middle of Q1 at the earliest.

Greg

On Tue, Dec 6, 2011 at 3:27 PM, Kenneth kennet...@aladdinschools.comwrote:

 Hi Greg,

 Can you give us some indication if the migration tool is ever going to
 include blobstore migration? That's the only thing holding me back. I know
 I can do it myself but I'd rather not. I see now that it has gone ga, have
 you stuck a fork in it and called it done?

 Thanks

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/f7KzNP-pCuwJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: DeadlineExceeded errors haunting every now and then

In docs it says that HRD is more costly *The High Replication Datastore
costs more due to the additional replication (see billing page for pricing
details). Due to the higher cost, we recommend High Replication Datastore
primarily for those developers building critical App Engine applications
that require the highest availability*.
It also says *queries on a single guestbook to be strongly consistent, but
also limits changes to the guestbook to 1 write per second (the supported
limit for entity groups). Therefore, writing to a single entity group per
guestbook is not ideal when high usage is expected. If your app is likely
to encounter heavy write usage, consider using another means. For example,
you can put recent posts in memcache with an expiration, and then display a
mix of recent posts from memcache and posts retrieved from the datastore.*
How could we trust by putting data to memcache if we know it can be evicted
any time? To use HRD is not viable for applications which need strong
consistency for high rate of puts. At the moment MS is perfect what it does
but bloody thing keeps dying.. I can't believe there is no way to improve
MS and you are switching to HRD. In long term HRD can cause more problems
for applications parts were it wasn't aticipating to receive higher rate of
inserts, and having a limit of 1 write per second is a disastrous.

On 6 December 2011 23:29, Gregory D'alesandre gr...@google.com wrote:

 Hi Kenneth,

 We are planning to add blobstore migration at some point but it won't
 likely be until the middle of Q1 at the earliest.

 Greg


 On Tue, Dec 6, 2011 at 3:27 PM, Kenneth kennet...@aladdinschools.comwrote:

 Hi Greg,

 Can you give us some indication if the migration tool is ever going to
 include blobstore migration? That's the only thing holding me back. I know
 I can do it myself but I'd rather not. I see now that it has gone ga, have
 you stuck a fork in it and called it done?

 Thanks

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/f7KzNP-pCuwJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] DataStore Tiering Was Dealine Exceeded Errors

I posted this logic once before but people only listen when they are
ranting.

 

You can only change an entity once per second. You can however maintain a
data store Cache that will save you money, writes, and prevent you from
ever seeing write limits.

 

By maintaining a Small DataStore of Writes To commit  things you want to
write to data store permanently can be put in to a data store which you
increment the task on.

 

When you do a Read from the datastore you check that the writes to commit
don't supersede the data in the main data store.

 

This will save you on indexing operations, and reduce your write throttling
and if done correctly can even move most of your reads to Memory rather than
DataStore.

 

This will increase your number of reads. Which will increase your Costs
slightly over MS, but likely is better than having MS down or MS Read
OnlyTime.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Andrius A
Sent: Tuesday, December 06, 2011 3:44 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: DeadlineExceeded errors haunting every
now and then

 

In docs it says that HRD is more costly The High Replication Datastore
costs more due to the additional replication (see billing page for pricing
details). Due to the higher cost, we recommend High Replication Datastore
primarily for those developers building critical App Engine applications
that require the highest availability. 

It also says queries on a single guestbook to be strongly consistent, but
also limits changes to the guestbook to 1 write per second (the supported
limit for entity groups). Therefore, writing to a single entity group per
guestbook is not ideal when high usage is expected. If your app is likely to
encounter heavy write usage, consider using another means. For example, you
can put recent posts in memcache with an expiration, and then display a mix
of recent posts from memcache and posts retrieved from the datastore. 

How could we trust by putting data to memcache if we know it can be evicted
any time? To use HRD is not viable for applications which need strong
consistency for high rate of puts. At the moment MS is perfect what it does
but bloody thing keeps dying.. I can't believe there is no way to improve MS
and you are switching to HRD. In long term HRD can cause more problems for
applications parts were it wasn't aticipating to receive higher rate of
inserts, and having a limit of 1 write per second is a disastrous.

On 6 December 2011 23:29, Gregory D'alesandre gr...@google.com wrote:

Hi Kenneth, 

 

We are planning to add blobstore migration at some point but it won't likely
be until the middle of Q1 at the earliest.

 

Greg

 

On Tue, Dec 6, 2011 at 3:27 PM, Kenneth kennet...@aladdinschools.com
wrote:

Hi Greg,

Can you give us some indication if the migration tool is ever going to
include blobstore migration? That's the only thing holding me back. I know I
can do it myself but I'd rather not. I see now that it has gone ga, have you
stuck a fork in it and called it done?

Thanks


--
You received this message because you are subscribed to the Google Groups
Google App Engine group.

To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/f7KzNP-pCuwJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
mailto:google-appengine%2bunsubscr...@googlegroups.com .
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

 

-- 
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
mailto:google-appengine%2bunsubscr...@googlegroups.com .
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

 

-- 
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: DeadlineExceededError on HRD and sky-high latency (python27+threadsafe)

Hey Brian,

thank you for your reply. I am still on 2.5 since 2.7 is still
experimental, and I don't really believe that MS is everything to blame,
there are issues with requests latencies, which I have monitored and there
was nothing wrong with datastore. I have been posting about that here and
opened a ticket but have not received an answer unfortunately.

I am moaning here all the time because we shouldn't be charged so much as
you confirmed already many times that there are issues and 2.7 is still
faulty.
The main point here is that we should not pay more just because GAE is now
called production ready. I will be repeating that until you go back to
preview release or admit that by dropping the prices or improve the
reliability.

Thanks,
Andrius

On 6 December 2011 22:56, Brian Quinlan bquin...@google.com wrote:

Hey Andrius,

On Wed, Dec 7, 2011 at 9:14 AM, Andrius A andriu...@gmail.com wrote:
You are not the only one, I don't like the fact that google puts a blame
on
MS, there is something wrong with GAE constantly going on and they should
admit that and explain us in more details what's happening.

There is a known bug where the combination of Python 2.7 and enabling
concurrent requests causes large latency increases. The Python 2.7
runtime is still experimental and bugs like this are expected.

The App Engine Administration Console uses the MS datastore (for a
variety of esoteric reasons) so its latency spikes will often
correlate with MS latency spikes.

We have a form for issue like this to be noticed, please fill in to get
more
attention:

https://docs.google.com/spreadsheet/viewform?hl=en_USformkey=dC1DMmpPSU1WZnk0d1FMa3JmNXIwaGc6MQ#gid=0

Can I also suggest that people file bugs using the public issue tracker at:
http://code.google.com/p/googleappengine/issues ?

I really believe into GAE and still think this is a great a platform,
but it
has issues which need be admitted and taken care of.

Cheers,
Brian

On 6 December 2011 17:41, Sarang sar...@mycontactid.net wrote:

Glad to see this post as I am also suffering at the hands of GAE for the
last month now. Every now and then DeadlineExceeded errors shoot up
giving
response times of over a minute!

See my thread:

https://groups.google.com/forum/?hl=en#!topic/google-appengine/NpqadsOUyZ4

Sarang

--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/-JsYHq24EyMJ.

Re: [google-appengine] Re: DeadlineExceeded errors haunting every now and then

On Wed, Dec 7, 2011 at 10:44 AM, Andrius A andriu...@gmail.com wrote:
 In docs it says that HRD is more costly The High Replication Datastore
 costs more due to the additional replication (see billing page for pricing
 details). Due to the higher cost, we recommend High Replication Datastore
 primarily for those developers building critical App Engine applications
 that require the highest availability.

I've filed a bug against this documentation.

 It also says queries on a single guestbook to be strongly consistent, but
 also limits changes to the guestbook to 1 write per second (the supported
 limit for entity groups). Therefore, writing to a single entity group per
 guestbook is not ideal when high usage is expected. If your app is likely to
 encounter heavy write usage, consider using another means. For example, you
 can put recent posts in memcache with an expiration, and then display a mix
 of recent posts from memcache and posts retrieved from the datastore.
 How could we trust by putting data to memcache if we know it can be evicted
 any time?

You wouldn't as definitive storage but it is reasonable to cache
recent posts there.

 To use HRD is not viable for applications which need strong
 consistency for high rate of puts.

You may have to think about how your organize your data in order to
satisfy your consistency and throughput requirements but I am very
skeptical when you say that it is simply not viable.

Google, for example, has been successful using the same technology for
many of its own high-volume properties.

 At the moment MS is perfect what it does
 but bloody thing keeps dying..

That is an architectural problem that is very hard to fix with making
the same kinds of trade-offs that we made when implementing HRD .

Cheers,
Brian

 I can't believe there is no way to improve MS
 and you are switching to HRD. In long term HRD can cause more problems for
 applications parts were it wasn't aticipating to receive higher rate of
 inserts, and having a limit of 1 write per second is a disastrous.


 On 6 December 2011 23:29, Gregory D'alesandre gr...@google.com wrote:

 Hi Kenneth,

 We are planning to add blobstore migration at some point but it won't
 likely be until the middle of Q1 at the earliest.

 Greg


 On Tue, Dec 6, 2011 at 3:27 PM, Kenneth kennet...@aladdinschools.com
 wrote:

 Hi Greg,

 Can you give us some indication if the migration tool is ever going to
 include blobstore migration? That's the only thing holding me back. I know I
 can do it myself but I'd rather not. I see now that it has gone ga, have you
 stuck a fork in it and called it done?

 Thanks

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/f7KzNP-pCuwJ.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: DeadlineExceededError on HRD and sky-high latency (python27+threadsafe)

Hi Andrius,

On Wed, Dec 7, 2011 at 11:15 AM, Andrius A andriu...@gmail.com wrote:
Hey Brian,

thank you for your reply. I am still on 2.5 since 2.7 is still experimental,
and I don't really believe that MS is everything to blame, there are issues
with requests latencies, which I have monitored and there was nothing wrong
with datastore. I have been posting about that here and opened a ticket but
have not received an answer unfortunately.

I couldn't see your name attached to an issue in the public issue
tracker. Could you provide a link?

You can send pricing-related issues to Peter at p...@google.com.

Cheers,
Brian

Thanks,
Andrius

On 6 December 2011 22:56, Brian Quinlan bquin...@google.com wrote:

Hey Andrius,

On Wed, Dec 7, 2011 at 9:14 AM, Andrius A andriu...@gmail.com wrote:
You are not the only one, I don't like the fact that google puts a blame
on
MS, there is something wrong with GAE constantly going on and they
should
admit that and explain us in more details what's happening.

There is a known bug where the combination of Python 2.7 and enabling
concurrent requests causes large latency increases. The Python 2.7
runtime is still experimental and bugs like this are expected.

The App Engine Administration Console uses the MS datastore (for a
variety of esoteric reasons) so its latency spikes will often
correlate with MS latency spikes.

We have a form for issue like this to be noticed, please fill in to get
more
attention:

https://docs.google.com/spreadsheet/viewform?hl=en_USformkey=dC1DMmpPSU1WZnk0d1FMa3JmNXIwaGc6MQ#gid=0

Can I also suggest that people file bugs using the public issue tracker
at:
http://code.google.com/p/googleappengine/issues ?

I really believe into GAE and still think this is a great a platform,
but it
has issues which need be admitted and taken care of.

Cheers,
Brian

On 6 December 2011 17:41, Sarang sar...@mycontactid.net wrote:

Glad to see this post as I am also suffering at the hands of GAE for
the
last month now. Every now and then DeadlineExceeded errors shoot up
giving
response times of over a minute!

See my thread:

https://groups.google.com/forum/?hl=en#!topic/google-appengine/NpqadsOUyZ4

Sarang

--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/-JsYHq24EyMJ.

--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

--
You received this

Re: [google-appengine] DataStore Tiering Was Dealine Exceeded Errors

thanks Brandon I am thinking about it all the time. I wonder if you could
help me with this particular case. I am selling an item in the auction,
which has a time to expire and the price changes with each bid received.
Lets say I have 50 users bidding in the same time.
How could I use Cache here to calculate the item price on demand within a
second using HRD? With MS I can do it easily without using non-reliable
memcache. Or would you say that GAE is not suitable for that?

Thank you,
Andrius

On 7 December 2011 00:10, Brandon Wirtz drak...@digerat.com wrote:

 I posted this logic once before but people only listen when they are
 ranting.

 ** **

 You can only change an entity once per second. You can however maintain a
 data store “Cache” that will save you money, writes, and prevent you from
 ever seeing write limits.

 ** **

 By maintaining a Small DataStore of “Writes To commit”  things you want to
 write to data store permanently can be put in to a data store which you
 increment the task on.

 ** **

 When you do a Read from the datastore you check that the writes to commit
 don’t supersede the data in the main data store.

 ** **

 This will save you on indexing operations, and reduce your write
 throttling and if done correctly can even move most of your reads to Memory
 rather than DataStore.

 ** **

 This will increase your number of reads. Which will increase your Costs
 slightly over MS, but likely is better than having MS down or MS Read
 OnlyTime.

 ** **

 ** **

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Andrius A
 *Sent:* Tuesday, December 06, 2011 3:44 PM
 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Re: DeadlineExceeded errors haunting
 every now and then

 ** **

 In docs it says that HRD is more costly *The High Replication Datastore
 costs more due to the additional replication (see billing page for pricing
 details). Due to the higher cost, we recommend High Replication Datastore
 primarily for those developers building critical App Engine applications
 that require the highest availability*. 

 It also says *queries on a single guestbook to be strongly consistent,
 but also limits changes to the guestbook to 1 write per second (the
 supported limit for entity groups). Therefore, writing to a single entity
 group per guestbook is not ideal when high usage is expected. If your app
 is likely to encounter heavy write usage, consider using another means. For
 example, you can put recent posts in memcache with an expiration, and then
 display a mix of recent posts from memcache and posts retrieved from the
 datastore.* 

 How could we trust by putting data to memcache if we know it can be
 evicted any time? To use HRD is not viable for applications which need
 strong consistency for high rate of puts. At the moment MS is perfect what
 it does but bloody thing keeps dying.. I can't believe there is no way to
 improve MS and you are switching to HRD. In long term HRD can cause more
 problems for applications parts were it wasn't aticipating to receive
 higher rate of inserts, and having a limit of 1 write per second is
 a disastrous.

 On 6 December 2011 23:29, Gregory D'alesandre gr...@google.com wrote:***
 *

 Hi Kenneth, 

 ** **

 We are planning to add blobstore migration at some point but it won't
 likely be until the middle of Q1 at the earliest.

 ** **

 Greg

 ** **

 On Tue, Dec 6, 2011 at 3:27 PM, Kenneth kennet...@aladdinschools.com
 wrote:

 Hi Greg,

 Can you give us some indication if the migration tool is ever going to
 include blobstore migration? That's the only thing holding me back. I know
 I can do it myself but I'd rather not. I see now that it has gone ga, have
 you stuck a fork in it and called it done?

 Thanks


 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.

 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/f7KzNP-pCuwJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 ** **

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

 ** **

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to

Re: [google-appengine] Re: DeadlineExceeded errors haunting every now and then

Hi Brian,

just an idea, do you think you could build highly reliable simple but fast
hash table similar to memcache but that could guarantee limited number of
single entities permanent existence per application?
Having that and HRD would make things work brilliantly!

On 7 December 2011 00:13, Brian Quinlan bquin...@google.com wrote:

 On Wed, Dec 7, 2011 at 10:44 AM, Andrius A andriu...@gmail.com wrote:
  In docs it says that HRD is more costly The High Replication Datastore
  costs more due to the additional replication (see billing page for
 pricing
  details). Due to the higher cost, we recommend High Replication Datastore
  primarily for those developers building critical App Engine applications
  that require the highest availability.

 I've filed a bug against this documentation.

  It also says queries on a single guestbook to be strongly consistent,
 but
  also limits changes to the guestbook to 1 write per second (the supported
  limit for entity groups). Therefore, writing to a single entity group per
  guestbook is not ideal when high usage is expected. If your app is
 likely to
  encounter heavy write usage, consider using another means. For example,
 you
  can put recent posts in memcache with an expiration, and then display a
 mix
  of recent posts from memcache and posts retrieved from the datastore.
  How could we trust by putting data to memcache if we know it can be
 evicted
  any time?

 You wouldn't as definitive storage but it is reasonable to cache
 recent posts there.

  To use HRD is not viable for applications which need strong
  consistency for high rate of puts.

 You may have to think about how your organize your data in order to
 satisfy your consistency and throughput requirements but I am very
 skeptical when you say that it is simply not viable.

 Google, for example, has been successful using the same technology for
 many of its own high-volume properties.

  At the moment MS is perfect what it does
  but bloody thing keeps dying..

 That is an architectural problem that is very hard to fix with making
 the same kinds of trade-offs that we made when implementing HRD .

 Cheers,
 Brian

  I can't believe there is no way to improve MS
  and you are switching to HRD. In long term HRD can cause more problems
 for
  applications parts were it wasn't aticipating to receive higher rate of
  inserts, and having a limit of 1 write per second is a disastrous.
 
 
  On 6 December 2011 23:29, Gregory D'alesandre gr...@google.com wrote:
 
  Hi Kenneth,
 
  We are planning to add blobstore migration at some point but it won't
  likely be until the middle of Q1 at the earliest.
 
  Greg
 
 
  On Tue, Dec 6, 2011 at 3:27 PM, Kenneth kennet...@aladdinschools.com
  wrote:
 
  Hi Greg,
 
  Can you give us some indication if the migration tool is ever going to
  include blobstore migration? That's the only thing holding me back. I
 know I
  can do it myself but I'd rather not. I see now that it has gone ga,
 have you
  stuck a fork in it and called it done?
 
  Thanks
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To view this discussion on the web visit
  https://groups.google.com/d/msg/google-appengine/-/f7KzNP-pCuwJ.
  To post to this group, send email to google-appengine@googlegroups.com
 .
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
 
  --
  You received this message because you are subscribed to the Google
 Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
 
  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at

RE: [google-appengine] DataStore Tiering Was Dealine Exceeded Errors

You should be doing something like 

 

12:00:00   AuctionID451 Brandons Dirty Socks  $84
Andrius

12:00:01   AuctionID451 Brandons Dirty Socks  $86
Greg

12:00:02   AuctionID451 Brandons Dirty Socks  $94
Brian

12:00:02   AuctionID453 Adult Mermaid Costume
$134   Brandon

12:00:03   AuctionID451 Brandons Dirty Socks  $104
Andrius

 

Every so often when this table gets too big, or too old you would commit it
to the larger datastore

 

When you ask for the data you would ask the Big data store for the price and
then check if there was a more recent price in the small data store.

MemCache and local instance memory could be used to check the Small data
store without a Read to the data store as one level of cache, but could also
contain the list of auctions that it contains the most recent data on so
that you wouldn't have to hit the main data store.

 

Over all this should be faster, cost less, and more reliable than MS.

 

 

 

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Andrius A
Sent: Tuesday, December 06, 2011 4:26 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] DataStore Tiering Was Dealine Exceeded
Errors

 

thanks Brandon I am thinking about it all the time. I wonder if you could
help me with this particular case. I am selling an item in the auction,
which has a time to expire and the price changes with each bid received.
Lets say I have 50 users bidding in the same time.

How could I use Cache here to calculate the item price on demand within a
second using HRD? With MS I can do it easily without using non-reliable
memcache. Or would you say that GAE is not suitable for that?

 

Thank you,

Andrius

 

On 7 December 2011 00:10, Brandon Wirtz drak...@digerat.com wrote:

I posted this logic once before but people only listen when they are
ranting.

 

You can only change an entity once per second. You can however maintain a
data store Cache that will save you money, writes, and prevent you from
ever seeing write limits.

 

By maintaining a Small DataStore of Writes To commit  things you want to
write to data store permanently can be put in to a data store which you
increment the task on.

 

When you do a Read from the datastore you check that the writes to commit
don't supersede the data in the main data store.

 

This will save you on indexing operations, and reduce your write throttling
and if done correctly can even move most of your reads to Memory rather than
DataStore.

 

This will increase your number of reads. Which will increase your Costs
slightly over MS, but likely is better than having MS down or MS Read
OnlyTime.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Andrius A
Sent: Tuesday, December 06, 2011 3:44 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: DeadlineExceeded errors haunting every
now and then

 

In docs it says that HRD is more costly The High Replication Datastore
costs more due to the additional replication (see billing page for pricing
details). Due to the higher cost, we recommend High Replication Datastore
primarily for those developers building critical App Engine applications
that require the highest availability. 

It also says queries on a single guestbook to be strongly consistent, but
also limits changes to the guestbook to 1 write per second (the supported
limit for entity groups). Therefore, writing to a single entity group per
guestbook is not ideal when high usage is expected. If your app is likely to
encounter heavy write usage, consider using another means. For example, you
can put recent posts in memcache with an expiration, and then display a mix
of recent posts from memcache and posts retrieved from the datastore. 

How could we trust by putting data to memcache if we know it can be evicted
any time? To use HRD is not viable for applications which need strong
consistency for high rate of puts. At the moment MS is perfect what it does
but bloody thing keeps dying.. I can't believe there is no way to improve MS
and you are switching to HRD. In long term HRD can cause more problems for
applications parts were it wasn't aticipating to receive higher rate of
inserts, and having a limit of 1 write per second is a disastrous.

On 6 December 2011 23:29, Gregory D'alesandre gr...@google.com wrote:

Hi Kenneth, 

 

We are planning to add blobstore migration at some point but it won't likely
be until the middle of Q1 at the earliest.

 

Greg

 

On Tue, Dec 6, 2011 at 3:27 PM, Kenneth kennet...@aladdinschools.com
wrote:

Hi Greg,

Can you give us some indication if the migration tool is ever going to
include blobstore migration? That's the only thing holding me back. I know I
can do it myself but I'd rather not. I see now that it has gone ga, have you
stuck a fork in it and called it

Re: [google-appengine] Re: DeadlineExceededError on HRD and sky-high latency (python27+threadsafe)

Thanks Brian for sending this info, as it is really tricky to contact you
guys.

This is the issue I am mostly concerned:
http://code.google.com/p/googleappengine/issues/detail?id=6309can=1q=175mscolspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log

On 7 December 2011 00:24, Brian Quinlan bquin...@google.com wrote:

Hi Andrius,

On Wed, Dec 7, 2011 at 11:15 AM, Andrius A andriu...@gmail.com wrote:
Hey Brian,

thank you for your reply. I am still on 2.5 since 2.7 is still
experimental,
and I don't really believe that MS is everything to blame, there are
issues
with requests latencies, which I have monitored and there was nothing
wrong
with datastore. I have been posting about that here and opened a ticket
but
have not received an answer unfortunately.

I couldn't see your name attached to an issue in the public issue
tracker. Could you provide a link?

I am moaning here all the time because we shouldn't be charged so much as
you confirmed already many times that there are issues and 2.7 is still
faulty.
The main point here is that we should not pay more just because GAE is
now
called production ready. I will be repeating that until you go back to
preview release or admit that by dropping the prices or improve the
reliability.

You can send pricing-related issues to Peter at p...@google.com.

Cheers,
Brian

Thanks,
Andrius

On 6 December 2011 22:56, Brian Quinlan bquin...@google.com wrote:

Hey Andrius,

On Wed, Dec 7, 2011 at 9:14 AM, Andrius A andriu...@gmail.com wrote:
You are not the only one, I don't like the fact that google puts a
blame
on
MS, there is something wrong with GAE constantly going on and they
should
admit that and explain us in more details what's happening.

There is a known bug where the combination of Python 2.7 and enabling
concurrent requests causes large latency increases. The Python 2.7
runtime is still experimental and bugs like this are expected.

The App Engine Administration Console uses the MS datastore (for a
variety of esoteric reasons) so its latency spikes will often
correlate with MS latency spikes.

We have a form for issue like this to be noticed, please fill in to
get
more
attention:

https://docs.google.com/spreadsheet/viewform?hl=en_USformkey=dC1DMmpPSU1WZnk0d1FMa3JmNXIwaGc6MQ#gid=0

Can I also suggest that people file bugs using the public issue tracker
at:
http://code.google.com/p/googleappengine/issues ?

I really believe into GAE and still think this is a great a platform,
but it
has issues which need be admitted and taken care of.

Cheers,
Brian

On 6 December 2011 17:41, Sarang sar...@mycontactid.net wrote:

Glad to see this post as I am also suffering at the hands of GAE for
the
last month now. Every now and then DeadlineExceeded errors shoot up
giving
response times of over a minute!

See my thread:

https://groups.google.com/forum/?hl=en#!topic/google-appengine/NpqadsOUyZ4

Sarang

--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/-JsYHq24EyMJ.

To post to this group, send email to
google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To post to this group, send email to
google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google
Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.

Re: [google-appengine] Re: DeadlineExceeded errors haunting every now and then

On Wed, Dec 7, 2011 at 11:34 AM, Andrius A andriu...@gmail.com wrote:
 Hi Brian,

 just an idea, do you think you could build highly reliable simple but fast
 hash table similar to memcache but that could guarantee limited number of
 single entities permanent existence per application?

Probably not.

Highly reliable suggests that you synchronously replicate your data
across several machines and data centers (assuming that you want
consistency) but it is hard to make that fast.

Cheers,
Brian

 Having that and HRD would make things work brilliantly!


 On 7 December 2011 00:13, Brian Quinlan bquin...@google.com wrote:

 On Wed, Dec 7, 2011 at 10:44 AM, Andrius A andriu...@gmail.com wrote:
  In docs it says that HRD is more costly The High Replication Datastore
  costs more due to the additional replication (see billing page for
  pricing
  details). Due to the higher cost, we recommend High Replication
  Datastore
  primarily for those developers building critical App Engine applications
  that require the highest availability.

 I've filed a bug against this documentation.

  It also says queries on a single guestbook to be strongly consistent,
  but
  also limits changes to the guestbook to 1 write per second (the
  supported
  limit for entity groups). Therefore, writing to a single entity group
  per
  guestbook is not ideal when high usage is expected. If your app is
  likely to
  encounter heavy write usage, consider using another means. For example,
  you
  can put recent posts in memcache with an expiration, and then display a
  mix
  of recent posts from memcache and posts retrieved from the datastore.
  How could we trust by putting data to memcache if we know it can be
  evicted
  any time?

 You wouldn't as definitive storage but it is reasonable to cache
 recent posts there.

  To use HRD is not viable for applications which need strong
  consistency for high rate of puts.

 You may have to think about how your organize your data in order to
 satisfy your consistency and throughput requirements but I am very
 skeptical when you say that it is simply not viable.

 Google, for example, has been successful using the same technology for
 many of its own high-volume properties.

  At the moment MS is perfect what it does
  but bloody thing keeps dying..

 That is an architectural problem that is very hard to fix with making
 the same kinds of trade-offs that we made when implementing HRD .

 Cheers,
 Brian

  I can't believe there is no way to improve MS
  and you are switching to HRD. In long term HRD can cause more problems
  for
  applications parts were it wasn't aticipating to receive higher rate of
  inserts, and having a limit of 1 write per second is a disastrous.
 
 
  On 6 December 2011 23:29, Gregory D'alesandre gr...@google.com wrote:
 
  Hi Kenneth,
 
  We are planning to add blobstore migration at some point but it won't
  likely be until the middle of Q1 at the earliest.
 
  Greg
 
 
  On Tue, Dec 6, 2011 at 3:27 PM, Kenneth kennet...@aladdinschools.com
  wrote:
 
  Hi Greg,
 
  Can you give us some indication if the migration tool is ever going to
  include blobstore migration? That's the only thing holding me back. I
  know I
  can do it myself but I'd rather not. I see now that it has gone ga,
  have you
  stuck a fork in it and called it done?
 
  Thanks
 
  --
  You received this message because you are subscribed to the Google
  Groups
  Google App Engine group.
  To view this discussion on the web visit
  https://groups.google.com/d/msg/google-appengine/-/f7KzNP-pCuwJ.
  To post to this group, send email to
  google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
 
  --
  You received this message because you are subscribed to the Google
  Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.
 
 
  --
  You received this message because you are subscribed to the Google
  Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


 --
 You received this

Re: [google-appengine] DataStore Tiering Was Dealine Exceeded Errors

Thanks Brandon, but in your example if I had 50 users bidding in 1 second
it would take another 50 sec for the final result to appear. That's not
going to work unfortunately because there will be another 20 users bidding
on next second. or I am not catching something?

What do you mean by small and big datastore? How do you implement it?

and I can't keep data in backends instance memory or memcache, because
backend can be restarted or memcache evicted anytime. I found that fastest
way to have this working is having 50 sharded counters in MS, so I can
really process the bids within a second!

On 7 December 2011 00:40, Brandon Wirtz drak...@digerat.com wrote:

 You should be doing something like 

 ** **

 12:00:00   AuctionID451 Brandons Dirty Socks
 $84 Andrius

 12:00:01   AuctionID451 Brandons Dirty Socks
 $86 Greg

 12:00:02   AuctionID451 Brandons Dirty Socks
 $94 Brian

 12:00:02   AuctionID453 Adult Mermaid Costume
 $134   Brandon

 12:00:03   AuctionID451 Brandons Dirty Socks
 $104   Andrius

 ** **

 Every so often when this table gets too big, or too old you would commit
 it to the larger datastore

 ** **

 When you ask for the data you would ask the Big data store for the price
 and then check if there was a more recent price in the small data store.**
 **

 MemCache and local instance memory could be used to check the Small data
 store without a Read to the data store as one level of cache, but could
 also contain the list of auctions that it contains the most recent data on
 so that you wouldn’t have to hit the main data store.

 ** **

 Over all this should be faster, cost less, and more reliable than MS.

 ** **

 ** **

 ** **

 ** **

 ** **

 ** **

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Andrius A
 *Sent:* Tuesday, December 06, 2011 4:26 PM
 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] DataStore Tiering Was Dealine Exceeded
 Errors

 ** **

 thanks Brandon I am thinking about it all the time. I wonder if you could
 help me with this particular case. I am selling an item in the auction,
 which has a time to expire and the price changes with each bid received.
 Lets say I have 50 users bidding in the same time.

 How could I use Cache here to calculate the item price on demand within
 a second using HRD? With MS I can do it easily without using non-reliable
 memcache. Or would you say that GAE is not suitable for that?

 ** **

 Thank you,

 Andrius

 ** **

 On 7 December 2011 00:10, Brandon Wirtz drak...@digerat.com wrote:

 I posted this logic once before but people only listen when they are
 ranting.

  

 You can only change an entity once per second. You can however maintain a
 data store “Cache” that will save you money, writes, and prevent you from
 ever seeing write limits.

  

 By maintaining a Small DataStore of “Writes To commit”  things you want to
 write to data store permanently can be put in to a data store which you
 increment the task on.

  

 When you do a Read from the datastore you check that the writes to commit
 don’t supersede the data in the main data store.

  

 This will save you on indexing operations, and reduce your write
 throttling and if done correctly can even move most of your reads to Memory
 rather than DataStore.

  

 This will increase your number of reads. Which will increase your Costs
 slightly over MS, but likely is better than having MS down or MS Read
 OnlyTime.

  

  

 *From:* google-appengine@googlegroups.com [mailto:
 google-appengine@googlegroups.com] *On Behalf Of *Andrius A
 *Sent:* Tuesday, December 06, 2011 3:44 PM
 *To:* google-appengine@googlegroups.com
 *Subject:* Re: [google-appengine] Re: DeadlineExceeded errors haunting
 every now and then

  

 In docs it says that HRD is more costly *The High Replication Datastore
 costs more due to the additional replication (see billing page for pricing
 details). Due to the higher cost, we recommend High Replication Datastore
 primarily for those developers building critical App Engine applications
 that require the highest availability*. 

 It also says *queries on a single guestbook to be strongly consistent,
 but also limits changes to the guestbook to 1 write per second (the
 supported limit for entity groups). Therefore, writing to a single entity
 group per guestbook is not ideal when high usage is expected. If your app
 is likely to encounter heavy write usage, consider using another means. For
 example, you can put recent posts in memcache with an expiration, and then
 display a mix of recent posts from memcache and posts retrieved from the
 datastore.* 

 How could we trust by putting data to memcache if we know it can be
 evicted any time? To use

[google-appengine] contact to send a pricing related issues

For developers who tried to contact a google regarding the pricing and
haven't received an answer we now have a direct email I believe.

Thanks for Brian ( from Google) for kindly giving us this contact.

You can send pricing-related issues to Peter at p...@google.com.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Downtime on 6 Dec?

We've seen some announcements, but one would hope there would be something 
about planned downtime on the status 
pagehttp://code.google.com/status/appengine. 
Is the datastore still going to maintenance mode at 5:30?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/tFMA-M0ccl8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: DeadlineExceededError on HRD and sky-high latency (python27+threadsafe)

2011-12-06 Thread Felippe Bueno

On Tue, Dec 6, 2011 at 8:56 PM, Brian Quinlan bquin...@google.com wrote:

There is a known bug where the combination of Python 2.7 and enabling
concurrent requests causes large latency increases. The Python 2.7
runtime is still experimental and bugs like this are expected.

Ok, I understand that.

Greg, could you please let us know why we have the new full-sized instance
hour for python and still not have python concurrent requests ?

I'm asking because you
wrotehttp://groups.google.com/group/google-appengine/browse_thread/thread/a1bfa432e0c002a7/739169f799d8e69a
:

Q: Will there be a solution for Python concurrency? Will this require any
code changes?
Python concurrency will be handled by our release of Python 2.7 on App
Engine. We’ve heard a lot of feedback from our Python users who are
worried
that the incentive is to move to Java because of its support for concurrent
requests, so we’ve made a change to the new pricing to account for
that. *While
Python 2.7 support is currently in progress it is not yet done *so we will
be **
providing a half-sized instance for Python (at half the price) until Python
2.7 is released.* *

Ok, I know. It was already released (as experimental), but I thought that
the purpose to holding full-sized instances was to give to python users
the concurrent requests.

Thanks

[google-appengine] Re: db.get() on a list of keys being split into multiple get_async() calls as opposed to a single get_async() call

2011-12-06 Thread Petey

I'm seeing the same issue and it's not because production gets every
entity. If that were the case then my app stats would show response
times for each get around 10ms because that is about what our single
gets usually take. Instead I'm seeing more like 300ms per get it is
doing, which is what the previous response time was for the bulk get.
And now it's showing several datastore_v3.Get calls in app stats that
are taking ~300ms each. We haven't changed our code and it just showed
up recently. Previously those 10 datastore_v3.Get calls in app stats
showed as one call. So either app stats is displaying things
differently or the bulk get is actually being called multiple times
when it only should be called once.



On Dec 3, 5:48 am, Timofey Koolin timo...@koolin.ru wrote:
 Production get every entity too. But do it in parallel mode - at the same 
 time.

 2011/12/3 Bryce Cutt pandas...@gmail.com:

  Did the production server change the way it handles a db.get() on a
  list of keys? I believe it used to do a single get_async on the list
  of keys but now it appears that it is doing a get_async for each key
  (according to appstats). The development server is still doing a
  single get_async for the entire list.

  --
  You received this message because you are subscribed to the Google Groups 
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to 
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/google-appengine?hl=en.

 --
 Blog:www.rekby.ru

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Downtime on 6 Dec?

There ya go! The status page still doesn't say anything about scheduled 
maintenance - it says 'investigating' datastore.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/O5jXx7Az_FUJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Downtime on 6 Dec?

2011-12-06 Thread Amy Unruh

hi,

Yes, it is occurring. These announcements are posted on the Google
Appengine Downtime Notify list.  See:
https://groups.google.com/forum/#!msg/google-appengine-downtime-notify/SvyazgjKcKk/EgWyyKzyD0wJ

On Tue, Dec 6, 2011 at 5:43 PM, John jwb...@gmail.com wrote:

 There ya go! The status page still doesn't say anything about scheduled
 maintenance - it says 'investigating' datastore.

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/O5jXx7Az_FUJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Downtime on 6 Dec?

Here we go, this bloody thing never tells the truth! :)

On 7 December 2011 01:43, John jwb...@gmail.com wrote:

 There ya go! The status page still doesn't say anything about scheduled
 maintenance - it says 'investigating' datastore.

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/O5jXx7Az_FUJ.

 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

RE: [google-appengine] DataStore Tiering Was Dealine Exceeded Errors

My code logic works at whatever resolution you need it to. those could be
10,000/th of a second if you need.  

 

Small and Big is really meant in terms of Permanent and Short Term.
EveryTime you Commit the writes you would start over on a new small Data
Store

 

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Andrius A
Sent: Tuesday, December 06, 2011 5:01 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] DataStore Tiering Was Dealine Exceeded
Errors

 

Thanks Brandon, but in your example if I had 50 users bidding in 1 second it
would take another 50 sec for the final result to appear. That's not going
to work unfortunately because there will be another 20 users bidding on next
second. or I am not catching something?

 

What do you mean by small and big datastore? How do you implement it?

 

and I can't keep data in backends instance memory or memcache, because
backend can be restarted or memcache evicted anytime. I found that fastest
way to have this working is having 50 sharded counters in MS, so I can
really process the bids within a second!

On 7 December 2011 00:40, Brandon Wirtz drak...@digerat.com wrote:

You should be doing something like 

 

12:00:00   AuctionID451 Brandons Dirty Socks  $84
Andrius

12:00:01   AuctionID451 Brandons Dirty Socks  $86
Greg

12:00:02   AuctionID451 Brandons Dirty Socks  $94
Brian

12:00:02   AuctionID453 Adult Mermaid Costume
$134   Brandon

12:00:03   AuctionID451 Brandons Dirty Socks  $104
Andrius

 

Every so often when this table gets too big, or too old you would commit it
to the larger datastore

 

When you ask for the data you would ask the Big data store for the price and
then check if there was a more recent price in the small data store.

MemCache and local instance memory could be used to check the Small data
store without a Read to the data store as one level of cache, but could also
contain the list of auctions that it contains the most recent data on so
that you wouldn't have to hit the main data store.

 

Over all this should be faster, cost less, and more reliable than MS.

 

 

 

 

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Andrius A
Sent: Tuesday, December 06, 2011 4:26 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] DataStore Tiering Was Dealine Exceeded
Errors

 

thanks Brandon I am thinking about it all the time. I wonder if you could
help me with this particular case. I am selling an item in the auction,
which has a time to expire and the price changes with each bid received.
Lets say I have 50 users bidding in the same time.

How could I use Cache here to calculate the item price on demand within a
second using HRD? With MS I can do it easily without using non-reliable
memcache. Or would you say that GAE is not suitable for that?

 

Thank you,

Andrius

 

On 7 December 2011 00:10, Brandon Wirtz drak...@digerat.com wrote:

I posted this logic once before but people only listen when they are
ranting.

 

You can only change an entity once per second. You can however maintain a
data store Cache that will save you money, writes, and prevent you from
ever seeing write limits.

 

By maintaining a Small DataStore of Writes To commit  things you want to
write to data store permanently can be put in to a data store which you
increment the task on.

 

When you do a Read from the datastore you check that the writes to commit
don't supersede the data in the main data store.

 

This will save you on indexing operations, and reduce your write throttling
and if done correctly can even move most of your reads to Memory rather than
DataStore.

 

This will increase your number of reads. Which will increase your Costs
slightly over MS, but likely is better than having MS down or MS Read
OnlyTime.

 

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Andrius A
Sent: Tuesday, December 06, 2011 3:44 PM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: DeadlineExceeded errors haunting every
now and then

 

In docs it says that HRD is more costly The High Replication Datastore
costs more due to the additional replication (see billing page for pricing
details). Due to the higher cost, we recommend High Replication Datastore
primarily for those developers building critical App Engine applications
that require the highest availability. 

It also says queries on a single guestbook to be strongly consistent, but
also limits changes to the guestbook to 1 write per second (the supported
limit for entity groups). Therefore, writing to a single entity group per
guestbook is not ideal when high usage is expected. If your app is likely to
encounter heavy write usage, consider using another means. For example, you
can put recent

Re: [google-appengine] Re: Downtime on 6 Dec?

I'm well aware of the announcements DL. The point is the status page is not 
updated. Letting things slide was fine when we got lots of resources for 
very little money. Now that's changed, and the service should meet some 
reasonable level of quality.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/GcPGNuvanqsJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Downtime on 6 Dec?

Yup. Status page http://code.google.com/status/appengine still says 
investigating; Anamoly.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/gDY14tUuImoJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: more than 12M operations in less than 10 hrs

2011-12-06 Thread James X Nelson

Appstats is what you are looking for.

http://code.google.com/appengine/docs/python/tools/appstats.html
http://code.google.com/appengine/docs/java/tools/appstats.html

Also look at http://code.google.com/appengine/docs/billing.html

You will see that different kinds of queries and operations use a
different number of operations.

If you can key-get your entities instead of querying on them, you use
half as many operations.

If you wrap your puts and gets with memcache put and get, you can skip
lots of excess DS reads.

If you can somehow use browser caching for some of your queries, you
can cut not only DS reads but http requests as well.
{hint: if you keep an updated timestamp, you can use this as last-
modified or to generate etags so browser can cache it, but get updates
when available}.


Anyway,

these are just general solutions.

Plugin appstats and it will show you where all your excess usage is.


On Dec 6, 11:44 am, cloudpre pbx.ku...@gmail.com wrote:
 Hi - my app is taking more than 12M reads in less than 10 hours- is
 there a tuner to check which calls are exhibiting this weird behavior?

 I mean how do we tune this?

 Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Downtime on 6 Dec?

2011-12-06 Thread DataSurfer

Is this downtime supposed to prevent deployments of high replication
apps?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Unable to use GAE for my domain?

2011-12-06 Thread Christoph Blank

Hey there,

I'm a bit upset, I wanted to switch my S3 hosted sites to GAE.

I have a primary domain in Google Apps, and an additional one
fubar.tld as alias.
Now I want to add an application on fubar.tld to GAE..

As far as I read this is not possible without losing the domain in the
existing primary domain as an alias.
Are you kidding me or is this the truth?

I've learned the hard way that it is a pain to use google apps
accounts as normal google accounts, and now this?
Do I really have to use amazon only because of this?

Thanks a lot for any help,
Christoph

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] is there any way to reuse a deleted application ID?

2011-12-06 Thread Guenlay

Hello,
I deleted my application few days ago because I want to use it with another 
account which I always login in my college PC, I didn't know Google 
reserved every deleted ID. Now the problem is the Application Identifier 
was my student ID, I need the domain name for my every continuous 
assessment.
Does Google have any customer service I can contact by email?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/paYps34Oa1wJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Communication Between 2 Google App Engine Apps

2011-12-06 Thread Joao L V Vasques

Hey guys,

I'm doing a project and I need to put 2 GAE Apps communicating. I know that 
a GAE can make HTTP requests but I can't make them to the App URL where I 
have my Python get method defined. Is there any way to make 2 GAEP Apps 
communicate?

Thanks for the attention

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/aqEYqAqMFG4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] App Engine Launcher doesn't work well with auto detect proxy

2011-12-06 Thread Guenlay

everytime I run my app, I need to double click the Browse button to display 
the webpage, otherwise I just get a blank endless loading page

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/Q5JVu70DVYMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Time lag with logging

2011-12-06 Thread Jane Abernethy

Hi,

I am noticing a bit of a time lag with logging - I'll log something, call
the method and the logs wont update for about 15 minutes. Is anyone else
experiencing this?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Datastore applied

2011-12-06 Thread Phil McDonnell

I'm in java on GWT app engine and I'm having some intermittent issues where
it looks like datastore put() commands are not being reflected by
subsequent get() calls.  From my reading I thought this was not possible.
 Am I mistaken?

Here's what I'm doing...

// STATE A
Entity x = ...
datastore.put(x);
// STATE B
...
x = ... some new values ...
datastore.put(x);
// STATE C
...
datastore.get(keyOfX);


The final get() sometimes returns an x of state C and sometimes it returns
an x of state B.  Any idea why this happens?  I'm assuming it's some sort
of concurrency thing within my app, but perhaps I've misunderstood the
datastore guarantees?  Should the get() always get an x from state C?

Thanks,
Phil

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: 1.6.1 Pre-release SDKs are live



And exactly which front-end instance class have we all currently been using? I 
hope the lowest one-- I don't want to suffer a performance hit and increased 
expenses.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/y0mS6iOJyVkJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Downtime on 6 Dec?

2011-12-06 Thread campos

Maintenance finished? Mine is still read only.

On Dec 7, 9:26 am, John jwb...@gmail.com wrote:
 We've seen some announcements, but one would hope there would be something
 about planned downtime on the status 
 pagehttp://code.google.com/status/appengine.
 Is the datastore still going to maintenance mode at 5:30?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Is the datastore coming back online at 6:30 PM or not?

If not then approximately how long?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/uKjsIinA8FsJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Timeouts to Paypal

Hey, we had to set up a reverse NGINX proxy through AWS EC2 to communicate 
with the PayPal production server. We could never get it to work reliably 
directly from GAE. I don't know why.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/obH2LepD1KMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Downtime on 6 Dec?

2011-12-06 Thread campos

My domain is down now?

I am ready to change my hosting. Google is charging me over $150 a
week but is providing non-stable hosting!

On Dec 7, 10:31 am, campos huwen...@gmail.com wrote:
 Maintenance finished? Mine is still read only.

 On Dec 7, 9:26 am, John jwb...@gmail.com wrote:







  We've seen some announcements, but one would hope there would be something
  about planned downtime on the status 
  pagehttp://code.google.com/status/appengine.
  Is the datastore still going to maintenance mode at 5:30?

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Datastore applied

2011-12-06 Thread Jeff Schnitzer

Are you executing in a transaction?  get()s in a transaction always show
the entity frozen in time as of the start of the transaction.  Even if
you put() a change, get() will show the old value until the transaction is
committed.

Jeff

On Tue, Dec 6, 2011 at 10:26 PM, Phil McDonnell
phil.a.mcdonn...@gmail.comwrote:

 I'm in java on GWT app engine and I'm having some intermittent issues
 where it looks like datastore put() commands are not being reflected by
 subsequent get() calls.  From my reading I thought this was not possible.
  Am I mistaken?

 Here's what I'm doing...

 // STATE A
 Entity x = ...
 datastore.put(x);
 // STATE B
 ...
 x = ... some new values ...
 datastore.put(x);
 // STATE C
 ...
 datastore.get(keyOfX);


 The final get() sometimes returns an x of state C and sometimes it returns
 an x of state B.  Any idea why this happens?  I'm assuming it's some sort
 of concurrency thing within my app, but perhaps I've misunderstood the
 datastore guarantees?  Should the get() always get an x from state C?

 Thanks,
 Phil

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




-- 
We are the 20%

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Datastore applied

2011-12-06 Thread Phil McDonnell

No, the puts/gets are executing normally.  I did try wrapping the puts() in
a transaction, but it didn't help.  If I wrap just the get() in a
transaction should that somehow trigger the puts() to apply?  Is there some
way to wait?

Thanks so much,
Phil

On Tue, Dec 6, 2011 at 10:17 PM, Jeff Schnitzer j...@infohazard.org wrote:

 Are you executing in a transaction?  get()s in a transaction always show
 the entity frozen in time as of the start of the transaction.  Even if
 you put() a change, get() will show the old value until the transaction is
 committed.

 Jeff

 On Tue, Dec 6, 2011 at 10:26 PM, Phil McDonnell 
 phil.a.mcdonn...@gmail.com wrote:

 I'm in java on GWT app engine and I'm having some intermittent issues
 where it looks like datastore put() commands are not being reflected by
 subsequent get() calls.  From my reading I thought this was not possible.
  Am I mistaken?

 Here's what I'm doing...

 // STATE A
 Entity x = ...
 datastore.put(x);
 // STATE B
 ...
 x = ... some new values ...
 datastore.put(x);
 // STATE C
 ...
 datastore.get(keyOfX);


 The final get() sometimes returns an x of state C and sometimes it
 returns an x of state B.  Any idea why this happens?  I'm assuming it's
 some sort of concurrency thing within my app, but perhaps I've
 misunderstood the datastore guarantees?  Should the get() always get an x
 from state C?

 Thanks,
 Phil

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.




 --
 We are the 20%

  --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Datastore applied

2011-12-06 Thread YuchTing


check http://code.google.com/status/appengine for GAE current status.

Is GAE datastore abnomal?

于 2011-12-7 10:26, Phil McDonnell 写道:

I'm in java on GWT app engine and I'm having some intermittent issues
where it looks like datastore put() commands are not being reflected by
subsequent get() calls.  From my reading I thought this was not
possible.  Am I mistaken?

Here's what I'm doing...

// STATE A
Entity x = ...
datastore.put(x);
// STATE B
...
x = ... some new values ...
datastore.put(x);
// STATE C
...
datastore.get(keyOfX);


The final get() sometimes returns an x of state C and sometimes it
returns an x of state B.  Any idea why this happens?  I'm assuming it's
some sort of concurrency thing within my app, but perhaps I've
misunderstood the datastore guarantees?  Should the get() always get an
x from state C?

Thanks,
Phil

--
You received this message because you are subscribed to the Google
Groups Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups Google 
App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] App Engine Community Update #15

2011-12-06 Thread Johan Euphrosine

Hi all,

I'm just back from vacation and so are App Engine community update.

I just posted App Engine Community Update #15 to Google+
https://plus.google.com/111042085517496880918/posts/Pos7tEtvBMY
-- 
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Downtime on 6 Dec?

2011-12-06 Thread sb

It always tells the truth.
Tomorrow there will be a different truth.

On Dec 6, 8:57 pm, Andrius A andriu...@gmail.com wrote:
 Here we go, this bloody thing never tells the truth! :)

 On 7 December 2011 01:43, John jwb...@gmail.com wrote:







  There ya go! The status page still doesn't say anything about scheduled
  maintenance - it says 'investigating' datastore.

   --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To view this discussion on the web visit
 https://groups.google.com/d/msg/google-appengine/-/O5jXx7Az_FUJ.

  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Downtime on 6 Dec?