Re: [google-appengine] Re: Outages?

[Alexander Trakhimenok]

Hey Ikai,

Our app id: petaclasses

QPS: 5-20 requests per second

Current instances in dashboard: 110 - 160
Usual instances: 8-15 

It's hard to say % of failed requests as we have also request that fail for 
other reasons (e.g. non existing pages, etc) and not sure how easily 
separate them.

By the way, are you guys considering to create a page where we can 
post/report this data in some structured way and "join" an issue so you can 
accumulate and understand the scale of an issue easily.

Alex

On Friday, 9 March 2012 15:15:35 UTC-4, Ikai Lan wrote:
>
> Hey everyone,
>
> Here are a few things that will help:
>
> 1. Application IDs (<--- if you have nothing else, at least provide this)
> 2. What is your QPS?
> 3. What % of your requests are errors?
>
> --
> Ikai Lan 
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com
>
>
>
> On Fri, Mar 9, 2012 at 7:24 AM, Ronoaldo José de Lana Pereira <
> rpere...@beneficiofacil.com.br> wrote:
>
>> +1 for seeing the same problems on my app.
>>
>> It started to be worse after maintenance on March 7.
>>
>> Em sexta-feira, 9 de março de 2012 08h33min36s UTC-3, Nikolai escreveu:
>>
>>> +1
>>> we had to move to our backup systems. Everything is full of 500 errors 
>>> or hardcore latency.
>>> Most of the 500 errors we see aren't even logged so this seems to be a 
>>> goole problem one abstraction layer above the app.
>>>
>>> And yes - sometimes we have got the same feeling, that we are the only 
>>> ones that use appengine in a production setting. You are not alone ;)
>>>
>>> regards,
>>> nikolai
>>>
>>> Am Dienstag, 6. März 2012 22:17:37 UTC+1 schrieb Adam Sherman:

 Am I the only one seeing short duration outages? They are being 
 reflected at:

 http://code.google.com/status/**appengine

 But I don't see anyone else complaining anywhere, so it makes me 
 worried.

 A.

  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msg/google-appengine/-/yixu1yAlMs4J.
>>
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to 
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at 
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/ErrbHpuYmWgJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Outages?

[Alexander Trakhimenok]

We are Python 2.5 (no concurrent).

Yes, it seems the start-up time is just crazy high for at least some or all 
instances.

I also noticed that there are lot's of instances that served just 1 request 
and have average latency 0ms and have QPS=0 average instance age about 8-9 
minutes (up to 11 minutes). For me it seems like an instance is created to 
serve static content and not used anymore and stays here until it die in a 
while.

At the moment we have 264 active instances and it's killing our budget :( - 
see the screenshot attached. We had 2 hours downtime due to exceeded budget.

Alex

On Friday, 9 March 2012 15:57:27 UTC-4, Ikai Lan wrote:
>
> Alex, to answer that question: yes. We are looking to revamp the 
> production issues tracker which is far from optimal. When users can join or 
> aggregate issues, it allows us to quickly separate actual infrastructure 
> hiccups from user code issues.
>
> Thanks for the info! Is there any other behavior you can report? Does it 
> sound reasonable that you have 110-160 instances because of long startup 
> teams leading to more instances required to serve the same load? Are you 
> Python, Java or Go, and do you have concurrent requests enabled?
>
> --
> Ikai Lan 
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com
>
>
>
> On Fri, Mar 9, 2012 at 11:48 AM, Alexander Trakhimenok <
> alexander.trakhime...@gmail.com> wrote:
>
>> Hey Ikai,
>>
>> Our app id: petaclasses
>>
>> QPS: 5-20 requests per second
>>
>> Current instances in dashboard: 110 - 160
>> Usual instances: 8-15 
>>
>> It's hard to say % of failed requests as we have also request that fail 
>> for other reasons (e.g. non existing pages, etc) and not sure how easily 
>> separate them.
>>
>> By the way, are you guys considering to create a page where we can 
>> post/report this data in some structured way and "join" an issue so you can 
>> accumulate and understand the scale of an issue easily.
>>
>> Alex
>>
>>
>> On Friday, 9 March 2012 15:15:35 UTC-4, Ikai Lan wrote:
>>>
>>> Hey everyone,
>>>
>>> Here are a few things that will help:
>>>
>>> 1. Application IDs (<--- if you have nothing else, at least provide this)
>>> 2. What is your QPS?
>>> 3. What % of your requests are errors?
>>>
>>> --
>>> Ikai Lan 
>>> Developer Programs Engineer, Google App Engine
>>> plus.ikailan.com
>>>
>>>
>>>
>>> On Fri, Mar 9, 2012 at 7:24 AM, Ronoaldo José de Lana Pereira <
>>> rpere...@beneficiofacil.com.**br >wrote:
>>>
>>>> +1 for seeing the same problems on my app.
>>>>
>>>> It started to be worse after maintenance on March 7.
>>>>
>>>> Em sexta-feira, 9 de março de 2012 08h33min36s UTC-3, Nikolai escreveu:
>>>>
>>>>> +1
>>>>> we had to move to our backup systems. Everything is full of 500 errors 
>>>>> or hardcore latency.
>>>>> Most of the 500 errors we see aren't even logged so this seems to be a 
>>>>> goole problem one abstraction layer above the app.
>>>>>
>>>>> And yes - sometimes we have got the same feeling, that we are the only 
>>>>> ones that use appengine in a production setting. You are not alone ;)
>>>>>
>>>>> regards,
>>>>> nikolai
>>>>>
>>>>> Am Dienstag, 6. März 2012 22:17:37 UTC+1 schrieb Adam Sherman:
>>>>>>
>>>>>> Am I the only one seeing short duration outages? They are being 
>>>>>> reflected at:
>>>>>>
>>>>>> http://code.google.com/status/appengine<http://code.google.com/status/appengine>
>>>>>>  
>>>>>> But I don't see anyone else complaining anywhere, so it makes me 
>>>>>> worried.
>>>>>>
>>>>>> A.
>>>>>>
>>>>>>  -- 
>>>> You received this message because you are subscribed to the Google 
>>>> Groups "Google App Engine" group.
>>>> To view this discussion on the web visit https://groups.google.com/d/**
>>>> msg/google-appengine/-/**yixu1yAlMs4J<https://groups.google.com/d/msg/google-appengine/-/yixu1yAlMs4J>
>>>> .
>>>>
>>>> To post to this group, send email to google-appengine@googlegroups.**
>>>> com .
>>>> To unsubscribe from this group, send email to 
>>>

[google-appengine] Re: Outages?

[Alexander Trakhimenok]

Our Python 2.5 app "petaclasses" is still having crazy amount of instances 
- over 250 at the moment against normal 10-15.

Our average costs to run the app is $25 per day - it's 4 hours till the end 
of the today and we already spend $154. What the  ??

Is it policy of Google AppEngine team that we (customers) should pay for 
mistakes made by GAE? Any chance to get it fixed and get a refund?

Thanks,
Alex

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/sLBthY7CRrsJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Outages?

[Alexander Trakhimenok]

Alex,

I believe it's better wait till Monday before starting a new thread as 
otherwise it would not get much attention. By Monday we would get actual 
billing history data for the 9th of March so we would be able to raise  a 
case with some data so we can prove a pattern.

In meantime I belive we can fill a spreadsheet - I've created a template: 
https://docs.google.com/spreadsheet/ccc?key=0AtxJ_-1aIO7idEhYT3pyLVZ4R3F0cGdTMFhyZkswVWc

Also we can raise and issue in the issue tracker.

Alex

On Saturday, 10 March 2012 06:20:40 UTC-4, Alex Popescu wrote:
>
> I'm also interested to find out how to get a refund after the reports I've 
> submitted in the last 24h showing in this thread [1] that since the 
> maintenance window Google AppEngine itself has been the only one 
> responsible for consuming my quota. While historical stats have also been 
> wiped out since the last maintenance I still have the billing history which 
> can be used as historical data.
>
> Alexander, should we start a separate thread to get this info from Google?
>
> [1] 
> https://groups.google.com/forum/?fromgroups#!topic/google-appengine/hpCCA3-QR04
>
> On Saturday, March 10, 2012 6:09:02 AM UTC+2, Alexander Trakhimenok wrote:
>>
>> Our Python 2.5 app "petaclasses" is still having crazy amount of 
>> instances - over 250 at the moment against normal 10-15.
>>
>> Our average costs to run the app is $25 per day - it's 4 hours till the 
>> end of the today and we already spend $154. What the  ??
>>
>> Is it policy of Google AppEngine team that we (customers) should pay for 
>> mistakes made by GAE? Any chance to get it fixed and get a refund?
>>
>> Thanks,
>> Alex
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/m8blzdjhG8kJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: NDB Caching Question

[Alexander Trakhimenok]

Richard, I would advise to go with the JSON property. In our project
we intensively use JSONs and update them in task queues & backends.
Actually we have a rule - every page should make just 3-5 DB requests.
In future we would consider to move from JSON to ProtoBuf but not for
now.

Also we've moved some rarely changed dictionaries (like geo locations
- e.g. all cities in the world) into the Python code. That pushed us
to use F2 instances due to higher memory demand but resulted in lower
latency and almost same costs. It's cheaper to upload new version of
app when needed.
--
Alexander Trakhimenok
Dev lead at http://www.myclasses.org/ project

On Apr 24, 6:07 pm, Richard Arrano  wrote:
> Thank you for the quick and very informative reply. I wasn't even
> aware this was possible with NDB. How would those x.yref.get() calls
> show up in AppStats? Or would they at all if it's just pulling it from
> memory?
>
> Thank you Kaan as well, I will actually experiment with the
> PickleProperty and see what's faster. I like that solution because the
> X kind is not one I expect to be heavily cached so I don't mind
> actually caching the pickled instance as I expect them to be evicted
> within a relatively short amount of time.
>
> I also wanted to ask: I saw someone did a speed test with NDB and I
> noticed he was pulling 500 entities of 40K and in the worst-case 0%
> cache hit scenario, it took something like 8-10 seconds. I was
> actually planning to have a piece of my application regularly query
> and cache ~2500 entities(of 2500) and sort on it to avoid a huge
> amount of indices(and a NOT IN filter that would really slow things
> down). Is this feasible or would you expect his results to scale, i.e.
> 500 entities with 0% cache hits * 5 ~= 40-50s in my usage scenario? Or
> was there something unique to his situation with his indices and large
> amount of data? In mine each entity has about 10 properties with zero
> indices. If this is the case I'll probably copy the entities into a
> JsonProperty that occasionally gets updated and simply query/cache
> that since I don't expect the 2500 entities to change very often.
>
> Thanks,
> Richard
>
> On Apr 24, 12:59 pm, Guido van Rossum  wrote:
>
>
>
>
>
>
>
> > On Monday, April 23, 2012 10:21:26 PM UTC-7, Richard Arrano wrote:
>
> > > I'm switching from db to ndb and I have a question regarding caching:
>
> > > In the old db, I would have a class X that contains a reference to a
> > > class Y. The Y type would be accessed most frequently and rarely
> > > change. So when I would query an X and retrieve the Y type it points
> > > to, I would store X in the memcache with the actual instance Y rather
> > > than the key. If X is invalidated in the memcache, then so is the Y
> > > instance but otherwise I would skip the step of querying Y upon re-
> > > retrieving X from the memcache. Is there any way to do this in ndb? Or
> > > must I re-query each Y type even if it is from memcache or context?
>
> > If you leave the caching to NDB, you probably needn't worry about this
> > much. It's going to be an extra API call to retrieve Y (e.g. y =
> > x.yref.get()) but that will generally be a memcache roundtrip. If you are
> > retrieving a lot of Xes in one query, there's a neat NDB idiom to prefetch
> > all the corresponding Ys in one roundtrip:
>
> > xs = MyModel.query(...).fetch()
> > _ = ndb.get_multi([x.yref for x in xs])
>
> > This effectively throws away the ys, but populates them in the context
> > cache. After this, for any x in xs, the call x.yref.get() will use the
> > context cache, which is a Python dict in memory. (Its lifetime is one
> > incoming HTTP request.)
>
> > You can even postpone waiting for the ys, using an async call:
>
> > xs = MyModel.query(...).fetch()
> > _ = ndb.get_multi_async([x.yref for x in xs])
>
> > Now the first time you reference some x.yref.get() it will block for the
> > get_multi_async() call to complete, and after that all subsequent
> > x.yref.get() calls will be satisfied from memory (no server roundtrip at
> > all).

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] HRD migration tool hangs for hours - how to find a root cause?

[Alexander Trakhimenok]

We are migrating our app to an HRD version and the migration tool process hangs 
on 2nd step "Copy" at 99% for 10+ hours.

We've restarted migration but it did not help - still hangs at 99% and the 
"approximately remaining" time is steadily and slowly growing.

I've been googling around and as I understand it can be because of indexes 
issue (5K index entry limit per entity) or some other reason that prevent an 
entity to be stored to the new HRD database. Actually removing some indexes 
helped us to move from 98.50% to 99% and  not  any further.

But we can not find any error/status logs that would point us on what exactly 
entity kind/id causing the issue.

On the Migration Tool screen it says:
=
If there is a problem, you can pause the migration. While paused, all migration 
mapreduces will be paused and but no other actions will be taken until an 
administrator either resumes the migration or reverts the process. To pause the 
migration, please click the button below.
=

But how do we know if there is a problem and what exactly the problem is? It 
would be cool to have a "last or list if error messages & entity keys for the 
migration process.

Any ideas how to identify and solve the issue? We have major release pending 
due to limitations of M/S database and would greatly appreciate any help.


I've raised a production issue for this as well 
http://code.google.com/p/googleappengine/issues/detail?id=7787


but may be someone from community can help us to resolve it before Google 
Engineers notice the ticket.


Alexander Trakhimenok


Lead developer on http://www.myclasses.org/ project powered by GAE (AppEngine + 
Python + Django).

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/AYfYbuyKGsYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] HRD Migration failed (got stuck at reverting). And few thoughts on customer support

[Alexander Trakhimenok]

Hello Google AppEngine team and GAE developers,

Just a heads up on status of migration tool (*for devs to be aware and 
prepared*) and am begging for support from Google.

We have a major release that migrates our app from M/S to HRD (*with all 
the goodies like XA transaction & Full Text Search*) that was supposed to 
be executed a week ago.

First we hit 2 bugs in HRD migration tool:

   1. *Migration failed due to old unindexed string properties with values 
   > 500 chars* (*it was possible to save such in old SDK*). Fixed by 
   re-putting entities ourselves by executing map/reduce.
   2. *Migration failed due to blob reference key to non existed blobs* (*there 
   are no reference constrains so no wonder we had a few in DB*). Fixed 
   ourselves by removing bad references using map/reduce.

For those 2 issues we have a production issue 
ticket: http://code.google.com/p/googleappengine/issues/detail?id=7787

Once this was fixed by us after getting information on root cause from 
Google (*there is no way to find errors details by yourself*) we decided to 
revert migration and execute it from scratch. And I guess it was our 
mistake.

The migration process got stuck in "Sync - Reverting..." state for 4 days 
by now:

*Sync*Reverting...

I've raised a separate issue for this 3 days 
ago: http://code.google.com/p/googleappengine/issues/detail?id=7822

Still no answer after 24 hours.

*To developers*:

   1. I advise you to check your data before migration for those 2 issues 
   described above.
   2. Try avoiding reverting if possible

*To Google GAE team*:

   1. It would be VERY cool for us (*GAE customer*) to be able to see error 
   logs for the migration process.
   2. It would be VERY-VERY cool to get response on production issues at 
   least in 1 working day (*24 hours*). We are paying customers and while 
   nto BIG ones but w do spend around $1K per month on GAE and do expect some 
   support. Why don't hire at least 1 person who would manage tickets and be 
   responsible for keeping your customers updated?
   3. Please help us to migrate to HRD - we are eager to move!
   4. And thanks for the wonderful platform!
   
Happy coding to everyone.

Alex
http://www.myclasses.org/ - Powered by GAE

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/wK_m8nnpXwcJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: GAE app with images stored on S3

[Alexander Trakhimenok]

We've implemented storing images to S3 from GAE for 
http://www.myclasses.org/ and pretty happy with results.

There is a 2 factors problem with serving images from GAE's blob-store:

   1. Generation of get_serving_url takes a while. If you have many images 
   to serve on a single page (*e.g. list if classes and their avatars in 
   our case*) it can really slow your code and as result cost you money. So 
   you don't want to do it on every page rendering.
   2. We tried to cache the URL returned by get_serving_url() but it 
   occurred it expires in a while (may be days or weeks) and there is no clear 
   understanding when exactly and how to invalidate the URLs or HTML of pages 
   we generate & cached in DB.

With S3 we can have static & stable URL what is perfect for us. And 
in conjunction with CloudFrount it works really fast - we see noticeable 
improvements in client pages rendering by browsers.

We used "boto" library to work with S3 that is not coupled with Django in 
any way (*as far as I know*) so you can try it as well.

Good luck,

Alex
Founder & Lead Dev @ Petamatic.com

On Tuesday, 4 December 2012 16:32:31 UTC-4, Emmanuel Mayssat wrote:
>
> They are several discussion about storing images in database (blob) versus 
> filesystem (s3).
> Did anyone implement an app on GAE with an image interface on S3?
>
> I am not even sure where to start
>
> Django has a 
> http://django-storages.readthedocs.org/en/latest/index.html
> but I use webapp2
>
> Thanks for any info,
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/GCtIWJ99pIcJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: How many entities can be placed into datastore?

[Alexander Trakhimenok]

You can have as many entities as you want and speed does not depend on 
number of entities.

Speed of .get() or .put() would be the same if you have 3 entities or 
3.000.000.000 entities.

Though the can be performance issues if you have high throughput - in this 
case minimize number of indexed fields and it's better if indexed values 
are not "serially increased".

Alex
Lead Developer at http://www.myclasses.org/ - powered by GAE

On Thursday, 21 February 2013 13:56:30 UTC-4, Deneb wrote:
>
> Good day, i have an strange questation - how much of entities i can create 
> for one namespace? We develop an app and for best performance we need about 
> 250 millions of entities in one namespace. Is this real and how it will 
> affects on App Engine speed? We suppose to use this entities only as 
> "get-by-key" queries.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: MongoDB Datastore Stub for Google App Engine SDK (Python)

[Alexander Trakhimenok]

Sounds promising. Would be cool if this can be included to GAE SDK.

Alex
http://sharp-developer.net/

On Sunday, 31 March 2013 12:14:33 UTC-4, Stanislav Heller wrote:
>
> Hello GAE community,
> Datastore MongoDB Stub is now ported to SDK 1.7.6, especially 
> devappserver2 is supported now.
>
> Please, report any bugs and errors into the github issue tracker 
> https://github.com/hellerstanislav/appengine-datastore-mongodb-stub/issues
>
> Cheers,
> Stanislav Heller
>
> Dne pondělí, 4. března 2013 19:08:01 UTC+1 Stanislav Heller napsal(a):
>>
>> Now the Mongodb Datastore Stub is tested and ported to SDK 1.7.5 too. The 
>> old SDK 1.7.4 has it's own branch on git for further fixes.
>>
>> Warning: devappserver2 not supported.
>>
>> Stanislav Heller
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: appengine 1.8.0 already deploying?

[Alexander Trakhimenok]

Our app also shows 1.8.0.

On Friday, 3 May 2013 11:32:59 UTC-4, Alex Burgel wrote:
>
> I was just checking the console and noticed that my app was running 1.8.0. 
> (see attached screenshot)
>
> Given that 1.7.8 is only in pre-release, is this supposed to happen? A bug 
> perhaps?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: Amazon S3 Server on GAE (Java)

[Alexander Trakhimenok]

What the point of the project? There is S3, Blobstore API, Google Drive?

Why would someone to want emulate S3 on top of GAE? If you need 
cross-platform app may be you should thing about creating an abstraction 
layer?

On Tuesday, 14 May 2013 04:51:52 UTC-4, xybrek wrote:
>
> Hello, I am currently working on a project that aims to provide 
> S3-compatible object storage API on top of th Google App Engine Platform.
> The project is a fork of littles3  from 
> Jesse Peterson; which is the most complete open source S3 server I found 
> for Java. 
>
> Also, the aim of the project is to provide a Java open implementation of 
> S3 
> (at least most of it, and at least to a usable state) to be able to host 
> your own S3 in GAE cloud, JBoss CapeDwarf or with AppScale. 
>
> And instead of starting from scratch I started with littles3 which works 
> fine already with Tomcat. 
> However, it does use a File API and other libraries that are not available 
> in the GAE platorm. 
>
> I named the project 'basket' and is hosted in Gitub: http://bit.ly/YSE3oY
>
> *Basket *has 3 modules:
>
> *api*
>
> Contains the models and the DAO interface
>
> *filesystem*
>
> Provides concrete implementation of DAO, in the case of littleS3 it was a 
> File based DAO, in case of Basket, I chose to use 
> Mungo  a JSON document datastore api; since it's 
> not possible to save Files with GAE, and GAE Vfs projects seems not active 
> anymore. 
>
> *web*
>
> Is the module that binds it all. It is the actual GAE app which will 
> provide the Restful api to S3 clients.
>
> So, if anyone is interested in joining let me know :-)
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [google-appengine] Re: Accumulated lots of data - deleting is is too expensive

[Alexander Trakhimenok]

Jeff, I would be strongly against doubling write prices and making deletes 
free.

In our app we have hundreds of millions records that we updated quite 
frequently but almost never delete. That would unnecessary double our costs 
just because someone did not think in advance about deletes.

I'm a huge fan of "pay as you go" and granular billig models so I believe 
GAE has right pricing structure. We all do mistakes, but we should pay just 
for our own mistakes, not someone's else. Isn't it?

Alex
Founder at www.myclasses.org, powered by GAE

On Friday, 24 May 2013 20:32:51 UTC-4, Jeff Schnitzer wrote:
>
> The economist in me thinks that Google should just double the price of 
> writes and make delete free.
>
> Jeff
>
>
> On Fri, May 24, 2013 at 12:57 PM, Jason Collins 
> 
> > wrote:
>
>> I too suspect that deletion is a truly expensive operation and that is 
>> directly reflected in pricing. Or worse, that the tablets remain forever 
>> fragmented and the space is never actually reused (as previously suggested 
>> on this thread).
>>
>> I've often advocated for a way for me to mark an entity as "for deletion" 
>> and allow Google to come around in some kind of batch operation to clean it 
>> up. It would be ideal if it were immediately removed from indexes (i.e., 
>> from sight) and I would be willing to pay for it until the background 
>> cleanup comes around (e.g., maybe at least every X days) - as long as the 
>> wait+background-cleanup costs were some fraction of just outright deleting 
>> it.
>>
>> Even without the immediate index removal, we have lots of use cases where 
>> the data could actually remain indexed because our particular use case 
>> naturally avoids these orphaned rows (e.g., think of all the blog posts and 
>> comments and +1's for a deleted account). I'm sure this is pretty common. 
>> So to be able to mark all these entities as "for deletion" or "reclaimable" 
>> would let a background process clean them up for little or no cost (apart 
>> from datastore storage while holding them during the "wait" period).
>>
>> j
>>
>> On Friday, 24 May 2013 13:09:43 UTC-6, vlad wrote:
>>>
>>> While it might be true that data deletion is expensive. The reality is 
>>> Google is losing customers over that! It is obvious that whoever started 
>>> this thread is not going to fork over $3600 for the privilege. His only way 
>>> out right now is to cancel his credit card and abandon the account. Since I 
>>> doubt GAE will let him just stop billing on that app. This is a sad 
>>> situation and a flaw in GAE's business model.
>>>
>>> On Friday, May 24, 2013 11:53:59 AM UTC-7, barryhunter wrote:




 On Fri, May 24, 2013 at 7:33 PM, John Wheeler wrote:

>
> It would be best if deletes were extremely cheap - give developers the 
> ability to do them at cost so they're not afraid to experiment on App 
> Engine
>

  What makes you think they are not close to 'cost' already?

 You seem to be assuming deletes are absurdly 'marked up' - for what 
 ever reason. 

 Why would Google doe that? 


 Deleting at 'scale' is not cheap. Your data is replicated around. All 
 those copies have to be found and 'deleted'. The indexes are seperate and 
 have to be deleted too. There may be many. 

 In fact most of the time the data isn't actually deleted. 
 Just Tombstoned. Marked as deleted, so the space is not actually reclaimed 
 right away (to be sold again). Would be to much work to remove the 'holes' 
 all the time. 

 The space will probably be reclaimed eventually, when the tablets are 
 compacted. But not right away
  

 In fact when a Application is deleted, wouldnt be surprised if Google 
 don't jsut absorb the storage cost, and not actully bother deleting the 
 data. Deletions will be relativly rare, and few will leave large amounts 
 of 
 data lying around. Will just be orphaned and ignored. 


  


  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com .
>> To post to this group, send email to 
>> google-a...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/google-appengine?hl=en
>> .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>  
>>  
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[google-appengine] Re: 1.9.0 Pre-Release SDKs are now available.

[Alexander Trakhimenok]

What about Go language SDK? Is it still alive?

Alex

On Wednesday, 5 February 2014 05:43:41 UTC+4, Richmond Manzana wrote:
>
> We want to inform you that the pre-release SDKs for Python, PHP and Java 
> are now available.
>
> As previously 
> announcedin
>  a Google code site announcement, new App Engine Binaries are no longer 
> available at:
> http://code.google.com/p/googleappengine/downloads/list
>
> Older binaries will remain available at the code.google.com site.
>
> 1.9.0 Pre-release SDKs are now available at these links:
>
> App Engine 1.9.0 Java prerelease 
> SDK
>
> App Engine 1.9.0 Python prerelease 
> SDK
>
> App Engine 1.9.0 PHP prerelease 
> SDK
>
> In the future, please look forward to the finding the latest binaries at 
> https://developers.google.com/appengine/downloads
>
>
> Also, please see the pre-release notes below.
>
> Cheers,
>
> Richmond Manzana
> Technical Program Manager
> Google App Engine
>
> App Engine SDK - Pre-Release Notes
>
> Version 1.9.0
>
> Python & PHP
> ==
> - New App Engine Application Identifiers must now start with a letter,
>   in addition to the existing requirements that the identifier be 6-30
>   characters which are letters, numbers, and hyphens, and not start or end 
> with
>   a hyphen.
>
> Python
> ==
> - The size limit on the Search API is now computed and enforced on a 
> per-index
>   basis, rather than for the app as a whole. The per-index limit is now 
> 10GB.
>   There is no fixed limit on the number of indexes, or on the total amount 
> of
>   Search API storage an application may use.
> - Users now have the ability to embed images in emails via the Content-Id
>   attachment header.
> https://code.google.com/p/googleappengine/issues/detail?id=965
> https://code.google.com/p/googleappengine/issues/detail?id=10503
> - Fixed an issue with NDB backup/restore corrupting certain compressed
>   entities.
> https://code.google.com/p/googleappengine/issues/detail?id=8599
>
> PHP
> ==
> - The PHP interpreter has been upgraded from PHP 5.4.19 to PHP 5.4.22.
> - Autoloading is now available in the SDK so developers will no longer 
> need to
>   explicitly require SDK files.
> - Expanded php.ini setting google_appengine.allow_include_gs_buckets to 
> allow
>   a path filter be included for improved security.
> - A warning message now appears if an application moves a user uploaded 
> file to
>   a Google Cloud Storage bucket/path. This is due to the fact that code 
> may be
>   included and lead to a local file inclusion vulnerability.
> - Added API functions CloudStorageTools::getMetadata() and
>   CloudStorageTools::getContentType() for retrieving the metadata and 
> content
>   type of Google Cloud Storage objects.
> https://code.google.com/p/googleappengine/issues/detail?id=10182
> - Fixed an issue with GCS folders not displaying correctly in Developers
>   Console.
> - Fixed an issue with PHP_SELF and SCRIPT_NAME not being implemented 
> correctly.
> https://code.google.com/p/googleappengine/issues/detail?id=9989
> https://code.google.com/p/googleappengine/issues/detail?id=10478
>
> Java
> ==
> - Java 6 applications cannot be deployed to Google App Engine from any 
> version
>   of the SDK. Existing Java 6 applications will continue to run. If you are
>   still relying on a Java 6 application in Google App Engine, we strongly
>   encourage you to start testing and deploying your app using Java 7. Java 
> 7
>   has been certified by Oracle to be fully backwards compatible with Java 
> 6,
>   as well as providing a number of benefits to developers. Java 7 
> applications
>   will be fully supported in future releases of the App Engine SDK.
>   If you absolutely need to continue to deploy Java 6 applications for
>   compatibility reasons, you can request that your application be 
> whitelisted
>   for Java 6 deployment from http://goo.gl/ycffXq. As we review each 
> whitelist
>   application by hand, please allow 10 working days for review, and note 
> that
>   only billed applications that show a proven incompatibility will be
>   considered for exemption.
> - Modules API methods have been moved out of the labs package in 
> preparation
>   for a general availability release.
> - MapReduce is now a Preview feature. Documentation and getting started 
> guides
>   are now available on developers.google.com.
> - The size limit on the Search API is now computed and enforced on a 
> per-index
>   basis, rather than

[google-appengine] Datastore space is wrongly allocated to "Other" kind?

[Alexander Trakhimenok]

Hi,

In production AppEngine datastore statistics show that 65% of datastore 
disk space is allocated to "Other" kind. (out of 3GB total). Datastore 
type: HRD

https://www.dropbox.com/s/cfxc6tgzn93dfmf/Screenshot%202014-12-01%2009.10.40.png?dl=0

We do not have the "Other" kind and all other kinds are definitely should 
not take up to 65% of space. All other kinds that is not included in the 
chart should take less 1% of total.

_AE_* entities are also pretty small.

This issue probably affects billing and in result we may be overcharged.

I've created an issue ticket for 
this: https://code.google.com/p/googleappengine/issues/detail?id=11507

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Urgent SOS: Need to re-enable GAE service after reopening billing account

[Alexander Trakhimenok]

I know it's little chances to get support here but I am bit desperate as 
our account has been disabled and we are losing money due to disruption to 
service.

When our credit card expired and we got new one I've updated CC details on 
billing page for Google AppEngine but did not at Google Cloud Billing. 
Unfortunately email associated with Google Cloud has not been monitored :(.

As a result our Google Cloud Billing account has been disabled and GAE 
service stopped to work (*all our websites are down*).

I've updated credit card details and successfully reopened billing account 
on Google Cloud but the GAE service still is "over quota".

The "Billing status" page says that "The current payment method is not 
valid. Update it under Billing > Billing Settings." But it *IS* valid both 
on Google AppEngine and Google Cloud.
I even manually paid $10 on Google Cloud to check the credit card details 
are fine.

See screenshots:
GAE => Billing settings: 
https://www.dropbox.com/s/yfk0ujh5bkgn90i/Screenshot%202015-01-21%2011.01.53.png?dl=0
GAE => Billing status: 
https://www.dropbox.com/s/gzyjj6aiwtwts1d/Screenshot%202015-01-21%2010.59.39.png?dl=0

Please note the "Billing status" is shown as "free" and the "Enable 
billing" is disabled. I added personal credit card as a "backup" option but 
that did not help.

GAE app: *myclassesapp*
Google Cloud Billing account for myclassesapp: 00E24E-43AF44-4A9EDD
Google Cloud Billing account ID: 0624-9112-4607-9990

I've send a request to billing support but it seems to me like a bug on GAE 
admin so probably someone from tech support can assist quicker.

We would really appreciated any support on resolving this issue as soon as 
possible.

Alex

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/67ee3a5b-eb3d-49ca-95c7-911483d33352%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: App suddenly lost billing settings

[Alexander Trakhimenok]

We have the same issue, GAE app: myclassesapp - all our websites are down :(

On Wednesday, 21 January 2015 08:02:16 UTC+3, PromptAppointment wrote:
>
> I need someone to help me with my app promptappointment, all of a sudden 
> it reverted back to the free pricing tier when I have billing enabled and 
> nothing is wrong with my credit card. This is a production app and I need 
> help asap. Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/5a8b7f6c-49dc-4973-84c7-b19bec226ebb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Urgent SOS: Need to re-enable GAE service after reopening billing account

[Alexander Trakhimenok]

Just going to dashboard in new console did not fix the issue for us.

BUT it seems if you update daily budget in the new console from "0" to 
whatever is appropriate helps. After daily budget changed our websites 
become accessible - not immediately but in couple of minutes. May be it 
would help others.

So far we've lost at least $100 (AdSense earnings) due to downtime 
(https://www.dropbox.com/s/9m21e2wctwa34tz/Screenshot%202015-01-22%2011.11.45.png?dl=0)
 
- any chances to recover it throw SLA refund? This are direct loss. We also 
lost about 80 registration and would be penalized by search engines 
(including Google) for being inaccessible but this hard to calculate and 
prove. Getting a compensation  for direct loss would be really nice.

*Luke*, the 
link https://code.google.com/p/googleappengine/issues/detail?id=11615 is 
not opening for me - error 403. Seems it has been 
deleted: 
https://code.google.com/p/googleappengine/issues/list?can=1&q=&sort=-id&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log

Alex

On Thursday, 22 January 2015 08:59:52 UTC+3, Luke Stone (Google Cloud 
Support) wrote:
>
>
> We see the issue in billing status for certain AppEngine accounts, we have 
> a root cause and we're working on automatic fixes.  We apologize if your 
> app is down because of this issue.  In the meantime please try the 
> workaround of going to your AppEngine Settings page in Google Developer 
> Console.  
>
>
> If you go to http://appengine.google.com/ and find your app, you should 
> see a link "Try the new Dashboard".  Click that and then, under Compute, 
> click App Engine.  That ought to fix it, and please respond if it doesn't 
> work. 
>
>
> I see that you have already sent a request to billing support - thanks for 
> that - and for anyone else who needs to contact us about this issue, here's 
> the link:
>
> https://support.google.com/cloud/contact/cloud_platform_billing
>
>
> This is on the issue tracker here, you can follow along and we will post 
> the root cause here
>
> https://code.google.com/p/googleappengine/issues/detail?id=11615
>
> -Luke 
>
>
>
> On Thursday, January 22, 2015 at 7:03:28 AM UTC+11, Ivan Lucas Vargas 
> wrote:
>>
>> Hi,
>>
>>   Exactly the same problem here. Some update? I'm really stuck at 
>> this point...
>>
>>
>>
>> On Wednesday, January 21, 2015 at 6:29:14 AM UTC-2, Alexander Trakhimenok 
>> wrote:
>>>
>>> I know it's little chances to get support here but I am bit desperate as 
>>> our account has been disabled and we are losing money due to disruption to 
>>> service.
>>>
>>> When our credit card expired and we got new one I've updated CC details 
>>> on billing page for Google AppEngine but did not at Google Cloud Billing. 
>>> Unfortunately email associated with Google Cloud has not been monitored :(.
>>>
>>> As a result our Google Cloud Billing account has been disabled and GAE 
>>> service stopped to work (*all our websites are down*).
>>>
>>> I've updated credit card details and successfully reopened billing 
>>> account on Google Cloud but the GAE service still is "over quota".
>>>
>>> The "Billing status" page says that "The current payment method is not 
>>> valid. Update it under Billing > Billing Settings." But it *IS* valid 
>>> both on Google AppEngine and Google Cloud.
>>> I even manually paid $10 on Google Cloud to check the credit card 
>>> details are fine.
>>>
>>> See screenshots:
>>> GAE => Billing settings: 
>>> https://www.dropbox.com/s/yfk0ujh5bkgn90i/Screenshot%202015-01-21%2011.01.53.png?dl=0
>>> GAE => Billing status: 
>>> https://www.dropbox.com/s/gzyjj6aiwtwts1d/Screenshot%202015-01-21%2010.59.39.png?dl=0
>>>
>>> Please note the "Billing status" is shown as "free" and the "Enable 
>>> billing" is disabled. I added personal credit card as a "backup" option but 
>>> that did not help.
>>>
>>> GAE app: *myclassesapp*
>>> Google Cloud Billing account for myclassesapp: 00E24E-43AF44-4A9EDD
>>> Google Cloud Billing account ID: 0624-9112-4607-9990
>>>
>>> I've send a request to billing support but it seems to me like a bug on 
>>> GAE admin so probably someone from tech support can assist quicker.
>>>
>>> We would really appreciated any support on resolving this issue as soon 
>>> as possible.
>>>
>>> Alex
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9a210e45-5141-4f21-96fa-283d658fb6bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: HTTP/2 Protocol Support

[Alexander Trakhimenok]

For some reason it does not show HTTP/2 or SPDY support on my app that use 
plain HTTP (no HTTPS).

Is it intended behavior? How I can make our websites (*hosted on custom 
domains)* to use HTTP/2?

Alex

On Thursday, 19 March 2015 20:50:00 UTC, Greg Jones wrote:
>
> It does. 
> You can install this chrome-extension (
> https://chrome.google.com/webstore/detail/spdy-indicator/mpbpobfflnpcgagjijhmgnchggcjblin
>  
> ) and it will show if you're connected to a site with anything other than 
> plain-old-http.
> If you check it when on, say, https://apprtc.appspot.com/ it should add a 
> blue icon in the URL-bar and show "HTTP/2 enabled (h2-14)". green means 
> SPDY (appspot.com showed this until recently. you can see it now on 
> twitter.com). When it's red that means "quic" is involved.
>
> On Thursday, 19 March 2015 13:07:27 UTC, Dan Dubois wrote:
>>
>> Does App Engine support the HTTP/2 protocol?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/48eac3d9-1451-4b87-82c9-d6affdd9bd04%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: HTTP/2 Protocol Support

[Alexander Trakhimenok]

Not an option for us - we have too many domains linked to the app and costs for 
acquiring SSL certificates would be prohibitive.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/d4e87f0a-bf3d-44d9-8eab-3d010a7bd85e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: How to alleviate 200gb/day - 20$+ daily outgoing bandwidth image serving costs

[Alexander Trakhimenok]

Use AWS. 30% is a quite a difference. Also you would have more control over 
headers and images can be closer to end customers - would not help with 
costs but it's nice.
Increasing cache time if you can. We set caching for over a month. 
Theoretically they can be cached by proxies as well (*but probably not much 
gain here unless majority of you customer is from the same corporate 
network*).
Make sure the consumer of the image caches it if possible. Test in all 
major browsers.



On Saturday, 2 May 2015 17:59:42 UTC+1, Kaan Soral wrote:
>
> Hi everyone
>
> I recently noticed a bump in costs after improving the static image 
> quality of my app, this bumped the daily bandwidth costs from ~7$'s to 
> 20$+'s
>
> Overall there are ~200 statically served images, I tried reducing the 
> quality of png's and turning them into jpg's, yet it degrades the image 
> quality, at the same level of quality, the size of the jpg's are identical 
> to the png's, so, as it is, paying the extra costs seem to be one option
>
> I'm also wondering whether there is an easier solution out there, maybe 
> using pagespeed as a CDN/cache with an easy config change, but as far as 
> I've researched, it didn't seem feasible/simple
>
> Setting up cloudflare for a subset of the traffic is another option, but 
> I'm not sure whether cloudflare can reliably serve all those images
>
> I checked some other CDN's / cloud solutions, they all seemed close to 
> AppEngine's pricing, CloudFront for example: 
> http://aws.amazon.com/cloudfront/pricing/ ~30% lower
>
> As it is, App Engine serves images well, yet the bandwidth costs are 
> higher than the instance hour costs, so I would like to reduce the 
> bandwidth cost if there is a simple solution out there that I'm missing
>
> (My default_expiration is "1h", I've also verified the "expires" header, 
> yet probably the consumer of the images doesn't cache them, I really wish 
> the issue was caching, but it doesn't seem the issue can be solved by 
> modifying the http caching)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/33f5882b-1adb-4452-b7d1-3fca3c3fb23a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Google Cloud Debugger on Google App Engine

[Alexander Trakhimenok]

Any plans about something similar for Python & Go?

On Wednesday, 13 May 2015 21:55:51 UTC+1, Keith Smith wrote:
>
> Google Cloud Debugger (Beta) is now available on GAE 
> 
>  for 
> Java based applications.
>
> Say goodbye to littering your code with logging statements to locate and 
> diagnose issues in production. With the Debugger agent installed by default 
> on GAE capturing local variables and a call stack is as easy as clicking on 
> a line of source code.
>
> T 
> ry
>  
> it out today  and send 
> feedback to cdbg-f...@google.com .
>
> Keith Smith
> Product Manager - Google Cloud Platform
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/eb2a005e-6cbb-40e5-8519-be0d668d1557%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: CPU time for trivial datastore roundtrip

[Alexander Trakhimenok]

Do you use Django or other framework(s)?

1st expensive call indicates that probably CPU spent on modules
loading - it's well known and widely discussed on the forum.
--
Alex

On Jul 12, 1:39 am, Anthony Mills  wrote:
> Nick, thanks for responding. I'll add a lot more detail and test data
> here.
>
> I'm doing three fetches: one to get the user data, one to get his
> starred items, one to get his own items. I've broken it down to
> examine each one individually. The user fetch is fast:
>
> userEntity = datastore.get(KeyFactory.createKey("User", makeUserKeyName
> (getCurrentUserId(;
>
> /service?a=g&t=u 200 56ms 27cpu_ms 12api_cpu_ms 0kb
> /service?a=g&t=u 200 35ms 21cpu_ms 12api_cpu_ms 0kb
> /service?a=g&t=u 200 26ms 15cpu_ms 12api_cpu_ms 0kb
> /service?a=g&t=u 200 49ms 15cpu_ms 12api_cpu_ms 0kb
> /service?a=g&t=u 200 37ms 15cpu_ms 12api_cpu_ms 0kb
> /service?a=g&t=u 200 27ms 15cpu_ms 12api_cpu_ms 0kb
>
> The saved query is not so fast. Each item has a list property "s"
> which is a list of all user IDs that have saved that item. It's served
> by a dedicated index, s asc + i asc.
>
> Query query = new Query("Item");
> query.addFilter("s", Query.FilterOperator.EQUAL, getCurrentUserId());
> query.addSort("i");
> List items = datastore.prepare(query).asList(withLimit
> (REQUEST_SIZE));
>
> /service?a=g&t=ts 200 97ms 710cpu_ms 678api_cpu_ms 0kb
> /service?a=g&t=ts 200 79ms 711cpu_ms 678api_cpu_ms 0kb
> /service?a=g&t=ts 200 84ms 705cpu_ms 678api_cpu_ms 0kb
> /service?a=g&t=ts 200 87ms 716cpu_ms 678api_cpu_ms 0kb
> /service?a=g&t=ts 200 84ms 696cpu_ms 678api_cpu_ms 0kb
> /service?a=g&t=ts 200 72ms 697cpu_ms 678api_cpu_ms 0kb
>
> The query to get the user's items is similarly slow. It is a query on
> a string property "u", ordered by "i". It's served by a dedicated
> index, u asc + i asc.
>
> Query query = new Query("Item");
> query.addFilter("u", Query.FilterOperator.EQUAL, getCurrentUserId());
> query.addSort("i");
> List items = datastore.prepare(query).asList(withLimit
> (REQUEST_SIZE));
>
> /service?a=g&t=tm 200 71ms 699cpu_ms 678api_cpu_ms 0kb
> /service?a=g&t=tm 200 90ms 710cpu_ms 678api_cpu_ms 0kb
> /service?a=g&t=tm 200 73ms 699cpu_ms 678api_cpu_ms 0kb
> /service?a=g&t=tm 200 67ms 692cpu_ms 678api_cpu_ms 0kb
> /service?a=g&t=tm 200 83ms 698cpu_ms 678api_cpu_ms 0kb
> /service?a=g&t=tm 200 80ms 722cpu_ms 678api_cpu_ms 0kb
>
> Adding the sort does not seem to add any extra time, and nor should
> it, given the index type.
>
> Anyway, put the three together and this is what you get:
>
> /service?a=g 200 929ms 1731cpu_ms 1368api_cpu_ms 24kb
> /service?a=g 200 159ms 1423cpu_ms 1368api_cpu_ms 24kb
> /service?a=g 200 171ms 1420cpu_ms 1368api_cpu_ms 24kb
> /service?a=g 200 178ms 1414cpu_ms 1368api_cpu_ms 24kb
> /service?a=g 200 146ms 1405cpu_ms 1368api_cpu_ms 24kb
>
> See, what really annoys me is this. Those requests are coming back in
> 170 ms. But I'm charged for 1400 ms of CPU time. How does that work?
> It should be doing blocking work on the server, right? Are there
> really nine servers all churning for that whole time?
>
> Thanks for your time,
>
> Anthony Mills
>
> On Jul 10, 11:07 am, Alex Popescu
>
>  wrote:
> > Here's some sample code (not exactly the one I've previously mentioned
> > as I haven't got the time to prepare it):
>
> > [code]
> > query = ContentEvent.gql('ORDER BY created_at DESC')
> > event_list = query.fetch(max_results, offset)
> > # now that we've fetched the initial list we need to navigate all the
> > references
> > # to make sure that there will be no further DB hits
> > keys = []
> > for event in event_list:
> >   keys.append(event.get_refkey('entry'))
> > entries = db.get(keys)
> > for index, event in enumerate(event_list):
> >   event.entry = entries[index]
> > [/code]
>
> > This code gets called with a max_results=20 and various offsets
> > (pagination).
> > Now the results:
>
> > 07-10 09:51AM 01.720 /_m/cs?p=6 200 173ms 690cpu_ms 522api_cpu_ms 4kb
> > 07-10 09:50AM 57.408 /_m/cs?p=5 200 203ms 690cpu_ms 522api_cpu_ms 4kb
> > 07-10 09:50AM 52.963 /_m/cs?p=4 200 261ms 717cpu_ms 522api_cpu_ms 5kb
> > 07-10 09:50AM 48.888 /_m/cs?p=3 200 164ms 745cpu_ms 522api_cpu_ms 5kb
> > 07-10 09:50AM 42.068 /_m/cs?p=2 200 1563ms 2034cpu_ms 522api_cpu_ms
> > 6kb
> > 07-10 09:46AM 40.239 /_m/cs?p=1 200 168ms 745cpu_ms 522api_cpu_ms 5kb
> > 07-10 09:46AM 30.222 /_m/cs?p=2 200 1422ms 2089cpu_ms 522api_cpu_ms
> > 6kb
> > 07-10 09:45AM 20.582 /_m/cs 200 2718ms 2145cpu_ms 522api_cpu_ms 5kb
>
> > As you can see the initial requests are above 2000cpu_ms. Then it
> > looks like there is some pre-fetching kicking in that pushes this time
> > down to 690-750 cpu_ms.
>
> > To add a bit more detail to the above code: there are 2 entities one
> > of them keeping a ReferenceProperty to the other. Still, I am making
> > sure there are only 2 hits to the datastore by fetching all child
> > entities in a single get (basically there are 2 hits to the datastore
> > each returning 20 results).
>
> > The cpu_ms

[google-appengine] Re: CPU time for trivial datastore roundtrip

[Alexander Trakhimenok]

This is expected behavior and nature of GAE data store.

This is the beauty that the time will consistent and have direct
relations to number of items retrieved and not depend on number of
records stored.
--
Alex

On Jul 15, 4:14 am, Anthony Mills  wrote:
> On Jul 14, 6:48 am, Anthony Mills  wrote:
>
> > I'll do some testing later tonight about what happens when
> > REQUEST_SIZE is 1, 5, 10, 20, etc., and report back.
>
> Ok, here are a selection of the results of my tests. "rs" controls the
> number of results asked for.
>
> /service?a=g&t=m&rs=1 200 32ms 39cpu_ms 31api_cpu_ms
> /service?a=g&t=m&rs=2 200 44ms 50cpu_ms 44api_cpu_ms
> /service?a=g&t=m&rs=3 200 56ms 64cpu_ms 57api_cpu_ms
> /service?a=g&t=m&rs=4 200 101ms 82cpu_ms 70api_cpu_ms
> /service?a=g&t=m&rs=5 200 53ms 90cpu_ms 84api_cpu_ms
> /service?a=g&t=m&rs=6 200 50ms 109cpu_ms 97api_cpu_ms
> /service?a=g&t=m&rs=7 200 41ms 117cpu_ms 110api_cpu_ms
> /service?a=g&t=m&rs=8 200 57ms 136cpu_ms 123api_cpu_ms
> /service?a=g&t=m&rs=9 200 43ms 144cpu_ms 136api_cpu_ms
> /service?a=g&t=m&rs=10 200 62ms 174cpu_ms 150api_cpu_ms
> /service?a=g&t=m&rs=20 200 47ms 306cpu_ms 282api_cpu_ms
> /service?a=g&t=m&rs=50 200 76ms 705cpu_ms 678api_cpu_ms
> /service?a=g&t=m&rs=70 200 86ms 974cpu_ms 942api_cpu_ms
>
> Interesting. The variable api_cpu_ms climbs very ... predictably. In
> fact, it would seem:
>
> api_cpu_ms = 13.2 * results + 17.5
>
> ... rounded off to the nearest millisecond. At least for queries. I
> haven't tried straight key fetches.
>
> This is really disturbing.
>
> // Anthony
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---

[google-appengine] Re: Do highly indexed entities take up more storage space?

[Alexander Trakhimenok]

Every index keeps a full copy of each entry.

So to count storage space roughly you take entry size multiply by
(number of index +1) and add may be 10%.

it's well described in docs.
--
Alex

On Aug 2, 12:49 am, Kyle Jensen  wrote:
> Hi,
>
> I 'put' a few thousand highly indexed entities into the datastore
> yesterday and I felt like they used up a lot of my 'storage quota'
> even though the entities had few properties
>
> Do highly indexed entities take up more storage space?
>
> Thanks! Kyle
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---

[google-appengine] Re: Best Way to delete mass amount of data

[Alexander Trakhimenok]

Hi someone1,

You code looks good and prerrty optimized.

Delete takes data store CPU time as it needs to update all indexes you
have on your models.

I see just few areas for improvements.

1. Try to fetch just keys from the "Tracking" kind as you do not need
the model instance to delete it or query descendants (you will need to
use different query for descendants).
2. Probably you are fetching more trackers then needed as you can time
out by DedlineExceededError - not sure how the iteretor over filter
works as I usualy use fetch().
3. In you inner cycle may be it make sense to accumulate as much child
keys as possible and then call a single db.delete(). As I understand
you can collect a list of keys from different kinds so you can combine
the "Tracking" and the descendants for deletion.

But I do not think this will significantly decrease your API CPU time.
Have no idea how much exactly but my guess would be around 5%.

Alex,
http://sharp-developer.net/

On Sep 11, 8:00 pm, someone1  wrote:
> I have tried asking/researching this before, but I really need a more
> efficient way to delete mass amounts of data from the datastore. In
> short, I am only able to remove .1GB for 6.5 hrs of CPU time, and all
> of that is datastore time.
>
> Here is the code:
>
> class DeleteKeywords(webapp.RequestHandler):
>     def get(self):
>         try:
>             trackers = Tracking.all().filter('delete_track',True)
>             for x in trackers:
>                 keys = db.query_descendants(x).fetch(100)
>                 while keys:
>                     db.delete(keys)
>                     keys = db.query_descendants(x).fetch(100)
>                 x.delete()
>         except DeadlineExceededError:
>             queue = taskqueue.Queue(name='delete-tasks')
>             queue.add(taskqueue.Task(url='/tasks/delete_tracks',
> method='GET'))
>             self.response.out.write("Ran out of time, need to delete
> more!")
>
> Its really small and simple, and I did not think it'd use up soo much
> CPU time. Why is it that the API CPU time is soo much smaller than the
> Datastore CPU time? Is there any way to consume more of the datastore
> time than the API CPU time?
>
> I'd really like to clear out that database without needing to wait
> almost 50 hours worth of CPU time (which is odd since it runs for
> maybe 30 minutes - 1 hour, only 1 task a minute, and it uses up all
> that time... is it calculating wrong?)
>
> Anybody have any suggestions?
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---

[google-appengine] Re: API for programatically adding GAE application domain alias in 'Google Apps' control panel: Service Settings

[Alexander Trakhimenok]

Guy,

If you file a feature request I'll definetly will star it.
--
Alex

On Apr 20, 7:58 pm, Guy Davis  wrote:
> Hi all,
>
> An app I'm building is going to use subdomains to separate groups of
> users.  For example:
>
>     group1.myapp.com
>     group2.myapp.com
>
> These are configured in the 'Service Settings' control panel of the
> Google Apps Dashboard for myapp.com.  Currently, this is a manual
> process with the Google Apps admin required to login and add a new URL
> mapping such as: group3.myapp.com
>
> Is there a way or are there any plans to offer an API to configure
> these mappings.  Ideally, I'd like my GAE app to be able to add a new
> mapping as needed without requiring administrator intervention?  I'm
> aware that my DNS provider (currently Dreamhost) needs to have the
> alias added as well, but they are said to be working on an API.
>
> Thanks in advance,
> Guy
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---

[google-appengine] Re: Time zone in admin console

[Alexander Trakhimenok]

Would be nice if we can select a timezone to display from a drop-down.

Am also tired to do -8 all the time.
-
Alex

On Jun 26, 12:00 pm, dburns  wrote:
> Thanks Nick.  I went looking and discovered the issue tracking system,
> and I believe this is already logged:
>
> http://code.google.com/p/googleappengine/issues/detail?id=822
>
> So I guess I'll wait :)  Thanks for your reply.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---