[google-appengine] Re: [appengine-python] Re: 1.6.0 Prerelease SDKs are out

[Steve Sherrie]

I'm seeing incorrect parsing of multipart/form-data request bodies with 
2.7. Is this due to the same issue?

Steven Sherrie

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/OQAeHmIjDg0J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] high replication datastore limited to 1 write per second?!

[Steve Sherrie]

This just refers to the entity group write limit that is the same in 
both MS and HR datastores.


On 11-10-21 03:30 PM, Philip wrote:

I am concerned about the statement:

"This allows queries on a single guestbook to be strongly consistent,
but also limits changes to the guestbook to * 1 write per second *
(the supported limit for entity groups)."
http://code.google.com/appengine/docs/python/datastore/hr/overview.html.

Is it true that writes are limited to 1 per second when using the high
replication datastore or is this an old limitation?



--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] need immidiate help..!!!

[Steve Sherrie]

And can you describe where you put the script (in relation to the 
appengine-sdk) and how you are running the application?


Steve

On 11-10-16 04:35 PM, shobhit suman wrote:


sir

 I am using a virtual machine with ubuntu running ||ly on my windows 7
On Mon, Oct 17, 2011 at 2:03 AM, Steve Sherrie <mailto:st...@wasteofpaper.com>> wrote:


Hmmm, not sure. That line is definitely right. Which OS are you
using? Just so we have a better idea of your environment.

Steve
-- 
You received this message because you are subscribed to the Google

Groups "Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/Fkz-ZitOpgMJ.

To post to this group, send email to
google-appengine@googlegroups.com
<mailto:google-appengine@googlegroups.com>.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
<mailto:google-appengine%2bunsubscr...@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.




--
SHOBHIT SUMAN
B.Tech(II year)
Computer Science Engineering
The LNM Institute of Information Technology
Jaipur, India.
#+91-7742475353

--
You received this message because you are subscribed to the Google 
Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] need immidiate help..!!!

[Steve Sherrie]

Hmmm, not sure. That line is definitely right. Which OS are you using? Just 
so we have a better idea of your environment.

Steve

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/Fkz-ZitOpgMJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] need immidiate help..!!!

[Steve Sherrie]

You corrected it incorrectly :)  "from google.appengine.google.ext import 
db" should be "from google.appengine.ext import db".

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/Fdn-5-apNtkJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] need immidiate help..!!!

[Steve Sherrie]

Please include the error and the stacktrace, and we'll be able to help. 
It's a bit much to ask someone to fix your script and give it back to you :)


Steve


On 11-10-16 03:53 PM, shobhit suman wrote:

sir,

I rectified this mistake but still I am getting the same error 
message.It will be kind enough if u cud please please run the code on 
your system for it  might be possibl that my sdk has got sum problm.




On Mon, Oct 17, 2011 at 1:18 AM, Steve Sherrie <mailto:st...@wasteofpaper.com>> wrote:


"from google.appengine.ext import db" instead of "from
google_appengine.ext import db".

Steve



On 11-10-16 03:45 PM, shobhit suman wrote:

I am trying to run my first application in python on google sdk.
this is a simple datbase with a query statement . but I am
getting an unexpected error statement of module not found.please
help me asap..

-- 
SHOBHIT SUMAN

B.Tech(II year)
Computer Science Engineering
The LNM Institute of Information Technology
Jaipur, India.
#+91-7742475353

-- 
You received this message because you are subscribed to the

Google Groups "Google App Engine" group.
To post to this group, send email to
google-appengine@googlegroups.com
<mailto:google-appengine@googlegroups.com>.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
<mailto:google-appengine+unsubscr...@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
-- 
You received this message because you are subscribed to the Google

Groups "Google App Engine" group.
To post to this group, send email to
google-appengine@googlegroups.com
<mailto:google-appengine@googlegroups.com>.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
<mailto:google-appengine%2bunsubscr...@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.




--
SHOBHIT SUMAN
B.Tech(II year)
Computer Science Engineering
The LNM Institute of Information Technology
Jaipur, India.
#+91-7742475353

--
You received this message because you are subscribed to the Google 
Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] need immidiate help..!!!

[Steve Sherrie]

"from google.appengine.ext import db" instead of "from 
google_appengine.ext import db".


Steve


On 11-10-16 03:45 PM, shobhit suman wrote:
I am trying to run my first application in python on google sdk. this 
is a simple datbase with a query statement . but I am getting an 
unexpected error statement of module not found.please help me asap..


--
SHOBHIT SUMAN
B.Tech(II year)
Computer Science Engineering
The LNM Institute of Information Technology
Jaipur, India.
#+91-7742475353

--
You received this message because you are subscribed to the Google 
Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Python 2.7 threadsafe: true running on local

[Steve Sherrie]

No, it looks like the dev_appserver is not yet capable of using 
'threadsafe: yes'. However, with 'threadsafe: no', and your handler 
setup like 'script: main.py', you *can* dev locally, and then make those 
2 changes before deploying.


I know that doesn't help if you're testing multithreading features, but 
I think we're gonna have to wait 'til 1.5.6, or whenever the 2.7 runtime 
leaves experimental status, before we get that upgrade.



Steve

On 11-10-11 07:02 PM, ivan starkov wrote:

Does anybody could run the base sample for Python 2.7
http://code.google.com/appengine/docs/python/gettingstartedpython27/helloworld.html
on*local machine* with  parameter threadsafe: true in yaml
This sample generate error: Could not find module helloworld.app

the only way this sample works on my computer
is to change this line "script:helloworld.app"
to this "script:helloworld.py.app"
and add
def main():
app.run();

if __name__ == "__main__":
main()
to helloworld.py

And also what is official way to add webapp2 and jinja2 on local computer
i know two working solutions:
1) Add webapp2 jinja2 dirs to my app folder and exlude them in app.yaml

2) Add this libs to GAPP lib folder and change dev_appserver.py adding 
this lines

os.path.join(DIR_PATH,'lib','jinja2'),
os.path.join(DIR_PATH,'lib','webapp2') to EXTRA_PATHS list

Is any elegant solution to make webapp2 and jinja2 work locally?






--
You received this message because you are subscribed to the Google 
Groups "Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/qFolbjU1WSIJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: App Engine SDK 1.5.5 released!

[Steve Sherrie]

The 2.7 specific libraries do not seem to be included in the SDK as yet, 
but you can just download them yourself and put them in the root SDK 
directory, which is by default added to sys.path.


Steve

On 11-10-12 03:09 AM, Claudiu Apetrei wrote:

Hello,

Downloaded the 1.5.5.

When i do the python2.7 example I get  "'exceptions.ImportError'>: No module named webapp2 ", Shouldn't 
"import webapp2" work on 1.5.5 development server ("python2.7 
../google_appengine/dev_appserver.py ./").


If I upload it to appengine, the application works.

10x,
Claudiu
--
You received this message because you are subscribed to the Google 
Groups "Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/4_W1YWkL4KkJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: How many times did Google Apps Engine went down since it's launch

[Steve Sherrie]

Close enough!

On 11-10-08 02:15 PM, Philip wrote:

Joshua, your calculation is wrong. A year has 8760 hours, 0.05% of
that time the service can be down ->  8760 * 0.0005 = 4.38 hours of
downtime.

On Oct 8, 7:52 pm, Joshua Smith  wrote:

Google had a lot of trouble with their original database, known as M/S.  It was 
responsible for the downtime you heard about.

So they invented a new database, known as HR, and it has never gone down since 
they released it.

So if you use HR (which is the default for new projects), history suggests that 
you will not see outages.

But be realistic: Everything goes down. Everything. There is no 100% uptime 
guarantee.  Google is promising 99.95% which means they are OK with 438 hours 
of downtime a year. And that's only a soft promise (like most SLAs, your 
recourse is negligible if you suffer losses because of their failures).

I'm quite sure that GAE on HR has better uptime than I could achieve with 
anything I could build and manage.

On Oct 8, 2011, at 11:42 AM, ahmed adel wrote:








What do you mean by HR?
What I meant is , If I have application hosted in Google Apps Engine ,
will it be affected in the future for example due to data center crash
like what happened before.
We will need this application be available starting from 7 AM to 1 PM
Cairo time. because of this I am asking about the crash time in Google
Apps engine.
I am asking this because my manager wanna be sure our app won't be
available for use. (internal and external).
Also I heard that Google apps engine was down many times and according
to this many companies changed the host from Apps engine to another
PAAS provider.
Another reason for my question is I suggested to develop over Google
Apps Engine for our Customers , but also I told Our management that we
can not trust Google Apps Engine to do this for now , because as you
know and correct me if I am wrong (Google don't care much about Apps
Engine Support)
so my question will be now , can I trust Google Apps engine to use it
for our Clients? in case of yes or no please tell me reasons for your
answer.
On Oct 6, 7:50 pm, Joshua Smith  wrote:

Depends how you define "down."
If you only consider HR, it hasn't gone completely down ever.  Some bugs have 
happened (like losing the ability to send mail to gmail addresses with .'s in 
them) which reduced functionality for HR, and there have been times when you 
could not update your app.  But I don't think HR has ever stopped serving since 
it was introduced.
M/S has been down lots of times, sometimes for extended periods. But that's 
irrelevant if you are using HR, which you should be if you care about downtime.
-Joshua
On Oct 6, 2011, at 1:42 PM, ahmed adel wrote:

I need to know how many time did Google apps engine went down since
launching this service also I need to know for how many hours and what
is the time exactly.
Also I need to know your opinion about the below.
We are planning to develop a product over Google apps  engine , a big
project actually for our company.
so should I start using Google apps engine or should I use another
PAAS and if so what is the recommended PAAS.
thx
--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group 
athttp://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group 
athttp://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Prerelease SDK 1.5.5 available for download!

[Steve Sherrie]

Ikai, is the plan to include the ability to use the 2.7 runtime locally in 
the 1.5.5 final release? or will it have to wait?

Steve

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/Vc3Ksw2YJqYJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: Prerelease SDK 1.5.5 available for download!

[Steve Sherrie]

To be clear, this release is intended to work with Python2.7 locally?

If that is the case, I'm seeing this error (works as expected when 
deployed):
*Encountered error loading module "main.application": : Could not find module main.application*
*
*
...where main.application is denoted like this:

*- url: /.**
*  script: main.application*

Anyone else dealing with the same issue?

A note: I also had to manually set *os.environ['APPENGINE_RUNTIME'] = 
'python27' *before importing webapp, otherwise I'd end up with regular 
webapp instead of webapp2.


Any info.., and all that...

Steve

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/F8bCmLZJR_AJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: calling out the app engine team on ssl for custom domains

[Steve Sherrie]

There ya go then.

Steve

On 11-09-24 03:31 PM, Brandon Wirtz wrote:

You will be most unhappy.  Many of my SEO clients are FORMER cloudflare
users who found out the hard way that when Cloudflare mistakes Google Bot
for a DDoS Attack they get delisted, or when one of Google's Human Quality
validation techs visit the site and get a captcha challenge they get
Delisted.

Cloudflare also has issues that because adult sites often use it, that they
get blocked by large organizations firewall.


-Original Message-
From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Steve Sherrie
Sent: Saturday, September 24, 2011 11:51 AM
To: google-appengine@googlegroups.com
Subject: Re: [google-appengine] Re: calling out the app engine team on ssl
for custom domains

I too wonder about the availability of cloud fare. I'd like to find out how
it goes for you if you end up using it.

Steve


On 11-09-24 02:48 PM, johnP wrote:

Jeff - thanks for the link to Cloudflare.  It certainly seems like an
interesting option. The risk with implementing them is that it's one
more layer that can fail.  Does anyone else have feedback about how
well it works?Also, does anyone have thoughts on whether the other
benefits of Cloudflare (edge-caching and security layer) is relevant
to Appengine users, is it essentially redundant?





On Sep 23, 2:50 am, Jeff Schnitzer   wrote:

Try this out:

http://blog.cloudflare.com/ssl-on-custom-domains-for-appengine-and-ot
her

When our business is ready to launch, this is our intended plan for
always-SSL for our app.  Cloudflare does edge caching for all content
too - possibly better than Google's (wholly undocumented and not
guaranteed) edge cache.

I hate to plug a solution that I haven't tried yet, but it looks
promising.  Try it out and let me know, or you can wait for my report
towards the end of the year.

Jeff

--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.




--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: calling out the app engine team on ssl for custom domains

[Steve Sherrie]

I too wonder about the availability of cloud fare. I'd like to find out 
how it goes for you if you end up using it.


Steve


On 11-09-24 02:48 PM, johnP wrote:

Jeff - thanks for the link to Cloudflare.  It certainly seems like an
interesting option. The risk with implementing them is that it's one
more layer that can fail.  Does anyone else have feedback about how
well it works?Also, does anyone have thoughts on whether the other
benefits of Cloudflare (edge-caching and security layer) is relevant
to Appengine users, is it essentially redundant?





On Sep 23, 2:50 am, Jeff Schnitzer  wrote:

Try this out:

http://blog.cloudflare.com/ssl-on-custom-domains-for-appengine-and-other

When our business is ready to launch, this is our intended plan for
always-SSL for our app.  Cloudflare does edge caching for all content
too - possibly better than Google's (wholly undocumented and not
guaranteed) edge cache.

I hate to plug a solution that I haven't tried yet, but it looks
promising.  Try it out and let me know, or you can wait for my report
towards the end of the year.

Jeff


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: How to throttle or batch transactional updates to an entity group

[Steve Sherrie]

Yeah it's a pretty in depth topic, and I know I've watched the video 
several times over. I hope it works for you :)


Steve


On 11-09-23 07:06 PM, Nikolaj wrote:

Steve, I thought I had gotten everything out of that talk but clearly
I was sleeping in class!

The memcache fork join queue looks very interesting - I'm definitely
giving that a try.

Nikolaj

On Sep 23, 1:00 am, Steve Sherrie  wrote:

Definitely checkout Brett Slatkin's Building high-throughput data
pipelines with Google App Engine
<http://www.google.com/events/io/2010/sessions/high-throughput-data-pi...>
I/O Session which should give you a fundamental understanding of the
caveats involved with batch writing.

You have to do a little work to make sure writes are not added to a
batch that's already applied (or is otherwise invalid), but Brett
explains how to do that with memcache locking.

Steve

On 11-09-22 07:11 PM, Brandon Wirtz wrote:








Start a another datastore that doesn't use the same entity, but uses an
incremental system.
On Write:
In a table with format
Writes to do, Entity to Overwrite, Data for Entity
1,Brandon Wirtz, +$120
2,Brandon Wirtz, -$120
3,Brandon Wirtz, +$0.66
Increment Write to do count, and store transaction.
If a  Read request for the entity comes in:
Get Entity
Get Latest committed "Write To Do Value"
Check Write Cache
Recalculate Value based on Entity Value and All Changes
On Batch:
For batch size calculate all values, and update entities
Store Late todo value as Last committed write.
You can now batch as often as every 2 seconds and not hit the entity write
limit.
-Original Message-
From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Nikolaj
Sent: Thursday, September 22, 2011 3:59 PM
To: Google App Engine
Subject: [google-appengine] How to throttle or batch transactional updates
to an entity group
Hi there,
I'm developing an application that requires me to create lots of child
entities within the same entity group transactionally. The tasks being
generated to do this come in unpredictable bursts - sometimes 100/s for an
entity group. This is clearly exceeding the 1/s limit for transaction
contention and causing some real problems for my poor application.
I need a way to either batch the required updates for each entity group so I
can run through them serially or otherwise schedule each update to run on a
1/s basis. The ideas I've come up with so far:
1. Use some kind of memcache'd timestamp or counter to schedule or throttle
the updates. My question is how to do this in a way that will work with
100's of tasks hungrily looking for their time slot? Should I do a CAS for
when the last task was scheduled (would this even work with the contention
we're talking about?) or can I use an atomic operation like incr() somehow?
2. Store the work items somewhere and then at some later time query for the
work items for an entity group and create the children in serial. I'd like
to avoid this, if possible, as it creates many other headaches. My
application is already using the taskqueue to great effect.
This really seems like a queue problem - if only there was some kind of
dynamic subqueue where I could add tasks with a 'namespace' of the entity
group. Then entity groups could run in parallel, but the entity group items
would still be in serial. Does something like that exist?
Any suggestions or experiences on how I can work around this would be
appreciated.
Many thanks,
Nikolaj
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] How to throttle or batch transactional updates to an entity group

[Steve Sherrie]

Definitely checkout Brett Slatkin's Building high-throughput data 
pipelines with Google App Engine 
 
I/O Session which should give you a fundamental understanding of the 
caveats involved with batch writing.


You have to do a little work to make sure writes are not added to a 
batch that's already applied (or is otherwise invalid), but Brett 
explains how to do that with memcache locking.



Steve



On 11-09-22 07:11 PM, Brandon Wirtz wrote:

Start a another datastore that doesn't use the same entity, but uses an
incremental system.

On Write:
In a table with format
Writes to do, Entity to Overwrite, Data for Entity
1,Brandon Wirtz, +$120
2,Brandon Wirtz, -$120
3,Brandon Wirtz, +$0.66

Increment Write to do count, and store transaction.


If a  Read request for the entity comes in:
Get Entity
Get Latest committed "Write To Do Value"
Check Write Cache
Recalculate Value based on Entity Value and All Changes

On Batch:
For batch size calculate all values, and update entities
Store Late todo value as Last committed write.

You can now batch as often as every 2 seconds and not hit the entity write
limit.


-Original Message-
From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Nikolaj
Sent: Thursday, September 22, 2011 3:59 PM
To: Google App Engine
Subject: [google-appengine] How to throttle or batch transactional updates
to an entity group

Hi there,

I'm developing an application that requires me to create lots of child
entities within the same entity group transactionally. The tasks being
generated to do this come in unpredictable bursts - sometimes 100/s for an
entity group. This is clearly exceeding the 1/s limit for transaction
contention and causing some real problems for my poor application.

I need a way to either batch the required updates for each entity group so I
can run through them serially or otherwise schedule each update to run on a
1/s basis. The ideas I've come up with so far:

1. Use some kind of memcache'd timestamp or counter to schedule or throttle
the updates. My question is how to do this in a way that will work with
100's of tasks hungrily looking for their time slot? Should I do a CAS for
when the last task was scheduled (would this even work with the contention
we're talking about?) or can I use an atomic operation like incr() somehow?

2. Store the work items somewhere and then at some later time query for the
work items for an entity group and create the children in serial. I'd like
to avoid this, if possible, as it creates many other headaches. My
application is already using the taskqueue to great effect.

This really seems like a queue problem - if only there was some kind of
dynamic subqueue where I could add tasks with a 'namespace' of the entity
group. Then entity groups could run in parallel, but the entity group items
would still be in serial. Does something like that exist?

Any suggestions or experiences on how I can work around this would be
appreciated.

Many thanks,

Nikolaj

--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.




--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Faster than get_by_key_name

[Steve Sherrie]

get_by_key_name first computes a db.Key from the key_name and then 
fetches the entity by that key, so it's already the quickest method. 
There's no practical time cost in the computing of the key.


Steve Sherrie

On 11-09-20 06:16 PM, yamadaag wrote:

Hello.

I often use "get_by_key_name" to get entities from Datastore.
I'm looking for faster way to get entities, does anyone know?





--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Give me my $50 credit back!

[Steve Sherrie]

*Better call Saul!*


On 11-09-13 09:13 PM, Carson Welsh wrote:

The deal was: "modify your billing settings and you get a $50 credit".

I got the credit, disabled billing and now my credit's gone.

I believe you are contractually obligated to restore my $50 credit: 
you never said anything about whether billing should be kept enabled 
or not.


Don't you guys ever get tired of bait and switch?

Don't you guys get tired of alienating App Engine developers?
--
You received this message because you are subscribed to the Google 
Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Unexpected Entity Group transaction contention

[Steve Sherrie]

Testing is happening presently for multi-entity-group transactions I 
believe. I saw a thread on it about a week ago.


Steve



On 11-09-13 03:23 PM, Brian Olson wrote:
Re-reading the documentation, this kinda makes sense, but it bit me 
recently so I want to tell the story and see what others think.


I make an entity Parent(). Some time later I make an entity 
Child(parent=some_parent) and I do this in a transaction. I do this a 
bunch, concurrently from task-queue entries.


I was surprised to learn that simply creating a Child in a 
transaction, without otherwise doing anything to the parent, neither 
.get() nor .put(), locks the parent and all its children.


def txn_make_child(some_parent):
  foo = Child(parent=some_parent)
  foo.put()
  # also transactionally enqueue a task to operate on the Child 
instance foo


Code very much like that was failing out due to too many transaction 
retries. I didn't expect /any/ transaction contention, because I 
thought I was just creating an object and enqueueing a task, and those 
were the only two things in the transaction in my head. But it turns 
out the above code locks some_parent and all its children. Boo.


I think I was expecting things like this to lock parent and all its 
children:

def txn_p_c_example(parent_key, child_key):
  parent = db.get(parent_key)
  child = db.get(child_key)
  # now they're clearly both involved, and involving the parent winds 
up locking all the children. I can accept that.

  parent.put()
  child.put()

I was able to re-code it to make Child have no ancestor, but there are 
still times when I would much rather still commit parent and child at 
exactly the same time.

--
You received this message because you are subscribed to the Google 
Groups "Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/NsvS8Fcq_EwJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] The Unofficial Google App Engine Price Change FAQ

[Steve Sherrie]

I was thinking the same! Looks lovely but after concentrating on the  
text, those headings were painfully bright :)


Great job though! Great explanation of everything.


Steve

On 11-09-09 08:33 PM, Ikai Lan (Google) wrote:

Thanks for this, Jeff!

But dude, grey on black?!

--
Ikai Lan
Developer Programs Engineer, Google App Engine
plus.ikailan.com  | twitter.com/ikai 





On Fri, Sep 9, 2011 at 5:05 PM, Jeff Schnitzer > wrote:


Here ya go:


http://blorn.com/post/10013293300/the-unofficial-google-app-engine-price-change-faq

Jeff

--
You received this message because you are subscribed to the Google
Groups "Google App Engine" group.
To post to this group, send email to
google-appengine@googlegroups.com
.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google 
Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: two questions regarding backends (configuring them and addressing them)

[Steve Sherrie]

Oh yeah, I guessed the *--backends* flag on *appcfg.py update *was 
intended as a replacement for running a separate backends update each time.


Steve

On 11-09-09 06:22 PM, Rishi Arora wrote:
Found my problem.  Looks like I must execute that appcfg.py update 
backends command each time I make a code change.  It seems obvious 
now, but of course, it never occurred to me!  I'm all good now, even 
with countdown set to 0, and tasqueue target set to either 
instance.backend_name or just backend_name.


It also might seem silly that I'm making my backend queue up requests 
for itself (in an attempt to break up requests), when in fact it could 
process the whole big request inline in one shot.  But its actually 
common code that might also get executed by a front-end in some 
circumstances.  I should look into a run-time condition that I can use 
to determine whether I'm in the context of a front-end or a backend, 
and queue up requests to myself only for front-end instances.  Do you 
know if there's a way to do that?


Thanks for your help!

Regards
-Rishi


On Fri, Sep 9, 2011 at 4:34 PM, Rishi Arora <mailto:rishi.ar...@ship-rack.com>> wrote:


I came across this

(http://code.google.com/appengine/docs/python/taskqueue/overview-push.html#Push_Queues_and_Backends).
 Which suggests that I should be able to use a specific backend
instance.  Not only was the backend started (really, enabled), but
it was up and running too, from serving up a request recently.  In
fact, I'm enqueueing the taskqueue request from the backend itself
(the backend logs prove it too).  I tried your method too (using
just the backend name, and no instance prefix), and that didn't
work either.  The only difference is that I'm not passing any
"params" parameter in the taskqueue.add() call.  The request is
just meant to be a trigger and does not carry any data.  Perhaps
the app engine scheduler thinks that my backend is busy (cause it
was processing the current request which enqueued the new
taskqueue request), and that causes it to send the request to a
front-end instance.  Maybe I should send the taskqueue rquest with
a "countdown" set to a comfortable 10 seconds or something.

Here's the snippet from the link above:

Push tasks typically must finish execution within 10 minutes. If
you have push tasks that require more time or computing resources
to process, you can use App Engine Backends to process these tasks
outside of the normal limits of App Engine applications. Backends
are addressable. When you create the push task, all you need to do
is address it to the URL of the backend you wish to process it.
The following code sample demonstrates how to create a push task
addressed to an instance 1 of a backend named backend1:
from google.appengine.api import taskqueue
...
def post(self):
key = self.request.get('key')

# Add the task to the default queue.
taskqueue.add(url='/path/to/my/worker/', params={'key':
key}, target='1.backend1')



On Fri, Sep 9, 2011 at 4:10 PM, Steve Sherrie
mailto:st...@wasteofpaper.com>> wrote:

I'm able to direct tasks to the backends using the *target*
arg as in your example. And the backend state is 'start' when
you're trying this?

Are you directing toward a specific backend instance
(/N.backend_name/) or just to /backend_name/? I am doing the
latter.


Steve



On 11-09-09 05:00 PM, Rishi Arora wrote:

Thanks.  Yes that worked for me.  It just seems that backend
documentation isn't very intuitive.  I couldn't tell from any
of the online docs that simply adding a backend to
backends.yaml won't configure a backend for you.  It won't
show up in the backends section of the admin console until
you execute appcfg.py update backends.

Another non-intuitive behavior - probably only for dynamic
backends.  The admin console has a start/stop button.  But
obviously, if you hit start, the dynamic backend won't
actually start.  That's clear from the documentation.  But,
if you hit "stop", it almost looks like that the backend is
made inaccessbile.  It won't process requests, and it won't
automatically restart on the next request.  You have to hit
the "start" button first.  I think for dynamic backends that
button should read "enable/disable".

Now the next thing I'm trying to get to work is sending a
backend a request from a task queue.  I'm using the default
task queue to do this, and simply doing this doesn't work (a
front-end

Re: [google-appengine] No Google, It's Not Us, It's You!

[Steve Sherrie]

But in all seriousness, you're saying it would be a gargantuan feat to 
improve those figures? 

To get such a extreme difference (3min cpu => 28+ frontend instance hours) 
you're probably heavily relying on synchronous RPCs in some way, and that's 
exactly the kind of thing that can be made substantially more cost-efficient 
under the new billing with relatively little effort. Is this the case?

Stove (I saw bridesmaids yesterday)

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/xNhQScOpLG4J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] No Google, It's Not Us, It's You!

[Steve Sherrie]

No offense (some offense), but it sounds like your application is wildly 
inefficient. You should crowd source a solution by offering up your code 
for review/improvement.


Steve Sherrie


On 11-09-09 05:11 PM, Steve wrote:
Yeah, great.  So when my app farts and uses 3 minutes of CPU time I 
can look forward to just barely blowing past the free quote by a half 
hour.  Yay...


Original Steve

--
You received this message because you are subscribed to the Google 
Groups "Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/HPOjtpKkzHEJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Re: two questions regarding backends (configuring them and addressing them)

[Steve Sherrie]

I'm able to direct tasks to the backends using the *target* arg as in 
your example. And the backend state is 'start' when you're trying this?


Are you directing toward a specific backend instance (/N.backend_name/) 
or just to /backend_name/? I am doing the latter.



Steve


On 11-09-09 05:00 PM, Rishi Arora wrote:
Thanks.  Yes that worked for me.  It just seems that backend 
documentation isn't very intuitive.  I couldn't tell from any of the 
online docs that simply adding a backend to backends.yaml won't 
configure a backend for you.  It won't show up in the backends section 
of the admin console until you execute appcfg.py update backends.


Another non-intuitive behavior - probably only for dynamic backends. 
 The admin console has a start/stop button.  But obviously, if you hit 
start, the dynamic backend won't actually start.  That's clear from 
the documentation.  But, if you hit "stop", it almost looks like that 
the backend is made inaccessbile.  It won't process requests, and it 
won't automatically restart on the next request.  You have to hit the 
"start" button first.  I think for dynamic backends that button should 
read "enable/disable".


Now the next thing I'm trying to get to work is sending a backend a 
request from a task queue.  I'm using the default task queue to do 
this, and simply doing this doesn't work (a front-end instance 
processes the taskqueue request):


taskqueue.add(url=[relative path to request handler], method='POST', 
target=[backend name from app.yaml])


Modifying the above statement to specify the entire URL for the 
backend in the "target" parameter does not help either - the taskqueue 
request is still processed by a front-end instance.  I also tried just 
getting rid of the "target" parameter, and specifying the entire url 
in the "url" parameter i.e. 
url='https://backend.appid.appspot.com/path/to/requesthandler'.  And 
even that did not help.  I fear forcing backends to process cron 
entries defined with a full URL like above won't do the trick either.


So is there no way to force a backend to process a request?



On Fri, Sep 9, 2011 at 3:45 PM, Steve Sherrie <mailto:st...@wasteofpaper.com>> wrote:


Also, you can do...
*
*
*appcfg.py update **--backends your_app*
*
*
*
*
Steve
-- 
You received this message because you are subscribed to the Google

Groups "Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/gvpp7TaOlykJ.

To post to this group, send email to
google-appengine@googlegroups.com
<mailto:google-appengine@googlegroups.com>.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
<mailto:google-appengine%2bunsubscr...@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google 
Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] No Google, It's Not Us, It's You!

[Steve Sherrie]

That's funny, cause Google just upped the free instance hours to 28 from 
24, so you'd be about the same, without any optimizations.


Steve

On 11-09-09 04:18 PM, Steve wrote:
Your new pricing scheme blows no matter how reasonable you try to make 
it sound.





--
You received this message because you are subscribed to the Google 
Groups "Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/fDnpZWr9GhQJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: two questions regarding backends (configuring them and addressing them)

[Steve Sherrie]

Also, you can do...
*
*
*appcfg.py update **--backends your_app*
*
*
*
*
Steve

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/gvpp7TaOlykJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: two questions regarding backends (configuring them and addressing them)

[Steve Sherrie]

Also, you can do...
*
*
*appcfg.py update **--backends /your_app*
*
*
*
*
Steve

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/7tFwG2wYt_8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] If the Scheduler worked right (and it doesn't)...

[Steve Sherrie]

It's great to see such a thorough breakdown of a real life request stream. 
It demonstrates the complexity of what might otherwise look like a simple 
task (scheduling efficiently).

About the Python 2.7 testing Will there be more testers accepted in the 
near future? I'm very keen to try it out, but I think I missed the boat for 
the first round.


Steve

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/google-appengine/-/0u_7J4NE-zEJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Google Bot Is Your Enemy

[Steve Sherrie]

Joshua, can you see my posts? Not sure why I can't post

On Sep 3, 4:15 pm, Joshua Smith  wrote:
> Is there some way to get google bot requests to go to a dedicated back end?
>
> On Sep 3, 2011, at 3:35 PM, Brandon Wirtz wrote:
>
>
>
>
>
>
>
> > The biggest Problem I have with the Scheduler/GAE isn’t GAE it is Google 
> > Bot.
>
> > Under the new model you are on the hook for 15 minutes of time for an 
> > instance that spins up.  Google Bot can’t be throttled on GAE.  If you go 
> > in to WebMasters Tools you get a “Your site has been assigned a special 
> > crawl rate”.  This was my favorite feature when I was paying for CPU 
> > cycles, but now when Google bot shows up and makes upwards of 100k requests 
> > in 5 minutes every 6 hours, Google Bot will be accounting for about 85% of 
> > the cost of hosting on several of the sites that have less than $50 a month 
> > in hosting costs under the current plan, that will be paying about $400 a 
> > month to serve requests to Google come November Pricing.
>
> > -Brandon
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude

[Steve Sherrie]

I've create this feature request for a way to create scheduler
profiles and then filter traffic into those profiles.
Star it if you think it would be useful.

http://code.google.com/p/googleappengine/issues/detail?id=5775

Steve

On Sep 3, 4:14 pm, Joshua Smith  wrote:
> This point is not so simple:
>
> > (and unfortunately may remain
> > that way because instance optimization is markedly revenue and profit
> > negative for GAE)
>
> It really is not in google's economic interest to have idle instances 
> running.  More idle instances means more infrastructure cost, and, if you 
> charge for it, more user backlash.  Idle instances are a necessary evil when 
> you want responsive, dynamic scaling.  But technically, you only need as many 
> as it takes to satisfy the simple equation (rate of join * max wait) / 
> startup time.  That is, you need to have enough on hand to handle the 
> incoming requests while you spin up some more.
>
> The key problem is that for probably all applications, the numerator in that 
> equation is completely different for tasks than it is for users browsing the 
> site.  And it is often different for different kinds of tasks.  (In fact, 
> it's often different for different users/urls/handlers as well, but sometimes 
> too much configurability is a bad thing.)
>
> The scheduler isn't as complicated as many people seem to believe.  The only 
> real magic in it is the heuristics for computing the three values in that 
> equation when the user sets them to "automatic."  The rest is simple trend 
> averaging.
>
> -Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

Re: [google-appengine] Some more real-world low-traffic scheduling data -- and a diagnosis

[Steve Sherrie]

The dynamic backends are spawned for a minimum of one hour (I believe), 
so the same problem (but worse) will exist in that case.


Steve

On 11-09-02 10:45 PM, Gregory D'alesandre wrote:

Hey Joshua,

This might be crazy, but with free apps you get 9 hours of backends 
per day for free as well.  Could you have the kiosk hit a dynamic 
backend and only users go through the frontends?


Greg

On Fri, Sep 2, 2011 at 12:55 PM, Joshua Smith 
mailto:joshuaesm...@charter.net>> wrote:


My app gets hit by a kiosk about every 3 minutes, and that chews
up about 1sec of CPU, and kicks off a task that eats another
250ms.  Users come and go now and then.  Today I tried 3 different
settings:

< 9am: Max idle = 1, max latency = 1s
9am - 1pm: Max idle = auto, max latency = 1s
1pm - now: Max idle = 1, max latency = 5s



For low-traffic, it appears that max idle = 1 is the same as
"auto".  That makes sense.  I'm going to leave it at auto, since
I'd like to be able to handle spikes well.

It looks like when I set the max latency way up, it sometimes lets
the second instance die, but never for very long.

When I've looked at the instances they never seem to have been
alive all that long.  So I think the scheduler is spinning up one,
adding another, letting the first die, adding another, and so on.

Right now I'm in one of those 1 instance troughs.  The site is
quite responsive, while I poke around, and it isn't starting
another instance.  So here in the trough, I'm getting the same
behavior that was reported by the person who tried a "hello world"
test.  Yet the above graph is what it is.

As I'm writing this, the number of instances just jumped up to 2!
 So now I can see what caused it:

The kiosk hits the URL with an XMLHttp request to make sure it's
alive, and if that works OK, then it refreshes itself.  This bit
of nastiness is there because it's impossible to get a browser to
handle failed page loads 100% consistently well.

When the kiosk hits the URL, a task is launched to do some
background processing recording that the heartbeat happened.   The
two hits therefore lead to two tasks.  I'm using countdown = 1.

These two tasks are bunching up in the queue, and being processed
essentially at the same time.  That requires, you guessed it, two
instances!

I have been thinking of task queues as sort of a background
process, and I certainly wouldn't expect the system to spin up an
instance just to handle a queued task.  But that thinking isn't
really right, since the docs explicitly suggest that spawning a
lot of tasks is a good way to get a bunch of instances to munch
your hard problem in parallel.

So my fix is actually not so hard.  I just need to pass a param
with the "just checking" initial kiosk request and use that to
avoid spawning the task.  That way I'll get hit-hit-task, not
hit-hit-task-task, and presumably the system won't feel compelling
to crank up a new instance.

If we get two kiosks going and they happen to get synchronized (as
such things tend to do), then I'll be screwed.  But for now, I
think I've got  my fix...

If you're still reading, I'll give you a reward: If you are trying
to diagnose why you have 2 instances when you have the sliders set
to 1/15, go to your log, view with Info and find the requests that
are spinning up a new instance.  Now look at all requests and find
that one that spun up the evil second instance.  Was it right on
the heels of another request?  I bet it was.  Is it your fault?
 (In my case, it certainly was my fault.)  Regardless, if you want
to avoid that second instance, you need to find a way to get those
requests to be farther apart.

-Joshua

-- 
You received this message because you are subscribed to the Google

Groups "Google App Engine" group.
To post to this group, send email to
google-appengine@googlegroups.com
.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google 
Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.


--
You received this message because you are subscribed to the Google Groups "Google 
App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@

[google-appengine] Re: Google Bot Is Your Enemy

[Steve Sherrie]

Joshua,

I just read your post about idle instances, and scheduling, and it
makes a lot of sense. Also given this google bot stuff, I wonder if it
would be useful to have the ability to filter traffic into different
'types', each of which have different scheduling requirements.

Steve

On Sep 3, 4:15 pm, Joshua Smith  wrote:
> Is there some way to get google bot requests to go to a dedicated back end?
>
> On Sep 3, 2011, at 3:35 PM, Brandon Wirtz wrote:
>
>
>
>
>
>
>
> > The biggest Problem I have with the Scheduler/GAE isn’t GAE it is Google 
> > Bot.
>
> > Under the new model you are on the hook for 15 minutes of time for an 
> > instance that spins up.  Google Bot can’t be throttled on GAE.  If you go 
> > in to WebMasters Tools you get a “Your site has been assigned a special 
> > crawl rate”.  This was my favorite feature when I was paying for CPU 
> > cycles, but now when Google bot shows up and makes upwards of 100k requests 
> > in 5 minutes every 6 hours, Google Bot will be accounting for about 85% of 
> > the cost of hosting on several of the sites that have less than $50 a month 
> > in hosting costs under the current plan, that will be paying about $400 a 
> > month to serve requests to Google come November Pricing.
>
> > -Brandon
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.

[google-appengine] Re: Class Action Lawsuit ?

[Steve Sherrie]

'My Cousin Vinnie' is like, my FAVOURITE movie!!1

On Sep 1, 2:31 pm, Will  wrote:
> Well, consider what kind of in-house lawyers they have. Looking into
> the user agreement carefully, I bet you can find pretty much 'we can
> do whatever we like, especially it is a three year long preview
> period'.
>
>
>
>
>
>
>
> On Thu, Sep 1, 2011 at 11:25 AM, zdravko  wrote:
> > Plenty of "bad faith" practices.
> > Some "monopoly" practices.
> > Ambulance chasers' dream.
>
> > On Sep 1, 2:20 pm, Angke Chen  wrote:
> >> Can we do it? I mean it was in Preview when it was cheap.
>
> >> Angke
>
> >> On Thu, Sep 1, 2011 at 11:18 AM, zdravko  wrote:
> >> > Why not and especially in America ?
>
> >> > --
> >> > You received this message because you are subscribed to the Google 
> >> > Groups "Google App Engine" group.
> >> > To post to this group, send email to google-appengine@googlegroups.com.
> >> > To unsubscribe from this group, send email to 
> >> > google-appengine+unsubscr...@googlegroups.com.
> >> > For more options, visit this group 
> >> > athttp://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Google App Engine" group.
> > To post to this group, send email to google-appengine@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > google-appengine+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.