[google-appengine] Re: Help resolve massive performance regression in 2.7 vs 2.5 runtime

2011-11-14 Thread Pol
Hi Brian, So on December 1st, the 50% discount for front-instances is gone. The idea is to compensate by switching to Python 2.7 with multithreading, but it looks like at this point it's a lose-lose situation: it runs more requests at the same time, but they take longer. We're mid-

[google-appengine] Re: Help resolve massive performance regression in 2.7 vs 2.5 runtime

2011-11-13 Thread Pol
the payment change 1 more month to January 1st 2012 wouldn't hurt :) On Nov 12, 10:09 pm, Brian Quinlan bquin...@google.com wrote: Hi Pol, Thanks for getting back to me. On Sun, Nov 13, 2011 at 4:50 PM, Pol p...@everpix.net wrote: Hi Brian, threadsafe is true There is a known issue

[google-appengine] Help resolve massive performance regression in 2.7 vs 2.5 runtime

2011-11-12 Thread Pol
a native Python implementation of the py-bcrypt package from http://www.mindrot.org/projects/py-bcrypt/; grabbed from here: https://github.com/erlichmen/py-bcrypt. So what's happening here and how can we fix this? Thanks, - Pol -- You received this message because you are subscribed to the Google

[google-appengine] Re: Help resolve massive performance regression in 2.7 vs 2.5 runtime

2011-11-12 Thread Pol
Hi Brian, threadsafe is true Here's an example password_hash for an original 8 characters password: $2a$04$cbM2uHwDphIG3jFFRpq1mui5aVjevnDUwhvQ77S/WG/qvJMpiXAL6 On Nov 12, 6:58 pm, Brian Quinlan bquin...@google.com wrote: Hi Pol, On Sun, Nov 13, 2011 at 1:48 PM, Pol p...@everpix.net wrote

[google-appengine] Re: Python 2.7 multithreaded: ereporter mapreduce compatibility

2011-11-12 Thread Pol
: google.appengine.ext.mapreduce.application login: admin Thanks for your help! On Nov 11, 1:26 pm, Brian Quinlan bquin...@google.com wrote: Hi Pol, On Sat, Nov 12, 2011 at 7:46 AM, Pol p...@everpix.net wrote: Indeed the problem is the parsing of app.yaml: - url: /mapreduce(/.*)?  script: $PYTHON_LIB/google/appengine/ext

[google-appengine] Re: Python 2.7 multithreaded: ereporter mapreduce compatibility

2011-11-11 Thread Pol
? Looking into the SDK source, these extensions are not 2.7 multithreaded ready. On Nov 10, 6:10 pm, Brian Quinlan bquin...@google.com wrote: On Fri, Nov 11, 2011 at 2:44 AM, Pol p...@everpix.net wrote: Hi, What are the plans to get these 2 extensions compatible with Python 2.7 multithreaded

[google-appengine] Re: Python 2.7 multithreaded: ereporter mapreduce compatibility

2011-11-11 Thread Pol
Ah OK it works, thanks. Didn't know had to replace path entirely... Was wondering about compatibility since there were still some CGI leftovers in these files. On Nov 11, 1:26 pm, Brian Quinlan bquin...@google.com wrote: Hi Pol, On Sat, Nov 12, 2011 at 7:46 AM, Pol p...@everpix.net wrote

[google-appengine] Python 2.7 multithreaded: ereporter mapreduce compatibility

2011-11-10 Thread Pol
Hi, What are the plans to get these 2 extensions compatible with Python 2.7 multithreaded? $PYTHON_LIB/google/appengine/ext/mapreduce/main.py $PYTHON_LIB/google/appengine/ext/ereporter/report_generator.py Thanks, - Pol -- You received this message because you are subscribed to the Google

[google-appengine] How to set up task queues to prevent bursts?

2011-10-30 Thread Pol
the majority fails and so on. So how can we configure a task queue to have a very high rate *but* prevent it to get to this rate too fast? I noticed the bucket size parameter, but I'm not sure how to use it properly. Thanks, - Pol -- You received this message because you are subscribed

[google-appengine] Re: How to set up task queues to prevent bursts?

2011-10-30 Thread Pol
in queue.yaml/xml This is documented for Java, but the same general rules apply for Python or Go applications.http://code.google.com/appengine/docs/java/config/queue.html#Defining... Nick Verne On Mon, Oct 31, 2011 at 3:13 PM, Pol p...@everpix.net wrote: Hi, Through a task queue, we

[google-appengine] Re: How to set up task queues to prevent bursts?

2011-10-30 Thread Pol
the initial errors but at least you'll get the queues cleared eventually. Jeff On Sun, Oct 30, 2011 at 10:07 PM, Pol p...@everpix.net wrote: Sorry, I realize I didn't explain everything: since the external system can handle an arbitrary load (as long as this load doesn't grow too fast

[google-appengine] Re: Announcing SSL for Custom Domains Trusted Tester Program

2011-10-18 Thread Pol
That's great news, but I'm wondering: if we put GAE over SSL on https://example.com, will doing AJAX from https://www.example.com to https://example.com still be considered cross-domain or not? On Oct 18, 9:22 am, Jeff Schnitzer j...@infohazard.org wrote: +1 to this! I *do* need it... because

[google-appengine] Re: Announcing SSL for Custom Domains Trusted Tester Program

2011-10-18 Thread Pol
That's great news, but I'm wondering: if we put GAE over SSL on https://example.com, will doing AJAX from https://www.example.com to https://example.com still be considered cross-domain or not? On Oct 18, 9:22 am, Jeff Schnitzer j...@infohazard.org wrote: +1 to this! I *do* need it... because

[google-appengine] Re: Announcing SSL for Custom Domains Trusted Tester Program

2011-10-18 Thread Pol
That's great news, but I'm wondering: if we put GAE over SSL on https://example.com, will doing AJAX from https://www.example.com to https://example.com still be considered cross-domain or not? On Oct 18, 9:22 am, Jeff Schnitzer j...@infohazard.org wrote: +1 to this! I *do* need it... because

[google-appengine] Need help tracking datastore transaction or write too big warnings

2011-10-14 Thread Pol
size? What are these exact limits? Thanks, - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr

[google-appengine] Re: Need help tracking datastore transaction or write too big warnings

2011-10-14 Thread Pol
Can anyone comment from Google comment on this 2nd point: write limits on entire transaction? Our entities are 4KB on average so I'm trying to track down what's happening but debugging on the server in a live product is really not trivial. On Oct 14, 12:25 pm, Simon Knott knott.si...@gmail.com

[google-appengine] date value out of range exception while committing transaction

2011-10-12 Thread Pol
Hi, I just saw some cases of run_in_transaction() throwing an exception date value out of range. What does this mean and how do I prevent this? Thanks, - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send

[google-appengine] Getting lots of Exception sending Rollback in the log

2011-10-12 Thread Pol
self.check_rpc_success(rpc) File /base/python_runtime/python_lib/versions/1/google/appengine/ datastore/datastore_rpc.py, line 1182, in check_rpc_success raise _ToDatastoreError(err) BadRequestError: The referenced transaction has expired or is no longer valid. - Pol -- You received

[google-appengine] Dashboard says 5.69GB of High Replication Data but Datastore Statistics report 2GB of entities?

2011-09-27 Thread Pol
Hi, Datastore Statistics do include the extra Metadata for entities, so where are these mysterious 3.69GB coming from?? - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine

[google-appengine] How to track down urlfetch() Application Error 2?

2011-09-26 Thread Pol
Hi, Several times a day we get some ApplicationError: 2 exceptions when communicating from GAE to EC2 through urlfetch(). How do I get more info to understand exactly what's happening? I don't see anything extra in the log and I'm not sure there's more info on the exception object itself. ...

[google-appengine] Custom Administration Console pages with admin login requirement result in redirect loops

2011-09-18 Thread Pol
: always - url: /admin/.* script: admin.py login: admin secure: always - url: / script: main.py secure: never - url: /register script: main.py secure: never - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group

[google-appengine] Need help with App Engine over HTTPS on IE8-9

2011-09-17 Thread Pol
www.everpix.net to gae.everpix.net, right? Thanks! - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr

[google-appengine] Re: Min idle instances setting not working at all?

2011-09-13 Thread Pol
:24 AM, Jon McAlister jon...@google.com wrote: What's the app-id? On Sun, Sep 11, 2011 at 9:53 AM, Pol-Online i...@pol-online.net wrote: Hi, Because we are about to launch our app very soon, I increased the number of idle instances from 1 to 5 yesterday and things looked correct

[google-appengine] Re: Min idle instances setting not working at all?

2011-09-13 Thread Pol
I typed too fast, I meant *max* idle instances in the Dashboard. Sorry about the confusion. That said, the description of the slider is not very clear: it's called max which is correct from a billing perspective, but from a functionality perspective, it really looks like you're setting the

[google-appengine] Re: Re-entrancy / deadlock bug with scheduler?

2011-09-11 Thread Pol
, Pol i...@pol-online.net wrote: Hi Jon, Great! everpix-alpha I still see what appears to be the bug now and then, but I don't have a reproducible case just waiting in a task queue anymore. On Sep 7, 1:19 pm, Jon McAlister jon...@google.com wrote: Hi Pol, I think I have a change

[google-appengine] Min idle instances setting not working at all?

2011-09-11 Thread Pol-Online
. Anyway, it looks like either the graph is wrong or the setting doesn't work, or the definition of what the setting does is wrong, because clearly it not stable at 5. I just activated Always On to work around the issue, so now it's back to 3+1. -Pol Pol-Online i

[google-appengine] Re: Min idle instances setting not working at all?

2011-09-11 Thread Pol
Oops looks like the image was stripped, you can see it here: http://oi52.tinypic.com/12349hi.jpg On Sep 11, 9:53 am, Pol-Online i...@pol-online.net wrote: Hi, Because we are about to launch our app very soon, I increased the number of idle instances from 1 to 5 yesterday and things looked

[google-appengine] Re: Why app engine new price model is totally wrong

2011-09-07 Thread Pol
on the perceived value of GAE is and what a fair price for the service, but putting this type of service together does require quite a bit of know how. - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email

[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.

2011-09-07 Thread Pol
Great to see the problem acknowledged, so why is the HDR line still all green at http://code.google.com/status/appengine? The HRD is having problems now (yes, not as bad pas a MS problems, but that's beside the point as the expectancies have been officially raised quite a bit for HRD),

[google-appengine] App Stats time measurements for GQL queries way off?

2011-09-07 Thread Pol
+ 0.006; link: http://everpix-alpha.appspot.com/_ah/stats/details?time=1315418651153 And the problem: App Stats states that Photo.get(...) took 1574ms while inline measurements show a much bigger 3.56s? Does App Stats ignore de- serialization of the entity properties or something? - Pol -- You

[google-appengine] Re: App Stats time measurements for GQL queries way off?

2011-09-07 Thread Pol
On Wed, Sep 7, 2011 at 11:12 AM, Pol i...@pol-online.net wrote: Hi, I'm trying to optimize our Python app and have App Stats installed. The code to measure (simplified):    query = db.GqlQuery(SELECT * FROM Event WHERE ANCESTOR IS :1 ORDER BY max_date DESC, self.user.key())    if cursor

[google-appengine] Re: Re-entrancy / deadlock bug with scheduler?

2011-09-07 Thread Pol
Hi Jon, Great! everpix-alpha I still see what appears to be the bug now and then, but I don't have a reproducible case just waiting in a task queue anymore. On Sep 7, 1:19 pm, Jon McAlister jon...@google.com wrote: Hi Pol, I think I have a change that will fix this, which should

[google-appengine] Re: App Stats time measurements for GQL queries way off?

2011-09-07 Thread Pol
difference in performance on AppEngine. Is TextProperty just slow? It's just used to store a few hundred bytes of EXIF as a repr()'d Python dict. On Sep 7, 1:29 pm, Pol i...@pol-online.net wrote: I don't know the exact count for this Photo.get(...) call but it can be around 2,000. If it's related

[google-appengine] Re-entrancy / deadlock bug with scheduler?

2011-09-05 Thread Pol
. - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options

[google-appengine] How to configure queue.yaml if you only care about max_concurrent_requests?

2011-09-05 Thread Pol
: ? max_concurrent_requests: N Thanks, - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com

[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.

2011-09-05 Thread Pol
Reviving this thread... GAE team: can we get an official comment on this really important issue? Thx On Sep 2, 1:14 pm, Pol i...@pol-online.net wrote: Our app has been on HRD since the beginning and indeed, the past few days for the time ever, I saw some timeouts. To be clear, not timeouts due

[google-appengine] Re: how to reindex a property that once set to indexed = false?

2011-09-05 Thread Pol
Talking abou this, what do you think of using bcrypt.hashpw(password, bcrypt.gensalt())? I've read in a few places it was supposed to be a good solution, but I discovered this morning that the AppEngine version, having to be pure Python, changes the default log_round for salt generation from 1024

[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.

2011-09-02 Thread Pol
This is really scary. I'm not so obsessed about the price increase as this is something you can control, but if the HRD starts becoming slower and slower, there's absolutely nothing you can do. According to these official graphs, it's literally 2X slower than 1 year ago. I'm OK with training some

[google-appengine] Re: Even the HR Datastore is (slightly) having performance issue now.

2011-09-02 Thread Pol
is stable and there are no datastore timeouts anymore. I am happy with the move. On Sep 2, 11:15 am, Pol i...@pol-online.net wrote: This is really scary. I'm not so obsessed about the price increase as this is something you can control, but if the HRD starts becoming slower and slower

[google-appengine] Re: Problems Deploying App - 500 Internal server Error

2011-08-01 Thread Pol
From my experience, when these happens they don't happen once, but for a continuous period of time say 15mn. So they appear to be reproducible 100% during this period. If you provide an direct email address people can use next time it happens, then the affected developer can provide the app id and

[google-appengine] Re: Really high request response time for trivial Python handlers (not explained by App Stats)

2011-08-01 Thread Pol
, Robert Kluin robert.kl...@gmail.com wrote: Sounds like scheduler issues. Do you see a pending ms in the logs (expand the entries)?  Have you adjusted the scheduler knobs under app settings in your app's dashboard? Robert On Jul 29, 2011 12:02 PM, Pol i...@pol-online.net wrote: Hi

[google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-29 Thread Pol
entity group. Hope this helps clarifies some points. Thanks much for your detailed answers indeed! - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe

[google-appengine] Really high request response time for trivial Python handlers (not explained by App Stats)

2011-07-29 Thread Pol
/api/user_info 200 2446ms 35cpu_ms 0kb Mozilla/ 5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0.1) Gecko/20100101 Firefox/ 5.0.1 As you can't see 5~23ms turned into 133~2446ms. What could possibly be happening? The latency clearly comes from outside my handler's code. Thanks for any help. - Pol

[google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-28 Thread Pol
On Jul 26, 9:40 pm, Jose Montes de Oca jfmontesde...@google.com wrote: Hi Pol, What this meas is that even if a transaction throws an exception this does not means the transaction failed, thats why you need to make your datastore transaction idempotent. So if you retry a transaction because

[google-appengine] Re: TransactionFailedError: Is the transaction committed or not?

2011-07-28 Thread Pol
I see two big problems here: 1) There appear to be absolutely no record, say in the dashboard, of failed transactions that eventually succeeded or really failed. For some type of apps, I could live with 1 in a million transaction failure that may or may not be eventually successful, but I need to

[google-appengine] Can sender address be spoofed in XMPP?

2011-07-28 Thread Pol
the HTTP headers for web requests? - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr

[google-appengine] 500 deployment errors, here we go again

2011-07-28 Thread Pol
server output --- - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more

[google-appengine] Re: 500 deployment errors, here we go again

2011-07-28 Thread Pol
Appears to be working again, indeed. On Jul 28, 6:47 pm, Chris Copeland ch...@cope360.com wrote: +1 Been happening the last 15 minutes or so. On Thu, Jul 28, 2011 at 11:38 AM, Pol i...@pol-online.net wrote: Anyone else observing? It's been failing for me for the last 20mn. Google

[google-appengine] Re: HR database, ancestor queries and data consistency: docs not 100% clear

2011-07-26 Thread Pol
. :) For the record, I did end up finding a bug in my code that explained the fact sometimes concurrent transactions with ancestor queries were apparently returning stalled results. - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post

[google-appengine] Re: HR database, ancestor queries and data consistency: docs not 100% clear

2011-07-25 Thread Pol
has possible race conditions. Same here: I was also assuming HR datastore + transaction + ancestor queries (by definition since inside a transaction). So is my code wrong or am I observing some very rare but expected race conditions? Thanks - Pol -- You received this message because you

[google-appengine] TransactionFailedError: Is the transaction committed or not?

2011-07-25 Thread Pol
transactions have been committed and eventually will be applied successfully. So if you get TransactionFailedError, to you need to execute the transaction again or will it automatically be applied later? - Pol -- You received this message because you are subscribed to the Google Groups Google App

[google-appengine] Re: Is anyone seeing deployment 500 server error now?

2011-07-18 Thread Pol
, please A HREF=http:// code.google.com/appengine/community.htmlreport/A your problem and mention this error message and the query that caused it./h2 h2/h2 /body/html --- end server output --- - Pol On Jul 17, 4:28 am, Tom Fishman tom.fish...@dishcrunch.com wrote: App deployment has been failing

[google-appengine] Model.get_or_insert() behavior on high-replication datastore

2011-07-16 Thread Pol
datastore if the entity key has no parent? - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr

[google-appengine] What happens to still pending async operations when the request completes (in Python)?

2011-07-14 Thread Pol
~100ms. Thanks, - Pol -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more

[google-appengine] Re: About expando properties

2011-07-13 Thread Pol
, 1:35 am, Pol i...@pol-online.net wrote: Hi, Regarding expando properties: 1) Can you prevent them from being indexed (I couldn't find anything about this, but I'd like official confirmation)? 2) What's the practical number of expando properties you shouldn't go above

[google-appengine] Re: How to write optimal queries?

2011-07-13 Thread Pol
It's been a few days since the last message on this thread - just making sure it doesn't fall through the cracks :) On Jul 8, 9:06 pm, Pol i...@pol-online.net wrote: Thanks for the slides, quite interesting - too bad there was no audio, it would have helped even more. So still, how are zig

[google-appengine] About expando properties

2011-07-12 Thread Pol
Hi, Regarding expando properties: 1) Can you prevent them from being indexed (I couldn't find anything about this, but I'd like official confirmation)? 2) What's the practical number of expando properties you shouldn't go above? If they are all indexed, then surely that can be quite expensive,

[google-appengine] Re: How to write optimal queries?

2011-07-08 Thread Pol
://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Thu, Jul 7, 2011 at 7:37 PM, Pol i...@pol-online.net wrote: I'm a little confused by your answer and the concept of zigzag joins. In practice, assuming the result set will end up being

[google-appengine] How to write optimal queries?

2011-07-07 Thread Pol
Hi, Assuming the app runs on an HR database and that the number of indexes is not a problem: Say that because of the data model, these queries are equivalent (i.e. return the exact same results), which one should be used to get the best performance? query = db.GqlQuery(SELECT * FROM Photo WHERE

[google-appengine] Re: How to write optimal queries?

2011-07-07 Thread Pol
obvservations been? Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Thu, Jul 7, 2011 at 6:21 PM, Pol i...@pol-online.net wrote: Hi, Assuming the app

[google-appengine] Re: Thread-safe Python Tips

2011-07-06 Thread Pol
On Jul 1, 10:10 am, Ikai Lan (Google) ika...@google.com wrote: It's possible for two operations to update townCache concurrently, but in your case it looks like it doesn't really matter. If TownModel is somehow updated between reads, it's theoretically possible for you to have an older

[google-appengine] Inconsistencies in GAE quota documentation?

2011-03-24 Thread Pol
Hi, On http://code.google.com/appengine/docs/quotas.html, in the Requests section, it says: The total number of requests to the app. The per-minute quotas for application with billing enabled allow for up to 500 requests per second—more than one billion requests per month. But further below, in