Sorry about the URL confusion. The processing on web2py version is done on /queues/position, while in the django one is done in /api/position as I didnt implemented queues on the django version.
Json dumps and loads are only used on web2py version, but not using the shipped simplejson (it realy had a performace issue) its done using native json lib for python 2.7. I just posted the django performance for a matter of rough comparsion, as the code and the aproch for both are sightly diferent, so the performance should be too. The main comparsion is about web2py with or without the models. On Wed, Aug 8, 2012 at 11:59 AM, Massimo Di Pierro < massimo.dipie...@gmail.com> wrote: > I do not understand. When you compare Django vs web2p[y performance, you > seem to be comparing the MCycles for > /api/1.0/position<https://appengine.google.com/logs?version_id=production.359553040742132058&app_id=s%7Egps-holdsat-hrd&filter_type=labels&filter=path%3A%22%2Fapi%2F1.0%2Fposition%22&severity_level_override=1&view=Search> > (in > case of Django) with > /main/queues/position<https://appengine.google.com/logs?version_id=testing.360875204440667710&app_id=s%7Eapache-felipemeirelles&filter_type=labels&filter=path%3A%22%2Fmain%2Fqueues%2Fposition%22&severity_level_override=1&view=Search> > (in > the case of web2py). Shouldn't you be comparing with > /main/api/position<https://appengine.google.com/logs?version_id=testing.360875204440667710&app_id=s%7Eapache-felipemeirelles&filter_type=labels&filter=path%3A%22%2Fmain%2Fapi%2Fposition%22&severity_level_override=1&view=Search>? > Or perhaps the names are reversed? > > Do they perform the same DB IO? Do they both store sessions in DB (web2py > does)? > > It is likely Django uses simplejson binary while web2py (for portability) > only ships with simplejson in pure python. This may be another performance > loss if you use json a lot. We need to fix this at the web2py level. > > Massimo > > On Wednesday, 8 August 2012 08:06:05 UTC-5, Felipe Meirelles wrote: >> >> Well, after 24h running the new model less code I think I can post some >> data: >> >> First of all I'll explain my application archtecture: >> >> I have a cloud server gattering information from around 150 gps on some >> asserts. Each assert sends information every 30s, and some sends additional >> reading information, wich give me a total arround 150k request/day. This >> cloud server is integrated with appengine trought a REST API, speaching >> Json to each other. >> >> Before the model less refactor I had the following status: >> *Current Load [image: >> help]<https://developers.google.com/appengine/kb/general#currentload> >> * URI Req/Seccurrent Requestslast 6 hrs Runtime MCycleslast hr Avg >> Latency >> /main/queues/position<https://appengine.google.com/logs?version_id=testing.360875204440667710&app_id=s%7Eapache-felipemeirelles&filter_type=labels&filter=path%3A%22%2Fmain%2Fqueues%2Fposition%22&severity_level_override=1&view=Search> >> 1.2 48.78K 351 1200ms >> /main/api/position<https://appengine.google.com/logs?version_id=testing.360875204440667710&app_id=s%7Eapache-felipemeirelles&filter_type=labels&filter=path%3A%22%2Fmain%2Fapi%2Fposition%22&severity_level_override=1&view=Search> >> 0.9 44.37K 267 600 ms >> As you can see, I have controllers, the /api/ receives a call from the >> cloud server and enqueues a process request on appengine (this is a very >> simple controller, no data processing at all), and the /queues/ wich >> actualy process the data, make some inserts/updates and handle the memcache. >> >> After the refactor: >> >> *Current Load [image: >> help]<https://developers.google.com/appengine/kb/general#currentload> >> * URI Req/Seccurrent Requestslast 6 hrs Runtime MCycleslast hr Avg >> Latency >> /main/queues/position<https://appengine.google.com/logs?version_id=testing.360875204440667710&app_id=s%7Eapache-felipemeirelles&filter_type=labels&filter=path%3A%22%2Fmain%2Fqueues%2Fposition%22&severity_level_override=1&view=Search> >> 1.1 49.34K 180 404 ms >> /main/api/position<https://appengine.google.com/logs?version_id=testing.360875204440667710&app_id=s%7Eapache-felipemeirelles&filter_type=labels&filter=path%3A%22%2Fmain%2Fapi%2Fposition%22&severity_level_override=1&view=Search> >> 0.9 44.80K 78 139 ms >> Im only posting the position URL becouse its the most complex and most >> hitted. >> >> In comparsion, the Django project (same logic, I have it still runing >> while the web2py one is not ready): >> >> *Current Load [image: >> help]<https://developers.google.com/appengine/kb/general#currentload> >> * URI Req/Seccurrent Requestslast 6 hrs Runtime MCycleslast hr Avg >> Latencylast hr >> /api/1.0/position<https://appengine.google.com/logs?version_id=production.359553040742132058&app_id=s%7Egps-holdsat-hrd&filter_type=labels&filter=path%3A%22%2Fapi%2F1.0%2Fposition%22&severity_level_override=1&view=Search> >> 0.8 50.13K 92 406 ms >> On the django version I didnt had the queue yet, so this one URL proccess >> the request at all. >> >> Final toughts are, after a model less aproch, I reached half of the >> performance from django (still, some parts of the code are sightly >> diferent, and a lot more Json conversion and memcache is used in favour of >> reducing datastore access, wich can explain the loss on processing >> performance). Also, after the model less aproch, compared to web2py it >> self, I got a huge performance gain, and expect a solid cost reduction >> (I'll post the cost reduction latter, when appengine consolidates the >> billing). >> >> Thanks you all for sharing your toughts. >> >> On Tuesday, August 7, 2012 6:56:15 PM UTC-3, Massimo Di Pierro wrote: >>> >>> Please share you benchmarks when you think you are done. >>> >>> On Tuesday, 7 August 2012 14:29:51 UTC-5, Felipe Meirelles wrote: >>>> >>>> Well, I'm still in the middle of tests but all points to a huge drop on >>>> cpu usage using a model less aproch. I've made, in a few days ill post >>>> numbers, but the cpu_ms is arround half of it was. >>>> >>>> I've set all db.define_table() in separete models, and just include >>>> them where i use them. Also, all the configs from 0.py were split in files, >>>> and are just called when I need them. >>>> >>>> Thanks for the help! >>>> >>>> On Tue, Aug 7, 2012 at 4:25 PM, Anthony <abas...@gmail.com> wrote: >>>> >>>>> The app I'm developing is highly dependent on auth. >>>>>> >>>>>> Does it mean that any request that uses auth won't be faster than >>>>>> 200ms on GAE? >>>>>> >>>>> >>>>> Note, if you're just checking for login, I don't think you need to >>>>> call auth.define_tables() (that should only be needed for registration, >>>>> login, and checking permissions/membership). Also, make sure to use >>>>> auth.define_tables(migrate=**False) to avoid migrations on the auth >>>>> tables in production. >>>>> >>>>> Anthony >>>>> >>>>> -- >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Att, >>>> Felipe Meirelles. >>>> >>>> -- > > > > -- Att, Felipe Meirelles. --
