Thanks for the clarification. On Wednesday, 8 August 2012 10:09:27 UTC-5, Felipe Meirelles wrote: > > Sorry about the URL confusion. The processing on web2py version is done on > /queues/position, while in the django one is done in /api/position as I > didnt implemented queues on the django version. > > Json dumps and loads are only used on web2py version, but not using the > shipped simplejson (it realy had a performace issue) its done using native > json lib for python 2.7. > > I just posted the django performance for a matter of rough comparsion, as > the code and the aproch for both are sightly diferent, so the performance > should be too. The main comparsion is about web2py with or without the > models. > > On Wed, Aug 8, 2012 at 11:59 AM, Massimo Di Pierro > <massimo....@gmail.com<javascript:> > > wrote: > >> I do not understand. When you compare Django vs web2p[y performance, you >> seem to be comparing the MCycles for >> /api/1.0/position<https://appengine.google.com/logs?version_id=production.359553040742132058&app_id=s%7Egps-holdsat-hrd&filter_type=labels&filter=path%3A%22%2Fapi%2F1.0%2Fposition%22&severity_level_override=1&view=Search> >> (in >> case of Django) with >> /main/queues/position<https://appengine.google.com/logs?version_id=testing.360875204440667710&app_id=s%7Eapache-felipemeirelles&filter_type=labels&filter=path%3A%22%2Fmain%2Fqueues%2Fposition%22&severity_level_override=1&view=Search> >> (in >> the case of web2py). Shouldn't you be comparing with >> /main/api/position<https://appengine.google.com/logs?version_id=testing.360875204440667710&app_id=s%7Eapache-felipemeirelles&filter_type=labels&filter=path%3A%22%2Fmain%2Fapi%2Fposition%22&severity_level_override=1&view=Search>? >> >> Or perhaps the names are reversed? >> >> Do they perform the same DB IO? Do they both store sessions in DB (web2py >> does)? >> >> It is likely Django uses simplejson binary while web2py (for portability) >> only ships with simplejson in pure python. This may be another performance >> loss if you use json a lot. We need to fix this at the web2py level. >> >> Massimo >> >> On Wednesday, 8 August 2012 08:06:05 UTC-5, Felipe Meirelles wrote: >>> >>> Well, after 24h running the new model less code I think I can post some >>> data: >>> >>> First of all I'll explain my application archtecture: >>> >>> I have a cloud server gattering information from around 150 gps on some >>> asserts. Each assert sends information every 30s, and some sends additional >>> reading information, wich give me a total arround 150k request/day. This >>> cloud server is integrated with appengine trought a REST API, speaching >>> Json to each other. >>> >>> Before the model less refactor I had the following status: >>> *Current Load [image: >>> help]<https://developers.google.com/appengine/kb/general#currentload> >>> * URI Req/Seccurrent Requestslast 6 hrs Runtime MCycleslast hr Avg >>> Latency >>> /main/queues/position<https://appengine.google.com/logs?version_id=testing.360875204440667710&app_id=s%7Eapache-felipemeirelles&filter_type=labels&filter=path%3A%22%2Fmain%2Fqueues%2Fposition%22&severity_level_override=1&view=Search> >>> 1.2 48.78K 351 1200ms >>> /main/api/position<https://appengine.google.com/logs?version_id=testing.360875204440667710&app_id=s%7Eapache-felipemeirelles&filter_type=labels&filter=path%3A%22%2Fmain%2Fapi%2Fposition%22&severity_level_override=1&view=Search> >>> 0.9 44.37K 267 600 ms >>> As you can see, I have controllers, the /api/ receives a call from the >>> cloud server and enqueues a process request on appengine (this is a very >>> simple controller, no data processing at all), and the /queues/ wich >>> actualy process the data, make some inserts/updates and handle the memcache. >>> >>> After the refactor: >>> >>> *Current Load [image: >>> help]<https://developers.google.com/appengine/kb/general#currentload> >>> * URI Req/Seccurrent Requestslast 6 hrs Runtime MCycleslast hr Avg >>> Latency >>> /main/queues/position<https://appengine.google.com/logs?version_id=testing.360875204440667710&app_id=s%7Eapache-felipemeirelles&filter_type=labels&filter=path%3A%22%2Fmain%2Fqueues%2Fposition%22&severity_level_override=1&view=Search> >>> 1.1 49.34K 180 404 ms >>> /main/api/position<https://appengine.google.com/logs?version_id=testing.360875204440667710&app_id=s%7Eapache-felipemeirelles&filter_type=labels&filter=path%3A%22%2Fmain%2Fapi%2Fposition%22&severity_level_override=1&view=Search> >>> 0.9 44.80K 78 139 ms >>> Im only posting the position URL becouse its the most complex and most >>> hitted. >>> >>> In comparsion, the Django project (same logic, I have it still runing >>> while the web2py one is not ready): >>> >>> *Current Load [image: >>> help]<https://developers.google.com/appengine/kb/general#currentload> >>> * URI Req/Seccurrent Requestslast 6 hrs Runtime MCycleslast hr Avg >>> Latencylast hr >>> /api/1.0/position<https://appengine.google.com/logs?version_id=production.359553040742132058&app_id=s%7Egps-holdsat-hrd&filter_type=labels&filter=path%3A%22%2Fapi%2F1.0%2Fposition%22&severity_level_override=1&view=Search> >>> 0.8 50.13K 92 406 ms >>> On the django version I didnt had the queue yet, so this one URL >>> proccess the request at all. >>> >>> Final toughts are, after a model less aproch, I reached half of the >>> performance from django (still, some parts of the code are sightly >>> diferent, and a lot more Json conversion and memcache is used in favour of >>> reducing datastore access, wich can explain the loss on processing >>> performance). Also, after the model less aproch, compared to web2py it >>> self, I got a huge performance gain, and expect a solid cost reduction >>> (I'll post the cost reduction latter, when appengine consolidates the >>> billing). >>> >>> Thanks you all for sharing your toughts. >>> >>> On Tuesday, August 7, 2012 6:56:15 PM UTC-3, Massimo Di Pierro wrote: >>>> >>>> Please share you benchmarks when you think you are done. >>>> >>>> On Tuesday, 7 August 2012 14:29:51 UTC-5, Felipe Meirelles wrote: >>>>> >>>>> Well, I'm still in the middle of tests but all points to a huge drop >>>>> on cpu usage using a model less aproch. I've made, in a few days ill post >>>>> numbers, but the cpu_ms is arround half of it was. >>>>> >>>>> I've set all db.define_table() in separete models, and just include >>>>> them where i use them. Also, all the configs from 0.py were split in >>>>> files, >>>>> and are just called when I need them. >>>>> >>>>> Thanks for the help! >>>>> >>>>> On Tue, Aug 7, 2012 at 4:25 PM, Anthony <abas...@gmail.com> wrote: >>>>> >>>>>> The app I'm developing is highly dependent on auth. >>>>>>> >>>>>>> Does it mean that any request that uses auth won't be faster than >>>>>>> 200ms on GAE? >>>>>>> >>>>>> >>>>>> Note, if you're just checking for login, I don't think you need to >>>>>> call auth.define_tables() (that should only be needed for registration, >>>>>> login, and checking permissions/membership). Also, make sure to use >>>>>> auth.define_tables(migrate=**False) to avoid migrations on the auth >>>>>> tables in production. >>>>>> >>>>>> Anthony >>>>>> >>>>>> -- >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Att, >>>>> Felipe Meirelles. >>>>> >>>>> -- >> >> >> >> > > > > -- > Att, > Felipe Meirelles. > >
--
