Re: how to speed up objects saving
Thanks everybody for your answers well, I've tried the solutions you've proposed and I've tested them on a part of my import procedure on my dev machine previously, this part was taking about 12 minute to complete -Justin & Rajesh my tables are MyIsam. I've read Django documentation on transaction commitment, even though this topic is not 100% clear for me, I've tried to do as you suggested, using commit_on_success and commit_manually (I don't know if it works on MyIsam tables) but there was either no improvement, either a loss of speed -Ivan I've tried commenting dispatcher.send() calls in both save and __init__ original methods. Well, I was expecting some kind of improvement, but somehow it takes almost exactly the same time to complete the job. I think that the problem stands on the 'database side' of Django - Matt I can tell that my CPU works quite a lot during the execution of these procedures, and mysql engine uses about 70%; maybe you have a lot of disk access? So far, as Ivan and Johan were suggesting, I've tried to transfer data from the original DB to my Django DB using raw SQL instructions and there was a considerable increase in speed. I've tested it only on some transactions but it saved about 40-60% of time, I'd say. Well, I didn't want to come to that ;) but obviously bypassing the Django machinery and letting MySQL do the job is the fastest solution. Thanks again, Alex -- View this message in context: http://www.nabble.com/how-to-speed-up-objects-saving-tp1555p15288461.html Sent from the django-users mailing list archive at Nabble.com. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: how to speed up objects saving
Alex, For this kind of "bulky transfer" the overhead created by the ORM will always be a performance bottleneck: you continuously create and destroy complex python objects. Where/if possible, resort to raw SQL: you could eg use an execute_many statement. Expect a performance improvement of x5 - x10... My 2 cents, Johan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: how to speed up objects saving
I believe I'm experiencing a similar problem. One thing that puzzles me is that the CPU usage remains very low while this slow (read: intensive) operation is carried out. On my development machine, neither the Python process or MySQL exceed 5% utilization. Does anyone have any idea why that may be? The situation is the same on both the dev server and through Apache. Alex - have you made any progress overriding the save method, and if so, what kind of speedup did you get? Thanks all, Matt. On Feb 1, 2:34 pm, Rajesh Dhawan <[EMAIL PROTECTED]> wrote: > In addition to Ivan's suggestions above, consider disabling auto- > commits and making manual transaction commits say every 100 records or > so. > > http://www.djangoproject.com/documentation/transactions/ > > This will eliminate the overhead of a commit on every instance save. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: how to speed up objects saving
In addition to Ivan's suggestions above, consider disabling auto- commits and making manual transaction commits say every 100 records or so. http://www.djangoproject.com/documentation/transactions/ This will eliminate the overhead of a commit on every instance save. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: how to speed up objects saving
> I'll try to redefine save methods in all the objects to see how much time it > saves You may want to redefine the __init__ methods as well (it's called when you create your objects before save), it adds overhead of two more signals that are probably unused and can be removed for speed. You can try comment out all dispatcher.send() lines of __init__ and __save__ methods of Model class in /django/db/models/base.py and save the file as base2.py in the same place inside your Django package and say something like `from django.db.models.base2 import Model as MyModel' and replace model.Model with MyModel in class definitions. I know it's hackish, but it's easier and it works. :) Hope this helps, Ivan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: how to speed up objects saving
Thanks a lot for the advice I'll try to redefine save methods in all the objects to see how much time it saves Alex Ivan Illarionov wrote: > > > You have two options: > 1. Execute raw SQL 'INSERT' queries > 2. Override the Model.save() or create new save_fast() method in your > Model class. The main speed eaters in Model.save() are > dispatcher.send() calls - so if you copy/paste the content of save > method from Django code without dispatcher.send() lines you'll get a > reasonable speed increase. > > --Ivan > > > > -- View this message in context: http://www.nabble.com/how-to-speed-up-objects-saving-tp1555p15227730.html Sent from the django-users mailing list archive at Nabble.com. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: how to speed up objects saving
You have two options: 1. Execute raw SQL 'INSERT' queries 2. Override the Model.save() or create new save_fast() method in your Model class. The main speed eaters in Model.save() are dispatcher.send() calls - so if you copy/paste the content of save method from Django code without dispatcher.send() lines you'll get a reasonable speed increase. --Ivan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
