Re: ANN: DSE v2.0.0-Beta - DSE uses 1.8% of the time compared to the Django ORM updating 100.000 records.
I will definitely be using this next time. Nice post. Brian On Tue, May 3, 2011 at 5:50 PM, Thomas Weholtwrote: > Hi, > > Just posted a new blog about more realistic performance of DSE against > a real database. Head over to http://weholt.blogspot.com/ to read it. > > Thanks for your feedback! > > Regards, > Thomas > > On Tue, May 3, 2011 at 1:55 PM, Brian Bouterse wrote: > > This would have been so useful on my last project. 14.4 Million records > > took a couple of days with a single thread using the ORM. This would > have > > really sped things up. > > Brian > > > > On Tue, May 3, 2011 at 7:50 AM, Thomas Weholt > > wrote: > >> > >> Inserting, updating or deleting lots of records using the Django ORM? > >> Want better performance? Check out DSE! > >> > >> It's free, released under the GPL licence. More information about the > >> possible performance gain can be seen here: > >> > >> > http://weholt.blogspot.com/2011/05/dse-uses-18-of-time-compared-to-django.html > >> > >> DSE at pypi: > >> http://pypi.python.org/pypi/dse/ > >> > >> Source at bitbucket.org: > >> https://bitbucket.org/weholt/dse2 > >> > >> Thanks for your attention. > >> > >> -- > >> Mvh/Best regards, > >> Thomas Weholt > >> http://www.weholt.org > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Django users" group. > >> To post to this group, send email to django-users@googlegroups.com. > >> To unsubscribe from this group, send email to > >> django-users+unsubscr...@googlegroups.com. > >> For more options, visit this group at > >> http://groups.google.com/group/django-users?hl=en. > >> > > > > > > > > -- > > Brian Bouterse > > ITng Services > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Django users" group. > > To post to this group, send email to django-users@googlegroups.com. > > To unsubscribe from this group, send email to > > django-users+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/django-users?hl=en. > > > > > > -- > Mvh/Best regards, > Thomas Weholt > http://www.weholt.org > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: ANN: DSE v2.0.0-Beta - DSE uses 1.8% of the time compared to the Django ORM updating 100.000 records.
Hi, Just posted a new blog about more realistic performance of DSE against a real database. Head over to http://weholt.blogspot.com/ to read it. Thanks for your feedback! Regards, Thomas On Tue, May 3, 2011 at 1:55 PM, Brian Boutersewrote: > This would have been so useful on my last project. 14.4 Million records > took a couple of days with a single thread using the ORM. This would have > really sped things up. > Brian > > On Tue, May 3, 2011 at 7:50 AM, Thomas Weholt > wrote: >> >> Inserting, updating or deleting lots of records using the Django ORM? >> Want better performance? Check out DSE! >> >> It's free, released under the GPL licence. More information about the >> possible performance gain can be seen here: >> >> http://weholt.blogspot.com/2011/05/dse-uses-18-of-time-compared-to-django.html >> >> DSE at pypi: >> http://pypi.python.org/pypi/dse/ >> >> Source at bitbucket.org: >> https://bitbucket.org/weholt/dse2 >> >> Thanks for your attention. >> >> -- >> Mvh/Best regards, >> Thomas Weholt >> http://www.weholt.org >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To post to this group, send email to django-users@googlegroups.com. >> To unsubscribe from this group, send email to >> django-users+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/django-users?hl=en. >> > > > > -- > Brian Bouterse > ITng Services > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > -- Mvh/Best regards, Thomas Weholt http://www.weholt.org -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: ANN: DSE v2.0.0-Beta - DSE uses 1.8% of the time compared to the Django ORM updating 100.000 records.
Thanks for DSE! I tried it earlier (version 0.6 I think). While I really liked the speed improvement, there were 2 areas that did not suit me too well: - error detection - I needed reliability more than speed and I was not comfortable with situations like "the IntegrityErrror is somewhere in last 1000 inserts" - multi-db support. I rely heavily on that and DSE was not mature enough there (I opened a ticket or two). I have not checked since if there's been any improvement. So, I'm not using DSE today but putting my 2 complaints aside, I really liked it when I tested it. Cheers Jirka -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Fwd: ANN: DSE v2.0.0-Beta - DSE uses 1.8% of the time compared to the Django ORM updating 100.000 records.
Amazingly, I'm actually due to run a clean up script on a database with 32 million rows in the next couple of days... Could you possibly answer: - How mature is the code? - Have you encountered any strange problems whilst using it? - How many other people have used this in production (that you know of) on a large data set? - Have you tested against SQL injection? - What performance benefits does this have, over using .raw() or .update() ?? Can you provide some benchmarks over 50 thousand rows? Cheers Cal -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: ANN: DSE v2.0.0-Beta - DSE uses 1.8% of the time compared to the Django ORM updating 100.000 records.
Hehe ... yes, that's exactly the kind of things DSE is good at. 1.8% of two days ... that's half an hour?? Mind you, I've so far not inserted that many records, so if you ever get the chance to try that out please let me know how it goes. Thomas On Tue, May 3, 2011 at 1:55 PM, Brian Boutersewrote: > This would have been so useful on my last project. 14.4 Million records > took a couple of days with a single thread using the ORM. This would have > really sped things up. > Brian > > On Tue, May 3, 2011 at 7:50 AM, Thomas Weholt > wrote: >> >> Inserting, updating or deleting lots of records using the Django ORM? >> Want better performance? Check out DSE! >> >> It's free, released under the GPL licence. More information about the >> possible performance gain can be seen here: >> >> http://weholt.blogspot.com/2011/05/dse-uses-18-of-time-compared-to-django.html >> >> DSE at pypi: >> http://pypi.python.org/pypi/dse/ >> >> Source at bitbucket.org: >> https://bitbucket.org/weholt/dse2 >> >> Thanks for your attention. >> >> -- >> Mvh/Best regards, >> Thomas Weholt >> http://www.weholt.org >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django users" group. >> To post to this group, send email to django-users@googlegroups.com. >> To unsubscribe from this group, send email to >> django-users+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/django-users?hl=en. >> > > > > -- > Brian Bouterse > ITng Services > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > -- Mvh/Best regards, Thomas Weholt http://www.weholt.org -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: ANN: DSE v2.0.0-Beta - DSE uses 1.8% of the time compared to the Django ORM updating 100.000 records.
This would have been so useful on my last project. 14.4 Million records took a couple of days with a single thread using the ORM. This would have really sped things up. Brian On Tue, May 3, 2011 at 7:50 AM, Thomas Weholtwrote: > Inserting, updating or deleting lots of records using the Django ORM? > Want better performance? Check out DSE! > > It's free, released under the GPL licence. More information about the > possible performance gain can be seen here: > > http://weholt.blogspot.com/2011/05/dse-uses-18-of-time-compared-to-django.html > > DSE at pypi: > http://pypi.python.org/pypi/dse/ > > Source at bitbucket.org: > https://bitbucket.org/weholt/dse2 > > Thanks for your attention. > > -- > Mvh/Best regards, > Thomas Weholt > http://www.weholt.org > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > > -- Brian Bouterse ITng Services -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
ANN: DSE v2.0.0-Beta - DSE uses 1.8% of the time compared to the Django ORM updating 100.000 records.
Inserting, updating or deleting lots of records using the Django ORM? Want better performance? Check out DSE! It's free, released under the GPL licence. More information about the possible performance gain can be seen here: http://weholt.blogspot.com/2011/05/dse-uses-18-of-time-compared-to-django.html DSE at pypi: http://pypi.python.org/pypi/dse/ Source at bitbucket.org: https://bitbucket.org/weholt/dse2 Thanks for your attention. -- Mvh/Best regards, Thomas Weholt http://www.weholt.org -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.