This is a pretty good response. There are a couple of points where a different point of view might be interesting, however. I've added some extra things, below.
On Thu, 2009-01-22 at 01:06 -0800, Webchemist wrote: > This is exactly my situation, when I started to work under drugme.ru - > this is rather big Russian social network website, > > Berco Beute: > > > - Stack. What would currently be a good combination webserver/protocol/ > > database? Possible options: Lighttpd, Cherokee, apache, FastCGI, > > mod_python, wsgi, mysql, progress... I realize there is no 'best' > > solution, but I would like to hear your recommendation. > > We started from Apache/mod_python, but this combination is not very > good for high load, thus we are using now Cherokee / FastCGI. It is > extremely fast and rather stable. > > > - Database optimization. The database layer is currently the major > > bottleneck. There is currently one database with over 150 tables, some > > of which are quite large. Some queries take close to minute to execute > > (yes, it's that bad). For redundancy reasons the database is > > replicated to another server (master/slave), but that doesn't help > > speeding up the response time. In the new version of the web > > application the database layer should be as scalable and maintainable > > as possible. This probably means using techniques such as 'sharding' > > and partitioning. The question is what both Python and Django offer in > > that respect. > This is not a domain for Django. First of all, try to migrate to > PostgreSQL. > > - Advantages of an ORM. > ORM has many advantages in speeding up development cycle, also has > some disadvantages: you loose in queries speed This isn't as true as people like to think. There's a slight extra bit of overhead due to the Python object -> SQL string conversion, but for any significant SQL query, the overhead is tiny as a proportion of total execution time. Most of the time spent retrieving results from the database will be spent inside and communicating with the database. > and ORM gives you a > possibility of memory leaks. That's isn't ever true. The ORM (nor pretty much any code written in pure Python) doesn't leak memory. It *uses* memory, for sure, sometimes quite a bit, as the standard Django queryset caches the results as they're loaded so that they can be reaccessed without a second database access (due to the point above). But it doesn't leak memory. When the queryset is garbage collected, at some point after it goes out of scope, all the memory is given back to the Python process. There is no difference between Django's behaviour and any other Python code that doesn't directly call C libraries. > First of all avoid to load big number of > objects in one query (for example, if you have for example 50000 > users, User.object.all() can kill you application under heavy web site > load). That's pretty standard, whether you're using an ORM or not. Don't load more data than you need. That helps in both the webserver machine and on the database side (less data to collect, less data to transmit across the network). There's another thing that we should document a bit better in this respect: if you don't need the queryset's caching behaviour and you're willing to be careful, you can iterate over the queryset by calling the iterator() method, rather than the (default) __iter__() method. The former is called, indirectly, by the latter, but it's only the latter that does the in-memory caching. 99% of cases won't care about this, but sometimes, if you cannot trim down a 50,000 result set, but know that you are only going to process results until you have "enough" and then throw everything away, you should use the iterator() method to get back the an iterator. [...] > > - Unit testing. Python supports many unit testing frameworks. Which > > would you recommend when rebuilding a large application from scratch? > > How would you set up your test environment? How to test all aspects of > > such a large system (preferably automatically)? > This is not a very powerful part of Django. Unit tests are quite well, > but functional testiong is not so good as would it be. This seems contrary to my experiences. (it may well be your experience, and I'm not criticising it for that reason). It seems to me that, similar to your answer about deployment, testing is something that is basically orthogonal to Django's functionality. Django is primarily a set of Python libraries, so you can use any testing support you like and there are a lot of them for Python. We provide some helpers, but they're not required, only useful (and recommended in the docs, to keep things simple). It happens that Django comes with a few utility classes and functions, and our documentation, in particular, talks about a subclass of Python's standard unittest.TestCase class. The Django subclass provides some useful default setup and teardown for initialising the database for tests. It's entirely optional whether people want to use those or not. It's a little disappointing that there aren't a lot more people blogging about using their test library of choice with Django. It's certainly possible and would perhaps fill a gap in the accumulated knowledge-base. Specifics of things that you (webchemist) find hard to emulate with your favourite framework would obviously be useful to know about, probably in a separate thread, but I think testing is an area where Django adopts a nice unobtrusive behaviour and doesn't require any particular framework for tests. Regards, Malcolm --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
