On Wed, Apr 7, 2010 at 8:11 AM, Alex Gaynor <alex.gay...@gmail.com> wrote: > Non-relational database support for the Django ORM > ================================================== > > Note: I am withdrawing my proposal on template compilation. Another student > has expressed some interest in working on it, and in any event I am now more > interested in working on this project. > > About Me > ~~~~~~~~ > > I'm a sophomore computer science student at Rensselaer Polytechnic Institute. > I'm a frequent contributor to Django (including last year's successful > multiple > database GSoC project) and other related projects; I'm also a committer on > both > `Unladen Swallow <http://code.google.com/p/unladen-swallow/>`_ and > `PyPy <http://codespeak.net/pypy/dist/pypy/doc/>`_. > > Background > ~~~~~~~~~~ > > As the person responsible for large swaths of multiple database support I am > intimately familiar with the architecture of the ORM, the code itself, and the > various concerns that need to be accounted for (pickleability, etc.). > > Rationale > ~~~~~~~~~ > > Non-relational databases tend to support some subset of the operations that > are > supported on relational databases, therefore it should be possible to perform > these operations on all databases. Some people are of the opinion that we > shouldn't bother to support these databases, because they can't perform all > operations, I'm of the opinion that the abstraction is already a little leaky, > we may as well exploit this for a common API where possible, as well as giving > users of these databases the admin and models forms for free. > > Method > ~~~~~~ > > The ORM architecture currently has a ``QuerySet`` which is backend agnostic, a > ``Query`` which is SQL specific, and a ``SQLCompiler`` which is backend > specific (i.e. Oracle vs. MySQL vs. generic). The plan is to change ``Query`` > to be backend agnostic by delaying the creation of structures that are SQL > specific, specifically join/alias data. Instead of structures like > ``self.where``, ``self.join_aliases``, or ``self.select`` all working in terms > of joins and table aliases the composition of a query would be stored in terms > of a tree containing the "raw" filters, as passed to the filter calls, with > things like ``Field.get_prep_value`` called appropriately. The > ``SQLCompiler`` > will be responsible for computing the joins for all of these data-structures.
I can see the intention here, and I can see how this approach could be used to solve the problem. However, my initial concern is that normal SQL users will end up carrying around a lot of extra overhead so that they can support backends that they will never use. Have you given any thought to how complex the datastructures inside Query will need to be, and how complex and/or expensive the conversion process will be? Other issues that spring to mind: * What about nonSQL datatypes? List/Set types are a common feature of Non-SQL backends, and are The Right Way to solve a whole bunch of problems. How do you propose to approach these datatypes? What (if any) overlap exists between the use of set data types and m2m? Is there any potential overlap between supporting List/Set types and supporting Arrays in SQL? * How does a non-SQL backend integrate with syncdb and other setup tools? What about inspectdb? * What about basic connection management? Is the existing Connection API likely to be compatible, or will modifications be required? * Why the choice of MongoDB specifically? Do you have particular experience with MongoDB? Does MongoDB have features that make it a good choice? * Given that you're only proposing a single proof-of-concept backend, have you given any thought to the needs of other backends? It's not hard to envisage that Couch, Cassandra, GAE etc will all have slightly different requirements and problems. Is there a common ground that exists between all data store backends? If there isn't, how do you know that what you are proposing will be sufficient to support them? There's also the issue of specificity in your proposal; I'll take you at your word that what you have proposed is a draft that requires elaboration. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
