Re: Django ORM support for NoSql databases
On Tuesday, December 17, 2013 12:16:04 PM UTC-5, Aymeric Augustin wrote: > > Django’s ORM is entirely designed to translate between an object oriented > API and SQL. That’s what its name says. It achieves this through roughly > five layers that bride the abstraction gap. > > Django’s ORM is fundamentally tied to SQL. It isn’t a choice, a policy or > a wish. It’s a fact. > I am one of the committers on the django-nonrel project. We have supported backends for mongodb and google appengine. There are others out there, but they're not part of the nonrel project. So knowing what it takes to use the django ORM to work with a NoSQL DB, I hope I can provide some useful information. There are a few issues that we run into, so you can see that they are not fundamental problems. 1. Things you can do on a SQL system that you can't on NoSQL. - The big ones are joins and aggregates (for some kinds of aggregates on some NoSQL systems). My perspective is that this is the trade off you make when choosing your datastore, so you should know this going in. 2. Things you can do on NoSQL that you can't in SQL. - This is very database specific, so its hard to give examples, but its the same thing that necessitates raw queries. Some things you either can't do in the ORM or you can do better by hand. 3. Assumptions django makes about the underlying datastore. - We have submitted a number of patches to fix these issues, many of which have been merged in. Our fork is becoming less of a fork every day, and I think with django 1.7 (and some changes on our side), we may be able to do away with the fork for at least the app engine backend. Some specific issues: - AutoField is assumed to work only with integers. MongoDB is an outlier here, it uses something called an ObjectId which is an opaque string. This is actually a big one because field validation occurs before you know which database you save to. - Inserts vs. updates. Most NoSQL databases don't make the distinction, there is only a PUT. We have a patch that causes the ORM to basically force_insert on every save, but I think we can do away with this and convert the update to a PUT in our compiler. - django.contrib app. We've had issues where queries in the contrib apps use joins and other unsupported things. We have patches for these, and would like to get them merged but this is tricky because they are liable to break again if core developers aren't aware of the issues. So those are the technical issues, not so complicated IMHO. I can provide more details if anyone is interested. > I can't see a good reason to do that, but maybe others do. In the mean > time, this question will continue to be asked until a position is > officially documented, > > The core team doesn’t issue press releases with official positions :) > However, as far as I know, most core devs would subscribe to Tom’s answer. > Fair enough on not issuing policy proclimations :-) But I think it would be helpful to get some feedback on the question: should django try to 'open up' the ORM to make it easier to write backends for NoSQL DBs? Add more hooks, sharper divisions between layers, etc. --Alex -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/dabe9a2c-e211-4c16-9f1d-21994cead052%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Specifying the field index types
On Wednesday, October 2, 2013 1:29:10 PM UTC-4, Zev Benjamin wrote:
> What kind of additional configuration? One advantage of not specifying
> the type of the db_index_type field is that you could instantiate an object
> that encapsulates the configuration.
>
That's true. I am partial to opening up Meta for other reasons, but in this
case, if you were to have an arbitrary config object that is passed to the
db backend, then you could specify the App Engine indexes.
App Engine itself creates indexes on each field, by default all fields are
indexed, so this is something that you have to explicitly turn off. There
are also composite indexes, which index across many fields, these are
controlled via a separate App Engine-specific file (so its unrelated to
this discussion). But on top of all that there's an extension to the app
engine backend called django-dbindexer that allows you to add more indexes,
like for case-insensitive queries or contains queries. App Engine can't do
these things out of the box. Currently, you use another special file for
these indexes, but it would be nice to have this all configured on the
field.
--Alex
>
> On Wednesday, October 2, 2013 1:19:31 PM UTC-4, Alex Burgel wrote:
>>
>> This is something that I'd also be interested in. For the Google App
>> Engine backend, you have the ability to create indexes that require more
>> configuration than just on/off. But I don't think a single additional field
>> would do the trick for my case.
>>
>> Another option would be to open up the Meta class for custom attributes.
>> There's a thread which discusses this here:
>>
>> https://groups.google.com/forum/#!topic/django-developers/JEDJYB41LGI
>>
>> The config might look something like this:
>>
>> class Article(db.models):
>> class Meta:
>> custom_indexes = {'field1': 'hash', 'field2': 'btree'}
>> field1 = db.IntegerField()
>> field2 = db.CharField()
>>
>> The tradeoff would be that the index would not be specified in the field
>> description, but it would be close enough. (Currently for the app engine
>> backend, you specify these in a separate file that the backend looks for.)
>>
>> On Monday, September 23, 2013 10:36:56 AM UTC-4, Zev Benjamin wrote:
>>>
>>> Hi,
>>>
>>> I'd like to be able to specify what kind of index the database should
>>> use for a particular field. I have a proof of concept branch that works
>>> with PostgreSQL at
>>> https://github.com/zbenjamin/django/compare/index-types, but I'd like
>>> to solicit opinions on whether there's a better way of doing it. The way
>>> it works in my branch is that you specify the index type by making the
>>> Field db_index argument a string containing the name of the index type
>>> you'd like to use. Specifying db_index=True uses the default index type.
>>>
>>> Example usage:
>>>
>>> class IndexTest(models.Model):
>>> unindexed = models.IntegerField(db_index=False)
>>> default_indexed = models.IntegerField(db_index=True)
>>> btree_indexed = models.IntegerField(db_index="btree")
>>> hash_indexed = models.IntegerField(db_index="hash")
>>>
>>>
>>>
>>> Zev
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/6b0ba289-45c7-403f-b6bf-0e646fd139ce%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Specifying the field index types
This is something that I'd also be interested in. For the Google App Engine
backend, you have the ability to create indexes that require more
configuration than just on/off. But I don't think a single additional field
would do the trick for my case.
Another option would be to open up the Meta class for custom attributes.
There's a thread which discusses this here:
https://groups.google.com/forum/#!topic/django-developers/JEDJYB41LGI
The config might look something like this:
class Article(db.models):
class Meta:
custom_indexes = {'field1': 'hash', 'field2': 'btree'}
field1 = db.IntegerField()
field2 = db.CharField()
The tradeoff would be that the index would not be specified in the field
description, but it would be close enough. (Currently for the app engine
backend, you specify these in a separate file that the backend looks for.)
On Monday, September 23, 2013 10:36:56 AM UTC-4, Zev Benjamin wrote:
>
> Hi,
>
> I'd like to be able to specify what kind of index the database should use
> for a particular field. I have a proof of concept branch that works with
> PostgreSQL at https://github.com/zbenjamin/django/compare/index-types,
> but I'd like to solicit opinions on whether there's a better way of doing
> it. The way it works in my branch is that you specify the index type by
> making the Field db_index argument a string containing the name of the
> index type you'd like to use. Specifying db_index=True uses the default
> index type.
>
> Example usage:
>
> class IndexTest(models.Model):
> unindexed = models.IntegerField(db_index=False)
> default_indexed = models.IntegerField(db_index=True)
> btree_indexed = models.IntegerField(db_index="btree")
> hash_indexed = models.IntegerField(db_index="hash")
>
>
>
> Zev
>
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/ac38c2ab-566f-4212-8eba-244f3a074851%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Thinking about NoSQL support
On Sunday, September 8, 2013 2:20:56 AM UTC-4, Benjamin Zuill-Smith wrote: > The django-nonrel project is alive and looking for contributors. >> > > Hi Alex, thanks for responding. I guess I was concerned by the lack of > commits lately, but looks like it's still alive and kicking by your post. > I'm interested in helping in this endeavor. I've been pouring over the docs > the past couple of days in my free time. I'm looking for some docs that > help newly interested people like me get familiar with the structure of > django files such as some kind of overall design architecture naming key > classes involved in underlying ORM, etc. > > Are there any such docs available? > There are two places you can start from. The best option is to just get an app up and running. If you're interested in mongodb, then you can take a look at the mongodb-engine docs here: http://django-mongodb-engine.readthedocs.org/en/latest/ Alternatively, the original creators of the django-nonrel project wrote some pretty extensive blog posts while they were building it: http://www.allbuttonspressed.com/projects/django-nonrel Some of those docs are Google App Engine specific, but many of the concepts apply to mongodb as well. Also, we have a separate mailing list here: https://groups.google.com/forum/#!forum/django-non-relational Join that list (its low traffic) and we can discuss how you can help out. Thanks! --Alex -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Thinking about NoSQL support
On Thursday, September 5, 2013 4:12:26 PM UTC-4, Benjamin Zuill-Smith wrote: > I've recently discovered Django and would love this framework in tandem > with a NoSql solution like MongoDB. I know there is project called > mongodb-engine but the solutions it brings feel hackish to me because it is > not built into Django but added like middleware under the hood. I'm weary > of starting any real project using it. Are there any contributors out there > considering adding native NoSql support? Plus, many NoSql solutions provide > common functionality which could translate into a common interface which > Django could provide. I'd be interested to contribute to such an effort > and I think it is worth looking into given the growth in NoSql solutions > out there. > The django-nonrel project is alive and looking for contributors. http://www.django-nonrel.org/ It currently supports App Engine and MongoDB. It officially runs on django 1.3, 1.4, with support for 1.5 in beta. One of the goals of the project has been to become the official NoSQL foundation for django. Currently we maintain a fork of django in order to work around some SQL specific assumptions, but the changes are minimal and many patches to fix this have been committed already. Once all the patches are in, then django-nonrel is nothing more than another django.db backend. This makes it very useful for projects that mix SQL and NoSQL databases. Obviously there are features in django that you can't use if the database doesn't support them (JOINs, some aggregates, etc.). And there will also be features in your database that you cannot access through the django ORM. There's nothing to stop you from dropping into 'native' db code should you need it. This is similar in concept to raw sql support in django... sometimes you just to need to write the query by hand. As for the arguments that document/nosql/whatever db != relational db, its close enough to not matter in a lot of very useful scenarios. --Alex -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Django+GoogleAppengine(datastore)
On Thursday, June 6, 2013 2:53:17 AM UTC-4, Rishi Kumar wrote: > > so can somebody help me on this.i need the details of django setup with > full funtionality of datastore,as we have with webapp2 framework. > Ask on the django-nonrel list. We have a django backend which works with the app engine datastore. https://groups.google.com/forum/#!forum/django-non-relational -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
