Re: four NoSQL backends you want? :)

[Daniel Greenfeld]

On Sep 28, 3:07 am, Thomas Wanschik  wrote:
> On 28 Sep., 02:45, Russell Keith-Magee 
> wrote:
> We've started a supported/unsupported feature list on 
> djangopackages:http://www.djangopackages.com/grids/g/cloud/
>
> So please help in order to get closer at least one step towards NoSQL
> in Django.

NoSQL != cloud, so I changed the name, slug, and description for
http://www.djangopackages.com/grids/g/cloud/

It is now: http://www.djangopackages.com/grids/g/nosql/

Perhaps it should even be nosql-backends?

Daniel Greenfeld

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: four NoSQL backends you want? :)

[Thomas Wanschik]

On 28 Sep., 02:45, Russell Keith-Magee 
wrote:
> So - the real goal is to ensure that you can use forms and generic
> views with NoSQL stores, not to ensure that you take an app built
> using a relational store, and deploy it on a NoSQL store.

If that's the real goal then we've reached it already more than half a
year ago with the backend for App Engine and a couple of months ago
with a backend for MongoDB.
To end this endless discussions and to proof what we claim, i call for
help. Please test generic views, forms, the admin,  in order to
build trust. Let us know how to improve the documentation and what's
missing.

The current documentation for the App Engine can be found here:
http://www.allbuttonspressed.com/projects/djangoappengine

The documentation for the MongoDB backend can be found here:
http://github.com/aparo/django-mongodb-engine

We've started a supported/unsupported feature list on djangopackages:
http://www.djangopackages.com/grids/g/cloud/

So please help in order to get closer at least one step towards NoSQL
in Django.

Bye,
Thomas Wanschik

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: four NoSQL backends you want? :)

[Russell Keith-Magee]

On Tue, Sep 28, 2010 at 8:31 AM, David P. Novakovic
 wrote:
> This has probably been discussed at great length previously... but my
> 2c follows:
>
> If you are using a column/doc store you are trying to solve a
> different problem than if you are using an SQL db.
>
> How important is 100% interop? Surely it's about documenting the
> differences between them and providing an interface to document stores
> that isn't completely foreign to people who know how to use the django
> ORM?

Sounds like you need to watch the couple of panel discussions at
DjangoCon.us and DjangoCon.eu where we discuss this exact point. :-)

100% interop with a relational store is a pipe dream. As you say,
they're different solutions for different problems. NoSQL support in
Django isn't about trying to make everything look like a hammer. Its
about ensuring that all the tools we have in the shed are compatible.

So - the real goal is to ensure that you can use forms and generic
views with NoSQL stores, not to ensure that you take an app built
using a relational store, and deploy it on a NoSQL store. Using a
common query interface is the easiest way to make this happen; and as
a bonus, it means that NoSQL stores gain a query interface that is
familiar.

> In which case it seems that the trust issue Russ was talking about
> earlier is the main barrier here. We need more trusted people to be
> using the non-rel code.

To be sure - this is a big barrier. The more eyeballs this code gets,
the better the end product will be.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: four NoSQL backends you want? :)

[Mark Bucciarelli]

On Tue, Sep 28, 2010 at 07:22:11AM +0800, Russell Keith-Magee wrote:
> On Tue, Sep 28, 2010 at 1:51 AM, Mark Bucciarelli  wrote:
> >
> > I guess the test suite must not cover all the basees, making it a
> > necessary but not sufficient criteria for inclusion?
> >
> 
> The problem is that NoSQL stores don't necessarily have feature parity
> with relational databases. 
>

For MonetDB, this is not an issue as they provide a SQL wrapper.

For the other No-SQL backends, is it feasible to specify the subset of
tests they must pass to be acceptable?

And if you define that unsupported operations should throw a particular
kind exception, you wouldn't need annotation sugar; the test logic can
branch based on what features the DB says it supports.

m

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: four NoSQL backends you want? :)

[Alex Gaynor]

On Mon, Sep 27, 2010 at 7:22 PM, Russell Keith-Magee
 wrote:
> On Tue, Sep 28, 2010 at 1:51 AM, Mark Bucciarelli  wrote:
>> On Sat, Sep 25, 2010 at 10:21:06PM +0800, Russell Keith-Magee wrote:
>>>
>>> The number 4 wasn't actually the important bit - it was that I want
>>> to see a range of noSQL approaches represented. I don't want to see
>>> 4 key-value stores; I want to see a key-value store, and a document
>>> store, and a column store, etc. I need to see that the API is rich
>>> enough to cover all noSQL bases.
>>>
>>
>> I've seen references to Django's test suite on this list in the past,
>> and I'm curious why the criteria can't simple be: "a clean run on our
>> test suite."
>>
>> I have a backend [1] for MonetDB (a column store DB).  They expose a SQL
>> wrapper around their non-SQL engine, which makes it pretty easy to
>> support.
>>
>> On my todo list was to see how it fares with the Django test-suite.
>> I guess the test suite must not cover all the basees, making it a
>> necessary but not sufficient criteria for inclusion?
>>
>> Correct?
>
> Well, sort of.
>
> The problem is that NoSQL stores don't necessarily have feature parity
> with relational databases. For example, the concept of a join is
> pretty fuzzy when it comes to stores like MongoDB and GAE. You *could*
> emulate join with a series of lookups, but that could get pretty
> spectacularly inefficient, so we're not going to force stores to
> emulate behaviors that aren't extensions of their native capabilities.
> Given that there isn't feature parity, Django's test suite won't be
> able to pass completely.
>
> We're in the process of migrating Django's test suite to use
> unittest2, which, in turn, means we have access to the @skipif and
> @skipunless decorators; we're going to be annotating a bunch of tests
> to make them 'feature aware' (e.g., don't run this test if the backend
> doesn't support transactions). A logical extension of this is to add
> other feature checks like "backend supports joins", and annotate the
> test suite appropriately. This would allow the noSQL backend to pass
> the suite, albeit with a lot of test skips.
>
> My (mild) concern with this approach is that a feature that this will
> make it too easy to get a suite 'passing', and we'll miss some detail
> in the process. For example, if we make it easy to skip the aggregates
> tests, and a backend actually does support aggregates, then it becomes
> easy to say "Oh, we *could* do aggregates, but we just haven't yet".
> Now, this may be entirely true, but it could also be the case that
> Django's backend API is structured in such as way that aggregates are
> impossible to implement.  Making it easy to skip tests masks the
> complications that may exist in the API.
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>

This doesn't seem like much of a regression to me.  While developing
my backend I basically ignored the existing tests because it was
impossible to learn anything from 700 test failures, of which some are
"expected and never will pass" (e.g. ones testing that we generate
only one JOIN), other are "this is something I just haven't
implemented yet, and really don't need to be told".  In fact, I view
being able to gradually increase the portion of the API you claim to
implement, as shown in your testing, as being a tremendous boon.

Alex


-- 
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: four NoSQL backends you want? :)

[David P. Novakovic]

This has probably been discussed at great length previously... but my
2c follows:

If you are using a column/doc store you are trying to solve a
different problem than if you are using an SQL db.

How important is 100% interop? Surely it's about documenting the
differences between them and providing an interface to document stores
that isn't completely foreign to people who know how to use the django
ORM?

In which case it seems that the trust issue Russ was talking about
earlier is the main barrier here. We need more trusted people to be
using the non-rel code.

Surely that doesn't mean the project should end. Maybe there is an
interim measure that can be taken, like the django project can
recommend that people use non-rel when they want column store support
in their app with the disclaimer that it is not officially supported.
>From there it could become the defacto standard, and following that it
may be accepted?

Just throwing alternative ideas out here.

D

On Tue, Sep 28, 2010 at 9:22 AM, Russell Keith-Magee
 wrote:
> On Tue, Sep 28, 2010 at 1:51 AM, Mark Bucciarelli  wrote:
>> On Sat, Sep 25, 2010 at 10:21:06PM +0800, Russell Keith-Magee wrote:
>>>
>>> The number 4 wasn't actually the important bit - it was that I want
>>> to see a range of noSQL approaches represented. I don't want to see
>>> 4 key-value stores; I want to see a key-value store, and a document
>>> store, and a column store, etc. I need to see that the API is rich
>>> enough to cover all noSQL bases.
>>>
>>
>> I've seen references to Django's test suite on this list in the past,
>> and I'm curious why the criteria can't simple be: "a clean run on our
>> test suite."
>>
>> I have a backend [1] for MonetDB (a column store DB).  They expose a SQL
>> wrapper around their non-SQL engine, which makes it pretty easy to
>> support.
>>
>> On my todo list was to see how it fares with the Django test-suite.
>> I guess the test suite must not cover all the basees, making it a
>> necessary but not sufficient criteria for inclusion?
>>
>> Correct?
>
> Well, sort of.
>
> The problem is that NoSQL stores don't necessarily have feature parity
> with relational databases. For example, the concept of a join is
> pretty fuzzy when it comes to stores like MongoDB and GAE. You *could*
> emulate join with a series of lookups, but that could get pretty
> spectacularly inefficient, so we're not going to force stores to
> emulate behaviors that aren't extensions of their native capabilities.
> Given that there isn't feature parity, Django's test suite won't be
> able to pass completely.
>
> We're in the process of migrating Django's test suite to use
> unittest2, which, in turn, means we have access to the @skipif and
> @skipunless decorators; we're going to be annotating a bunch of tests
> to make them 'feature aware' (e.g., don't run this test if the backend
> doesn't support transactions). A logical extension of this is to add
> other feature checks like "backend supports joins", and annotate the
> test suite appropriately. This would allow the noSQL backend to pass
> the suite, albeit with a lot of test skips.
>
> My (mild) concern with this approach is that a feature that this will
> make it too easy to get a suite 'passing', and we'll miss some detail
> in the process. For example, if we make it easy to skip the aggregates
> tests, and a backend actually does support aggregates, then it becomes
> easy to say "Oh, we *could* do aggregates, but we just haven't yet".
> Now, this may be entirely true, but it could also be the case that
> Django's backend API is structured in such as way that aggregates are
> impossible to implement.  Making it easy to skip tests masks the
> complications that may exist in the API.
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: four NoSQL backends you want? :)

[Russell Keith-Magee]

On Tue, Sep 28, 2010 at 1:51 AM, Mark Bucciarelli  wrote:
> On Sat, Sep 25, 2010 at 10:21:06PM +0800, Russell Keith-Magee wrote:
>>
>> The number 4 wasn't actually the important bit - it was that I want
>> to see a range of noSQL approaches represented. I don't want to see
>> 4 key-value stores; I want to see a key-value store, and a document
>> store, and a column store, etc. I need to see that the API is rich
>> enough to cover all noSQL bases.
>>
>
> I've seen references to Django's test suite on this list in the past,
> and I'm curious why the criteria can't simple be: "a clean run on our
> test suite."
>
> I have a backend [1] for MonetDB (a column store DB).  They expose a SQL
> wrapper around their non-SQL engine, which makes it pretty easy to
> support.
>
> On my todo list was to see how it fares with the Django test-suite.
> I guess the test suite must not cover all the basees, making it a
> necessary but not sufficient criteria for inclusion?
>
> Correct?

Well, sort of.

The problem is that NoSQL stores don't necessarily have feature parity
with relational databases. For example, the concept of a join is
pretty fuzzy when it comes to stores like MongoDB and GAE. You *could*
emulate join with a series of lookups, but that could get pretty
spectacularly inefficient, so we're not going to force stores to
emulate behaviors that aren't extensions of their native capabilities.
Given that there isn't feature parity, Django's test suite won't be
able to pass completely.

We're in the process of migrating Django's test suite to use
unittest2, which, in turn, means we have access to the @skipif and
@skipunless decorators; we're going to be annotating a bunch of tests
to make them 'feature aware' (e.g., don't run this test if the backend
doesn't support transactions). A logical extension of this is to add
other feature checks like "backend supports joins", and annotate the
test suite appropriately. This would allow the noSQL backend to pass
the suite, albeit with a lot of test skips.

My (mild) concern with this approach is that a feature that this will
make it too easy to get a suite 'passing', and we'll miss some detail
in the process. For example, if we make it easy to skip the aggregates
tests, and a backend actually does support aggregates, then it becomes
easy to say "Oh, we *could* do aggregates, but we just haven't yet".
Now, this may be entirely true, but it could also be the case that
Django's backend API is structured in such as way that aggregates are
impossible to implement.  Making it easy to skip tests masks the
complications that may exist in the API.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: four NoSQL backends you want? :)

[Mark Bucciarelli]

On Sat, Sep 25, 2010 at 10:21:06PM +0800, Russell Keith-Magee wrote:
>
> The number 4 wasn't actually the important bit - it was that I want
> to see a range of noSQL approaches represented. I don't want to see
> 4 key-value stores; I want to see a key-value store, and a document
> store, and a column store, etc. I need to see that the API is rich
> enough to cover all noSQL bases.
>

I've seen references to Django's test suite on this list in the past,
and I'm curious why the criteria can't simple be: "a clean run on our
test suite."

I have a backend [1] for MonetDB (a column store DB).  They expose a SQL
wrapper around their non-SQL engine, which makes it pretty easy to
support.

On my todo list was to see how it fares with the Django test-suite.
I guess the test suite must not cover all the basees, making it a
necessary but not sufficient criteria for inclusion?

Correct?

Thanks,

m

[1] http://github.com/mbucc/monetdb-python

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: four NoSQL backends you want? :)

[Russell Keith-Magee]

On Sun, Sep 26, 2010 at 6:40 PM, Waldemar Kornewald
 wrote:
> On Sun, Sep 26, 2010 at 6:54 AM, Russell Keith-Magee
>  wrote:
>> Also, you say a "huge" amount of analysis -- at the moment, *any*
>> analysis would be a step in the right direction. I haven't seen *any*
>> summary of the features that are and are not supported by various
>> backends. The summary pages that I directed your attention to (for
>> CSRF and session messages) were not hundred page tomes. They were
>> concise, 2-3 page summaries. A similar effort, signed off by someone
>> (or someones)  I trust is all I'm asking for.
>
> I think I misread that part. Here's a little listing of what's
> supported by the djangoappengine backend:
> http://www.allbuttonspressed.com/projects/djangoappengine
> I'm not sure if that's what you want to see or if it's too concise.

The "Supported and Unsupported features" section is pretty close to
what I would like to see.

If each of the candidate storage engines had a similar summary from
the perspective of a query-refactor compatible backend, we'd be well
on the way.

If you can get review from others in the community, that would be even better.

>> So far, your contributions to Django consist of two things:
>>  * Maintaining the non-rel fork
>>  * A large proposal to change the way file uploads work.
>>
>> The file uploads patch has been through several drafts, consumed a lot
>> of my time, and still isn't in a trunk ready state. This doesn't lead
>> me to a position of trusting that your other work is trunk ready; and
>> the discussions that we've had (both on file uploads, and on earlier
>> iterations of noSQL) haven't led me to a position where I feel I can
>> trust that your design decisions are compatible with the general
>> philosophy of Django's core.
>
> There seems to be a communication problem (is it a mistake on my
> side?). You seem to expect a cleaned up trunk-ready patch for the file
> uploads feature. In my last email I thought it was clear that I
> expected to get feedback on the high-level end-user API before
> starting to clean up everything, improving the function/class naming,
> etc. I have more fundamental questions like: Should the "tag" be based
> on the path to the Form/Model/ModelAdmin class? BTW, on Thursday
> Jannis Leidel has started commenting on the proposal, too (thanks a
> lot!):
> http://code.djangoproject.com/ticket/13960

I don't know if there's a communication problem. I've told you my
position. At the moment, when I see your file uploads proposal, I see
a time sink. There is still a lot of design work required. We've
already been through a couple of iterations, and I don't feel like
we're really getting any closer to a final solution.

My Django time is all volunteered. I need to use it maximize utility
of that time. Time that I spend helping you finesse a design is time
that I don't spend fixing bugs, committing features, or working on
designs from others. I have made a judgement call that continuing to
work on your file uploads proposal isn't an effective use of my time
at present considering the time it will take to get the design
finished, and the potential benefits of the change when it is
introduced.

In principle, I like the idea of having better support for external
file stores. But I'm not the only core team member (or, for that
matter, member of the Django community) that can provide feedback. If
Jannis can help you get your design closer to being trunk-ready, then
that's great.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: four NoSQL backends you want? :)

[Waldemar Kornewald]

On Sun, Sep 26, 2010 at 6:54 AM, Russell Keith-Magee
 wrote:
> On Sun, Sep 26, 2010 at 1:24 AM, Waldemar Kornewald
>  wrote:
>> On Sep 25, 4:21 pm, Russell Keith-Magee 
>> wrote:
>>> My reason for wanting this is that I'm simply not an expert in any of
>>> these backends. I know SQL quite well, but I haven't had occasion to
>>> try out other backends in depth. I can judge the technical merits of a
>>> patch based on what I know, but I don't want to make a judgement based
>>> on incompletely knowledge. I need to rely on those that I know and
>>> trust to give me confidence that nothing has been missed.
>>
>> This is really the biggest problem. If you work in a company where
>> your lead developer or manager doesn't know enough about the
>> technology and for that reason wants to have a huge amount of analysis
>> and review how are you going to get anything done? You'd surely call
>> such a company dysfunctional.
>
> OK - I call bullshit.
>
> What you appear to be saying is that when a person in a management
> position knows that they don't have expertise in a particular area,
> the appropriate response is to find the first person that volunteers a
> solution, and accept their answer as gospel truth?
>
> I am well skilled in SQL. I am probably one of a handful of people who
> can call themselves an expert on the inner workings of Django's ORM. I
> am an expert on the mental model that Django's ORM is trying to push,
> and the sorts of architectural decisions that will and won't be
> acceptable to the core.
>
> I am not well skilled in NoSQL. I have tinkered enough to understand
> the broad issues, but I don't have particular expertise in any of the
> backends. I'm interested in adding NoSQL support to the core of
> Django. Therefore, I'm calling for people who *do* have expertise, and
> whose expertise I know I can trust, to provide direction that nothing
> important has been missed in the analysis.
>
> Calling for expertise when you know you're lacking isn't dysfunctional
> management. It's responsible management.

Agreed. However, in its *current* state, the only guaranteed
contributors you have are people who aren't trusted and the only
contributor who has decision power is you. This constellation is
dysfunctional and that's what I'm talking about.

Your calling for expertise is the right thing to do. And my whole post
said this: If your call for expertise is not successful the team stays
dysfunctional (unless you acquire the required expertise yourself) and
this means any further work on NoSQL support has a high chance of
being wasted time.

Note, I still want to get the file uploads patch into Django. I don't
intend to completely stop contributing. It's just the current
situation around the NoSQL project that I don't find very attractive.

> Also, you say a "huge" amount of analysis -- at the moment, *any*
> analysis would be a step in the right direction. I haven't seen *any*
> summary of the features that are and are not supported by various
> backends. The summary pages that I directed your attention to (for
> CSRF and session messages) were not hundred page tomes. They were
> concise, 2-3 page summaries. A similar effort, signed off by someone
> (or someones)  I trust is all I'm asking for.

I think I misread that part. Here's a little listing of what's
supported by the djangoappengine backend:
http://www.allbuttonspressed.com/projects/djangoappengine
I'm not sure if that's what you want to see or if it's too concise.

The MongoDB backend has basically the same constraints except that it
natively also supports __iexact, __istartswith, __endswith, and
__iendswith (though, not very efficiently). It also adds a field for
embedded documents.

> So far, your contributions to Django consist of two things:
>  * Maintaining the non-rel fork
>  * A large proposal to change the way file uploads work.
>
> The file uploads patch has been through several drafts, consumed a lot
> of my time, and still isn't in a trunk ready state. This doesn't lead
> me to a position of trusting that your other work is trunk ready; and
> the discussions that we've had (both on file uploads, and on earlier
> iterations of noSQL) haven't led me to a position where I feel I can
> trust that your design decisions are compatible with the general
> philosophy of Django's core.

There seems to be a communication problem (is it a mistake on my
side?). You seem to expect a cleaned up trunk-ready patch for the file
uploads feature. In my last email I thought it was clear that I
expected to get feedback on the high-level end-user API before
starting to clean up everything, improving the function/class naming,
etc. I have more fundamental questions like: Should the "tag" be based
on the path to the Form/Model/ModelAdmin class? BTW, on Thursday
Jannis Leidel has started commenting on the proposal, too (thanks a
lot!):
http://code.djangoproject.com/ticket/13960

Bye,

Re: four NoSQL backends you want? :)

[Russell Keith-Magee]

On Sun, Sep 26, 2010 at 1:24 AM, Waldemar Kornewald
 wrote:
> On Sep 25, 4:21 pm, Russell Keith-Magee 
> wrote:
>> My reason for wanting this is that I'm simply not an expert in any of
>> these backends. I know SQL quite well, but I haven't had occasion to
>> try out other backends in depth. I can judge the technical merits of a
>> patch based on what I know, but I don't want to make a judgement based
>> on incompletely knowledge. I need to rely on those that I know and
>> trust to give me confidence that nothing has been missed.
>
> This is really the biggest problem. If you work in a company where
> your lead developer or manager doesn't know enough about the
> technology and for that reason wants to have a huge amount of analysis
> and review how are you going to get anything done? You'd surely call
> such a company dysfunctional.

OK - I call bullshit.

What you appear to be saying is that when a person in a management
position knows that they don't have expertise in a particular area,
the appropriate response is to find the first person that volunteers a
solution, and accept their answer as gospel truth?

I am well skilled in SQL. I am probably one of a handful of people who
can call themselves an expert on the inner workings of Django's ORM. I
am an expert on the mental model that Django's ORM is trying to push,
and the sorts of architectural decisions that will and won't be
acceptable to the core.

I am not well skilled in NoSQL. I have tinkered enough to understand
the broad issues, but I don't have particular expertise in any of the
backends. I'm interested in adding NoSQL support to the core of
Django. Therefore, I'm calling for people who *do* have expertise, and
whose expertise I know I can trust, to provide direction that nothing
important has been missed in the analysis.

Calling for expertise when you know you're lacking isn't dysfunctional
management. It's responsible management.

Also, you say a "huge" amount of analysis -- at the moment, *any*
analysis would be a step in the right direction. I haven't seen *any*
summary of the features that are and are not supported by various
backends. The summary pages that I directed your attention to (for
CSRF and session messages) were not hundred page tomes. They were
concise, 2-3 page summaries. A similar effort, signed off by someone
(or someones)  I trust is all I'm asking for.

> You yourself always say how limited your
> time is and that you're working on this in your spare time. Well,
> we're in the same situation. I don't know if I can and want to invest
> a lot of time, so I can give you a detailed analysis of everything,
> especially if you only trust people you know personally, anyway.

Lets be clear here. I *didn't* say "someone I know personally". I said
"someone known to the core" -- by which I mean someone who has a long
term reputation of contributing, or of being involved in a project
that
 makes their expertise undeniable.

Example 1: I've worked very closely with Alex Gaynor, on patches large
and small. Over many years, I've come to trust his judgement. If Alex
says a patch is good, experience tells me that he's probably right.

Example 2: The most recent Django NoSQL panel contained
representatives from Mahalo, Urban Airship, Mochi Media, and Engine
Yard. These are serious companies, and you don't get a work for
serious companies unless you have serious skills. If someone from one
of these companies vouches for a patch, I have a reason to believe
that it will be good.

Of the list of names you've given of people who have contributed NoSQL
patches -- including yourself -- I recognize none of them. I don't
know where they work. They don't have a history of contribution to
Django that enables me to have confidence in their design taste and
coding skills.

That doesn't mean that their code is inherently bad -- it just means I
have no prima facie reason to trust that they know what they're
talking about.

Saying "John Smith is betting his company on Cassandra" doesn't tell
me anything. John Smith could be a particularly bad gambler, or he
could be gambling his company that currently serves 2 customers
generating 5 page requests a day.  Unless I know that John Smith has a
history of making well informed design choices, or I know that his
company is already successful, a recommendation from John Smith isn't
worth a whole lot. It's worth *something*, but by itself, it's not
enough for me to risk the reputation of the entire Django project.

> I understand your side: You don't feel qualified to
> make a decision. That's fine. But from my side it looks like I'm in
> fact wasting my time because I don't get trusted, anyway.

You appear to have some fundamental misunderstandings about how trust works.

Trust comes when you do something over and over again in a way that
enables someone else to have confidence that you're going to do it the
same way next time the same problem arises.

Trust comes 

Re: four NoSQL backends you want? :)

[Florent Gallaire]

When a free software project fail to progress because of a bad
leadership, you can :

1) make a putsh.
2) make a fork.

Florent

On Sat, Sep 25, 2010 at 7:24 PM, Waldemar Kornewald
 wrote:
> On Sep 25, 4:21 pm, Russell Keith-Magee 
> wrote:
>> My reason for wanting this is that I'm simply not an expert in any of
>> these backends. I know SQL quite well, but I haven't had occasion to
>> try out other backends in depth. I can judge the technical merits of a
>> patch based on what I know, but I don't want to make a judgement based
>> on incompletely knowledge. I need to rely on those that I know and
>> trust to give me confidence that nothing has been missed.
>
> This is really the biggest problem. If you work in a company where
> your lead developer or manager doesn't know enough about the
> technology and for that reason wants to have a huge amount of analysis
> and review how are you going to get anything done? You'd surely call
> such a company dysfunctional. You yourself always say how limited your
> time is and that you're working on this in your spare time. Well,
> we're in the same situation. I don't know if I can and want to invest
> a lot of time, so I can give you a detailed analysis of everything,
> especially if you only trust people you know personally, anyway. I
> mean, try convincing someone who doesn't know SQL that Django's ORM
> supports JOINs and aggregates and UPDATE operations. It's practically
> impossible. That's exactly how I feel. We've got a lot of users with
> experience in their particular NoSQL DBs and some of them are even
> betting their business on those backends and use them for commercial
> projects without any problems and even that doesn't seem to be
> convincing enough. I understand your side: You don't feel qualified to
> make a decision. That's fine. But from my side it looks like I'm in
> fact wasting my time because I don't get trusted, anyway. So, is
> anyone in the Django core team experienced enough with NoSQL DBs to
> make a qualified analysis and is that person serious about helping to
> get NoSQL support into Django 1.4? Without such a person helping us it
> doesn't make sense to continue this project.
>
> If you want to know whether users face surprises with Django-nonrel
> please read this post from yesterday, for example:
> http://groups.google.com/group/google-appengine-python/msg/2ee52c2aa8ab720b
>
>> Lastly, we need to resolve the AutoField problem. This is the biggest
>> outstanding technical issue. I can't say I've got any particularly
>> brilliant ideas on how to solve it. Suggestions are welcome.
>
> Wasn't your original suggestion good enough? We could detect whether
> the user only has SQL backends and activate the old validation
> behavior, but show a deprecation warning whenever things are expected
> to change. If the user has a NoSQL backend the new validation behavior
> is automatically activated. A few releases later we can get rid of the
> old behavior.
>
>> As for timing: I'm on record saying a number of times now that this
>> isn't something I'm targeting for 1.3. We need to have a release that
>> is low on features and high on bugfixes. Given the time required to
>> make large scale database backend changes, I simply don't have enough
>> bandwidth to satisfy the community needs for 1.3 *and* the needs of
>> query-refactor. However, if people like yourself that are motivated in
>> this are can get the pieces in place during the 1.3 development cycle,
>> query-refactor could be an early delivery in the 1.4 timeframe (i.e.,
>> early next year).
>
> That's fine.
>
> Bye,
> Waldemar
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: four NoSQL backends you want? :)

[Waldemar Kornewald]

On Sep 25, 4:21 pm, Russell Keith-Magee 
wrote:
> My reason for wanting this is that I'm simply not an expert in any of
> these backends. I know SQL quite well, but I haven't had occasion to
> try out other backends in depth. I can judge the technical merits of a
> patch based on what I know, but I don't want to make a judgement based
> on incompletely knowledge. I need to rely on those that I know and
> trust to give me confidence that nothing has been missed.

This is really the biggest problem. If you work in a company where
your lead developer or manager doesn't know enough about the
technology and for that reason wants to have a huge amount of analysis
and review how are you going to get anything done? You'd surely call
such a company dysfunctional. You yourself always say how limited your
time is and that you're working on this in your spare time. Well,
we're in the same situation. I don't know if I can and want to invest
a lot of time, so I can give you a detailed analysis of everything,
especially if you only trust people you know personally, anyway. I
mean, try convincing someone who doesn't know SQL that Django's ORM
supports JOINs and aggregates and UPDATE operations. It's practically
impossible. That's exactly how I feel. We've got a lot of users with
experience in their particular NoSQL DBs and some of them are even
betting their business on those backends and use them for commercial
projects without any problems and even that doesn't seem to be
convincing enough. I understand your side: You don't feel qualified to
make a decision. That's fine. But from my side it looks like I'm in
fact wasting my time because I don't get trusted, anyway. So, is
anyone in the Django core team experienced enough with NoSQL DBs to
make a qualified analysis and is that person serious about helping to
get NoSQL support into Django 1.4? Without such a person helping us it
doesn't make sense to continue this project.

If you want to know whether users face surprises with Django-nonrel
please read this post from yesterday, for example:
http://groups.google.com/group/google-appengine-python/msg/2ee52c2aa8ab720b

> Lastly, we need to resolve the AutoField problem. This is the biggest
> outstanding technical issue. I can't say I've got any particularly
> brilliant ideas on how to solve it. Suggestions are welcome.

Wasn't your original suggestion good enough? We could detect whether
the user only has SQL backends and activate the old validation
behavior, but show a deprecation warning whenever things are expected
to change. If the user has a NoSQL backend the new validation behavior
is automatically activated. A few releases later we can get rid of the
old behavior.

> As for timing: I'm on record saying a number of times now that this
> isn't something I'm targeting for 1.3. We need to have a release that
> is low on features and high on bugfixes. Given the time required to
> make large scale database backend changes, I simply don't have enough
> bandwidth to satisfy the community needs for 1.3 *and* the needs of
> query-refactor. However, if people like yourself that are motivated in
> this are can get the pieces in place during the 1.3 development cycle,
> query-refactor could be an early delivery in the 1.4 timeframe (i.e.,
> early next year).

That's fine.

Bye,
Waldemar

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: four NoSQL backends you want? :)

[Russell Keith-Magee]

On Sat, Sep 25, 2010 at 8:47 PM, Waldemar Kornewald
 wrote:
> Hi,
> in my last discussion on django-users Russell told me that he'd like
> to see four proof-of-concept (or better) backends before considering
> NoSQL for inclusion in trunk.

The number 4 wasn't actually the important bit - it was that I want to
see a range of noSQL approaches represented. I don't want to see 4
key-value stores; I want to see a key-value store, and a document
store, and a column store, etc. I need to see that the API is rich
enough to cover all noSQL bases.

> The primary point was that enough
> eyeballs have looked at the API, first. Now we finally have four
> backends for Django-nonrel:
>
> App Engine (used in production), developed by Thomas Wanschik and me
> http://www.allbuttonspressed.com/projects/djangoappengine
>
> MongoDB (used in production), developed by Flavio Percoco Premoli,
> Alberto Paro, George Karpenkov, and me
> http://github.com/FlaPer87/django-mongodb-engine
>
> Cassandra (alpha/experimental), developed by Rob Vaterlaus with very
> minor hints from me
> http://github.com/vaterlaus/django_cassandra_backend
>
> ElasticSearch (alpha/experimental), developed by Alberto Paro
> http://github.com/aparo/django-elasticsearch
>
> All of them are based on our nonrel backend API (djangotoolbox.db). It
> should be relatively easy to extend the API to support both
> Django-nonrel and Alex' query-refactor branch at the same time. The
> backends will probably only need minor changes, if at all.
>
> Russell and the rest of the Django core team, if we make those four
> backends, developed with 12 eyeballs ;) work on the query-refactor
> branch, what else is necessary to get NoSQL support into trunk? Please
> don't tell me that you also want a CouchDB backend. :(
>
> My only requirement before porting the backends is that the
> (Native)AutoField issue gets resolved, first. That will simplify our
> porting effort and allow us to reuse our existing unit tests and
> project code.

Firstly, I need to see these backends actually implemented on query-refactor.

Secondly, I need to have a better understanding of what is meant when
you say "alpha/experimental" or "in production" support for a backend.
I have no doubt that you can get any backend working for simple cases.
I have no doubt that those simple cases may well be sufficient for
certain production purposes.

It's the complex cases that have me concerned. I need to know that a
user of  won't be overwhelmingly surprised if
they try to use Django to access their data store. What data store
features features are and aren't supported? When a feature isn't
supported, is this due to conceptual problems mapping the store onto
the ORM, technical limitations of the store itself, or technical
limitations imposed by the query-refactor branch? When a feature *is*
supported, is it supported in a way that makes conceptual sense?

To counter this, I would like to see a comprehensive summary of the
state of play of the level of support for each storage backend. This
includes:
 * How the ORM maps to the backend
 * Capabilities that are lost as a result of this mapping
 * Capabilities that have been implemented, and how
 * Capabilities that can't/haven't be implemented (and why)

Features like CSRF and session messages maintained wiki pages with
this sort of problem analysis prior to their merge to trunk. This
serves as excellent design documentation describing why the feature is
the way it is. I'd like to see an analogous page for query-refactor.

Thirdly, I'd like to see some independent verification of the claims
being made. It's fine to have the person who wrote a particular
backend saying "feature XYZ is complete", but I'd like to see
independent verification of this fact -- preferably from someone known
to the core team, not just a random stranger. The panelists at the
DjangoCon 2010 NoSQL panel would be a good starting place to look for
this sort of verification, but they're certainly not the only
candidates.

My reason for wanting this is that I'm simply not an expert in any of
these backends. I know SQL quite well, but I haven't had occasion to
try out other backends in depth. I can judge the technical merits of a
patch based on what I know, but I don't want to make a judgement based
on incompletely knowledge. I need to rely on those that I know and
trust to give me confidence that nothing has been missed.

Lastly, we need to resolve the AutoField problem. This is the biggest
outstanding technical issue. I can't say I've got any particularly
brilliant ideas on how to solve it. Suggestions are welcome.

As for timing: I'm on record saying a number of times now that this
isn't something I'm targeting for 1.3. We need to have a release that
is low on features and high on bugfixes. Given the time required to
make large scale database backend changes, I simply don't have enough
bandwidth to satisfy the community needs for 1.3 *and* the needs of
query-refactor. 

four NoSQL backends you want? :)

[Waldemar Kornewald]

Hi,
in my last discussion on django-users Russell told me that he'd like
to see four proof-of-concept (or better) backends before considering
NoSQL for inclusion in trunk. The primary point was that enough
eyeballs have looked at the API, first. Now we finally have four
backends for Django-nonrel:

App Engine (used in production), developed by Thomas Wanschik and me
http://www.allbuttonspressed.com/projects/djangoappengine

MongoDB (used in production), developed by Flavio Percoco Premoli,
Alberto Paro, George Karpenkov, and me
http://github.com/FlaPer87/django-mongodb-engine

Cassandra (alpha/experimental), developed by Rob Vaterlaus with very
minor hints from me
http://github.com/vaterlaus/django_cassandra_backend

ElasticSearch (alpha/experimental), developed by Alberto Paro
http://github.com/aparo/django-elasticsearch

All of them are based on our nonrel backend API (djangotoolbox.db). It
should be relatively easy to extend the API to support both
Django-nonrel and Alex' query-refactor branch at the same time. The
backends will probably only need minor changes, if at all.

Russell and the rest of the Django core team, if we make those four
backends, developed with 12 eyeballs ;) work on the query-refactor
branch, what else is necessary to get NoSQL support into trunk? Please
don't tell me that you also want a CouchDB backend. :(

My only requirement before porting the backends is that the
(Native)AutoField issue gets resolved, first. That will simplify our
porting effort and allow us to reuse our existing unit tests and
project code.

Bye,
Waldemar Kornewald

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.