Re: deprecation of AUTH_PROFILE_MODULE

[Daniel Greenfeld]

I agree that email-as-username should be a built-in User abstract model (or 
something) in Django. It's an incredibly common use case and for once I 
think Django could use some additional functionality.

+1 to email as username model in core.

Danny

On Sunday, March 3, 2013 10:21:55 AM UTC-8, Florian Apolloner wrote:
>
> Hi Jacob,
>
> On Sunday, March 3, 2013 5:08:24 PM UTC+1, Jacob Kaplan-Moss wrote:
>>
>> I actually strongly disagree: I think Django *should* ship an 
>> "authenticate-using-email" system.
>>
>
> Out of curiosity, since I barely have this need by myself: Is it 
> "authenticate-using-email" or "use-email-as-username". Authentication via 
> Email can be done via a backend easily (though we should make the email 
> column unique then). So I am wondering what the main issue here is: is it 
> having username beeing a required field (an uuidv4 should suffice as a 
> workaround currently) or something different? 
>
> I can't promise it'd go in -- I'm not going to 
>> overrule a -1 from Florian by fiat -- but I think having a concrete 
>> patch on the table will make it easier to make a decision.
>
>
> No worries, if it would exist it would have been a -0.5; either way, a 
> concrete patch would certainly help. If it does make it into core I'd still 
> prefer the most minimal solution possible.
>
> Regarding the current proposal: Luke, can you tell us how your changes 
> would play with the current forms etc? Especially since some forms have a 
> hard coded dependency on a concrete user model (by design). I'd prefer not 
> to duplicate all those forms if somehow possible.
>
> Cheers,
> Florian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Switch to database-level autocommit

[Shai Berger]

On Sunday 03 March 2013, Jacob Kaplan-Moss wrote:
> Shai: do you have any real-world code that'd have data loss
> with this bug?

The code in sites I work on usually uses transaction middleware or 
@commit_on_success, so it would not suffer (nor benefit) from this change. So I 
tried to look for openly available code.

I found this very clear example on Stack Overflow -- I'm not sure it counts as 
"real-world code", but it definitely becomes subtly broken with database-level  
autocommit:

http://stackoverflow.com/questions/8965391/delete-duplicate-rows-in-django-db

Then I turned to somewhere I thought might be susceptible, and took a look at 
django-mptt. I wish I had more time and stamina at this point to find an actual 
smoking gun, but what I did find, was no use of transaction control: Not in 
code (the only transactional code is a call to 
transaction.commit_unless_managed at the end of the move_node operation), and 
not even in documentation (except for docstrings for context-managers which 
delay updates). The code there is pretty complex and abstract, but I'm pretty 
sure it does make decisions based on reads that affect consequent writes -- and 
I'm pretty sure that adding a commit between the former and the latter will 
make it fragile in new and interesting ways. So that is real-world code that 
I'm almost sure now has data loss possibilities, unless used in explicit 
transactions.

Which brings me to this comment:

> Looking through the tons of code I've got available
> can't reveal a single place where this chance would have any negative
> effect. OTOH, there're lots of places where it'd have a positive
> effect.
> 
Please correct me if I'm misunderstanding: For this to hold, you need lots of 
pieces of code where autocommit is on (otherwise there's no positive effect), 
and then you need to be sure that, in all these places, either reads don't 
really affect consequent writes, or some constraint holds that is equivalent to 
serializability -- otherwise, negative effect is possible. That sounds quite 
implausible.

Is the implausible really the case? Otherwise, what am I missing?

Thanks,
Shai.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: first() and last(), earliest() and latest()

[Stephen Burrows]

So if I understand correctly, you want something that returns 0-or-1 
objects, without having to do try/except? (Or alternately, try: qs[:1][0] 
except IndexError.)

First of all, I don't understand why first/last are the names you've 
chosen. This isn't about getting the first object or the last object. It's 
about getting first/last or None. That behavior is inconsistent with the 
way that Django's ORM works. (Though granted, it wouldn't be the only 
internal inconsistency.) A name like get_or_none() would make more sense to 
me, but apparently that was vetoed in the previous thread.

Second, I don't understand what you expect to gain from adding these 
methods. Yes, I know you save a few lines of code. If it doesn't matter 
whether you get an object or not, you save three lines. If you have to take 
different paths, you only save one line. Essentially you're replacing this:

try:
 obj = Model.instance.order_by(...)[:1][0]
except IndexError:
 # Do something
else:
 # Do something else

With this:

obj = Model.instance.order_by(...).first()
if obj is None:
 # Do something
else:
 # Do something else

The gains don't seem to justify the means.

Third, this seems to be about replacing latest/earliest. Personally, I 
don't have much use for either method, since they can be replaced with the 
first snippet above without any trouble. If anything, I'd rather see 
latest/earliest removed from the API than add new methods that do 
essentially the same thing. What I might see being useful would be a method 
to reverse the ordering of a queryset - which it turns out already 
exists.

Fourth, the methods proposed don't seem to do anything beyond act as 
shortcuts. In contrast, get_or_create (which on the surface is a similar 
shortcut) also adds protection against database inconsistencies by manually 
handling transactions.

On Wednesday, February 27, 2013 5:34:16 PM UTC-5, Wim Feijen wrote:
>
> Hi all,
>
> We struggled to get a proper definition for a first() and last() method 
> vs. earliest() and latest() . I'd like to make one proposal. After that, I 
> really like your opinion on which syntax you prefer.
>
> First, let me give you a lenghty introduction. We discussed several use 
> cases on this mailing 
> list.
>  
> Then, I realized that:
>
> .filter(last_name__startswith='b').order_by('last_name').first()
> is an acceptable compromise for me to use in stead of:
> .first(last_name__startswith='b').order_by('last_name')
>
> Last weekend Aymeric explained to me that earliest can actually accomplish 
> the same:
> .filter(last_name__startswith='b').earliest('last_name')
>
> Then, I find "earliest" an inappropriate name, because it does not 
> describe functioning well.
>
> Therefore, my proposal is, if we are going to implement the earliest and 
> latest method, we should definitely rename them to: first and latest.
>
> After that, there is only one question left:
>
> Which style do you prefer?
>
> .filter(last_name__startswith='b').order_by('last_name').first()# 
> clear and long
> .first(last_name__startswith='b').order_by('last_name')# first method 
> has filter syntax.
> .filter(last_name__startswith='b').first('last_name')   # first method has 
> order_by syntax.
>
> So, what do you think?
>
> Best regards,
>
> Wim
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Switch to database-level autocommit

[Shai Berger]

Hi,

Here's some thoughts on possible problems with Aymeric's plan. I like
the look of the suggested transaction API, I'm raising these to help
make sure it can be realized.

On Sunday 03 March 2013, Aymeric Augustin wrote:
> On 1 mars 2013, at 13:48, Aymeric Augustin 
>  wrote:
> 
> I'd like to add an @atomic decorator that:
>   - Really guarantees atomicity, by raising an exception if you attempt 
> commit within an atomic block,

Fragility alert: databases commit implicitly. Most notably, Oracle and MySql 
for every DDL command.

It is true that the commands that trigger this are not usually found in 
normally-executed application
code. However, since the proposal is all about running in autocommit mode 
unless a transaction
was started explicitly, it should be noted that once such a command is 
executed, it does not
just break the transaction in the middle, it renders everything after it 
non-transactional.

Most DDL commands executed by Django apps, 1.6 and on, should be done through 
the upcoming
schema alteration mechanisms -- code under core's control, so things can be set 
up correctly 
(that is, when such a command is executed under @atomic with a 
non-DDL-transactional backend,
it should raise an exception). They just need care. Other than that, strong 
documentation warnings
should be given around the use of raw sql in transactions.

>   - Supports nesting within an existing transaction, with savepoints.

The use of savepoints in Django apps so far has been very little, as far as I 
know. One point
I'm unsure of is the interaction of savepoints with cursors, since querysets 
are lazy; so the scenario
I'm worrirf about is, generally speaking,

@atomic
def main():
for obj in query_with_more_than_100_objects:
try:
handle(obj)
except Bad:
pass

@atomic
def handle(obj):
if good(obj):
mark_good(obj)
obj.save()
else:
raise Bad(obj)

Will the next (database) fetch after an exception is raised get the right 
objects?

My reading of the Postgresql documentation is that it will do the right thing, 
not so sure about
the other backends.

> 
> I'm reasonably confident that this plan can work.

My concerns about autocommit notwithstanding, I hope you're right.

Shai.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Switch to database-level autocommit

[Aymeric Augustin]

On 1 mars 2013, at 13:48, Aymeric Augustin  
wrote:

> Basically, Django intends to provide autocommit by default. Rather than fight 
> the database adapter that itselfs fights the database, I propose to simply 
> turn autocommit on, and stop implicitly starting and committing transactions. 
> Explicit is better than implicit.

This is implemented in https://github.com/django/django/pull/873 and ready for 
review.

Important notes:
- Some ORM APIs aren't atomic any longer, as noted by Shai and Christophe; 
others are using a hack: `enter_transaction_management(forced=True)`. I need a 
better transaction API to fix that, see below.
- I disabled three tests that tested the internals of transaction management in 
non-autocommit mode, which no longer exists. I don't know how much I'm going to 
refactor, so I haven't rewritten them yet.

I'd like to merge to master at this point, since I've reached a consistent, if 
not final, state.

> This obviously has far-reaching consequences on transaction handling in 
> Django, but the current APIs should still work. (Fixing them is part 2 of the 
> plan.) 


Part 2 of the plan is more ambitious — it's about improving the transaction 
APIs. Here's the general idea.

Currently, the most useful but also most problematic API is the 
@commit_on_success decorator. It's confusing because it mixes two roles.

1) Switch the transaction state from "auto mode" to "managed mode" ie. disable 
autocommit.
- Nesting works as expected because transaction state is managed as a 
stack.
- This stack is a pillar of the current implementation but I'm not 
convinced that it's an useful concept in general.

2) Define the context for a transaction — but in a very fragile way.
- It doesn't guarantee to rollback all changes if an exception occurs, 
because you can commit within the block.
- Nesting breaks atomicity, because the inner block triggers a commit 
or rollback before the end of the outer block.
- To sum up, it really does what the name says, and nothing else :)

I'd like to add an @atomic decorator that:
- Really guarantees atomicity, by raising an exception if you attempt 
commit within an atomic block,
- Supports nesting within an existing transaction, with savepoints.
- In option, support "merging" with an existing transaction, for 
pathological cases where the overhead of savepoints would be excessive: 
@atomic(merge=True)
Partial rollback is very Pythonic: it works by raising an exception and 
catching it higher in the stack. Each @atomic context that's exited with an 
exception rolls back to its savepoint.

Ideally, this decorator would start a transaction but not enable "managed 
mode". The point of "managed mode" is that you can commit at any time, and the 
next query will automatically reopen a transaction. Since @atomic forbids 
committing until the end of the block, this is no longer a useful property. 

Another good reason for staying in "auto mode" is that savepoints don't work on 
SQLite in "managed mode": cursor.execute("SAVEPOINT foo") triggers a COMMIT 
because Python's sqlite3 module tries to be smart and fails miserably.

The end goal is:
- for defining transaction blocks: to deprecate @commit_on_success / 
@commit_manually and provide @atomic instead.
- for controlling transaction state: to deprecate @commit_on_success / 
@autocommit and provide connection.set_autocommit(False/True) instead — only 
usable outside @atomic
- for the TransactionMiddleware: to provide exactly the same semantics as 
@atomic, and to keep the ability to disable it with @autocommit
- to remove the stack of transaction states, which mostly exists to support the 
current decorators, once the deprecation cycle completes.

Of course the two APIs should work in parallel until the old ones are gone.

I'm reasonably confident that this plan can work.

-- 
Aymeric.



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Switch to database-level autocommit

[Christophe Pettus]

On Mar 3, 2013, at 10:13 AM, Aymeric Augustin wrote:

> In practice, the solution is probably called @xact. Applying it to each 
> public ORM function should do the trick. Therefore, I'd like to ask your 
> permission to copy it in Django. Technically speaking, this means relicensing 
> it from PostgreSQL to BSD.

Absolutely; it would be my honor.  Just contact me off-list and we can sort out 
the details.

--
-- Christophe Pettus
   x...@thebuild.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: deprecation of AUTH_PROFILE_MODULE

[Florian Apolloner]

Hi Jacob,

On Sunday, March 3, 2013 5:08:24 PM UTC+1, Jacob Kaplan-Moss wrote:
>
> I actually strongly disagree: I think Django *should* ship an 
> "authenticate-using-email" system.
>

Out of curiosity, since I barely have this need by myself: Is it 
"authenticate-using-email" or "use-email-as-username". Authentication via 
Email can be done via a backend easily (though we should make the email 
column unique then). So I am wondering what the main issue here is: is it 
having username beeing a required field (an uuidv4 should suffice as a 
workaround currently) or something different? 

I can't promise it'd go in -- I'm not going to 
> overrule a -1 from Florian by fiat -- but I think having a concrete 
> patch on the table will make it easier to make a decision.


No worries, if it would exist it would have been a -0.5; either way, a 
concrete patch would certainly help. If it does make it into core I'd still 
prefer the most minimal solution possible.

Regarding the current proposal: Luke, can you tell us how your changes 
would play with the current forms etc? Especially since some forms have a 
hard coded dependency on a concrete user model (by design). I'd prefer not 
to duplicate all those forms if somehow possible.

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Switch to database-level autocommit

[Aymeric Augustin]

On 3 mars 2013, at 17:09, Christophe Pettus  wrote:

> Right now, the only real example I've heard (there might be more is):
> 
> 1. The ORM generates multiple updating operations for a single API-level 
> operation.
> 2. The developer did nothing to manage their transaction model (no decorator, 
> no middleware), but,
> 3. Is relying on Django to provide a transaction in this case.
> 
> That situation does exist, but it does seem pretty edge-case-y.  Does it 
> exist in any case besides model inheritance?  If not, could we have the ORM 
> wrap those operations in a transaction in that particular case?

Yes, I plan to ensure that all public ORM APIs are atomic. I'm not sure that 
they currently are; under autocommit, they probably aren't.

I will make a proposal to improve transaction management right after I've 
finished implementing autocommit.

In practice, the solution is probably called @xact. Applying it to each public 
ORM function should do the trick. Therefore, I'd like to ask your permission to 
copy it in Django. Technically speaking, this means relicensing it from 
PostgreSQL to BSD.

Thanks,

-- 
Aymeric.



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Switch to database-level autocommit

[Christophe Pettus]

On Mar 2, 2013, at 3:49 PM, Jacob Kaplan-Moss wrote:
> I'm with Aymeric: the current behavior is bad enough, and this is a
> big enough improvement, and the backwards-incompatibility is minor
> enough.

Right now, the only real example I've heard (there might be more is):

1. The ORM generates multiple updating operations for a single API-level 
operation.
2. The developer did nothing to manage their transaction model (no decorator, 
no middleware), but,
3. Is relying on Django to provide a transaction in this case.

That situation does exist, but it does seem pretty edge-case-y.  Does it exist 
in any case besides model inheritance?  If not, could we have the ORM wrap 
those operations in a transaction in that particular case?
--
-- Christophe Pettus
   x...@thebuild.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: deprecation of AUTH_PROFILE_MODULE

[Jacob Kaplan-Moss]

On Fri, Mar 1, 2013 at 12:04 PM, Florian Apolloner
 wrote:
> Doing it outside of Django core if fine, inside core is -0 to -1 from me.

I actually strongly disagree: I think Django *should* ship an
"authenticate-using-email" system. It's *SUCH* a common desire, and
it's actually fairly tricky to get all the pieces in place to do it
right. I just cruised StackOverflow: it's asked a ton, and the answers
all have something not-quite-right about them, especially now that 1.5
has shipped.

So I'm +1 on including something in core. Luke, do you think you can
work up a patch? I can't promise it'd go in -- I'm not going to
overrule a -1 from Florian by fiat -- but I think having a concrete
patch on the table will make it easier to make a decision. So if you
don't mind doing a bit of "spec work" that'd probably give you us best
chances of getting this in.

Jacob

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Transforming django-admin.py to a shell script

[Alon Nisser]

Yes! Actually I also bumped into the annoying `ImportError: No module named 
django.core` problem, so fixing virtualenv would be a blessing.
*But.. *I still think this is the wrong path, since python setuptools has a 
better solution for that with cross platform compatibility etc.
Take a look 
at https://github.com/TurboGears/tg2devtools/blob/master/setup.py *vs *
https://github.com/django/django/blob/master/setup.py
you can see that Turbogears (as well as almost any other python 
webframework or commandline tool) uses setuptools entry points to establish 
a commandline script (under the working scripts/bin folder, according to 
the platform). why wouldn't Django do that? and then solve both problems in 
one shoot?

On Friday, March 1, 2013 8:25:50 PM UTC+2, Alon Nisser wrote:
>
> and then it could be called as `django-admin somecommend` instead of 
> `python django-admin.py somecommand`. 
> since python (using setuptools entry points) makes making a python script 
> into a shell script quite easy I guess this has been Discussed before but I 
> didn't find the discussion and the explanation why doesn't Django team deem 
> this path worthy.
> I think most python web frameworks use some kind of a shell script (or 
> using a customized paster/gearbox to provide this functionality) except for 
> Django.
>
> thanks for the clarification
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.