Re: Djangopeople.net status
On Wed, Jul 13, 2011 at 5:49 PM, Yonsy Manuel Solis Ponce wrote: > well, i am interested to help > we need to fork in github from https://github.com/brutasse/djangopeople.net > ? Yes, the fork tree is slightly confusing but this repo contains all the work that's been done recently. There is also a wiki page with different tasks listed, feel free to add your ideas and choose a task! Bruno > -- > Yonsy Solis > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/django-developers/-/jsu-jhCvDmoJ. > To post to this group, send email to django-developers@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Deferred constraint checking causing problems during test runs
On Sun, Jul 10, 2011 at 3:27 PM, Jim Dalton wrote: > On Jul 10, 2011, at 3:13 AM, Simon Riggs wrote: > >> Maintaining the property of deferrable constraints seems important >> here, so changing the deferrability of constraints, or overriding it >> using the SET CONSTRAINTS command at the top of the transaction might >> not be what we want. > > Well, that's just it. We want tests to behave as much like production code as > possible, so we actually *don't* want constraint checks to be deferred. > > When you're working with DB normally in production, i.e. outside of a > transaction, constraints are checked immediately. It's only when you get into > a transaction that they are deferred. Each of our tests run inside of a > transaction, which is an artificial construct. So to emulate production, I'm > arguing that this property is not something we want (*unless* we are testing > a transaction or transaction like behavior, in which case it does make sense > to temporarily suspend constraint checks). My role here is to help with Postgres details, so any comments I make are just attempts at providing useful information. It sounds like there is a slight confusion about the way PostgreSQL works. Everything is a transaction, so there is no difference between inside/outside a transaction. You can choose whether you wish immediate or deferred constraint checking. It's completely user selectable, though the default is immediate. If you're seeing deferred checks, they can be changed. ISTM that the tests should work the same way as things do in production, and it looks possible to make that happen. >> What I would recommend is that we issue an explicit SET CONSTRAINTS >> ALL IMMEDIATE command immediately before the ROLLBACK at *end* of >> test. This will fire any outstanding checks. That way all constraint >> checks will occur in the same place they would during a commit, yet we >> can maintain the situation that the test ends with a rollback. > > This would conceivably work I think. I'm pretty sure Ramiro was exploring > this approach actually. My feeling, however, is that this still allows you to > get away with stuff you might not otherwise get away with in production. I > also think it's more helpful seeing exactly where an integrity issue came up > so you can address it. This is, for example, what allowed me to understand > the handful of bugs that were hiding, i.e. because I could trace the > Intergrity Error to the exact line of code that was triggering it. It does seem sensible to make the tests work like production. Are we sure they are different, given comments above? > Your questions raise one issue that had not occurred to me though. One > possible "problem" with putting constraint checks at the beginning is that > there is now way for a test to recover from them. If you try to put bad data > into the DB with immediate constraint checks on, you will raise and error > *and* if I'm not mistaken the transaction will be rolled back at that very > instant. So if for some reason you knew you were putting bad data in and > wanted to recover from it in your test and keep going, that would not be > possible. I'm not sure that's actually a problem, but it's certainly > something to consider. It's another dimension of behavior that is changed. If you want to recover from an error and then continue, you can use SAVEPOINTs. These allow you to put a mark in the sand and return to it in case of error. e.g. in SQL BEGIN; INSERT -- succeeds SAVEPOINT s1 INSERT -- fails check ROLLBACK to SAVEPOINT s1; INSERT -- succeeds COMMIT; I hope that info helps the decision process. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Deferred constraint checking causing problems during test runs
On Jul 14, 2011, at 1:31 AM, Simon Riggs wrote: > On Sun, Jul 10, 2011 at 3:27 PM, Jim Dalton wrote: >> On Jul 10, 2011, at 3:13 AM, Simon Riggs wrote: >> >>> Maintaining the property of deferrable constraints seems important >>> here, so changing the deferrability of constraints, or overriding it >>> using the SET CONSTRAINTS command at the top of the transaction might >>> not be what we want. >> >> Well, that's just it. We want tests to behave as much like production code >> as possible, so we actually *don't* want constraint checks to be deferred. >> >> When you're working with DB normally in production, i.e. outside of a >> transaction, constraints are checked immediately. It's only when you get >> into a transaction that they are deferred. Each of our tests run inside of a >> transaction, which is an artificial construct. So to emulate production, I'm >> arguing that this property is not something we want (*unless* we are testing >> a transaction or transaction like behavior, in which case it does make sense >> to temporarily suspend constraint checks). > > My role here is to help with Postgres details, so any comments I make > are just attempts at providing useful information. Thank you, btw, for offering your input in that capacity. I found it quite reassuring when I read your first reply that someone with your credentials was giving feedback on this issue. :) > > It sounds like there is a slight confusion about the way PostgreSQL > works. Everything is a transaction, so there is no difference between > inside/outside a transaction. > You can choose whether you wish immediate or deferred constraint > checking. It's completely user selectable, though the default is > immediate. If you're seeing deferred checks, they can be changed. > > ISTM that the tests should work the same way as things do in > production, and it looks possible to make that happen. Okay, this is interesting -- and thank you for clarifying this point, which I had grossly oversimplified. I think you actually shed a huge amount of light on part of the problem here (at least for me). So in normal, everyday use, Django opens a connection to Postgresql with an open transaction. Per the docs, it commits this transaction immediately whenever you do an INSERT or UPDATE etc. At that point Postgresql would run its constraint checks and rollback the transaction on error or else commit. During a test, we open up a big transaction during setUp and then *disable* the transaction commit and rollback operations, i.e. a call to either does nothing. Since we can't nest transactions in Postgresql (right?) this has been the only sensible way to allow tests to proceed. The problem has been -- I am positing -- that we're getting away with a lot of stuff as a result, because constraint checks are *never* checked during testing. The small slew of bugs I found is a demonstration of that, to me. My solution up to now, however, is also "unlifelike", which your comment has helped me to realize. It works kind of similar to the way Django works normally because in some ways Django sort of emulates immediate checks in its default behavior. But I think my present solution is a kludge in the other direction and I agree is not close enough to reality. I'm thinking this might not be too hard to solve though. I'm thinking that rather than *only* enabling constraint checks immediately as I was suggesting before, we could instead focus on the transaction-related methods that we are presently just blanking out (by this I mean when tests are set up, transaction.commit() transaction.rollback() etc. are being monkey patched with a function that does nothing). Even though we can't nest a transaction explicitly, we can better emulate the behavior of normal Django operation. We can keep things in deferred mode as they are in production, but then do a temporary flip to IMMEDIATE at commit() which will fire the constraint checks. Presumably, we could even do a SAVEPOINT, as you suggested, at the start so that the test could be recovered and continue on if e.g. the IntegrityError needed to be caught during testing. I imagine that this will still flush out most if not all of the little errors I had discovered (and prevent us from writing test cases going forward with similar problems) but at the same time will very closely emulate production. The challenge will be wrangling all this together with the other DBs (i.e. MySQL and SQLite) which don't share this execution model. Should be workable though. Anyhow, thank you once again Simon for shedding light on this for me. Cheers Jim -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/djan
Re: Deferred constraint checking causing problems during test runs
On Thu, Jul 14, 2011 at 2:36 PM, Jim Dalton wrote: > So in normal, everyday use, Django opens a connection to Postgresql with an > open transaction. Per the docs, it commits this transaction immediately > whenever you do an INSERT or UPDATE etc. At that point Postgresql would run > its constraint checks and rollback the transaction on error or else commit. Yes, each statement is a transaction in itself unless you explicitly request an extended transaction. > During a test, we open up a big transaction during setUp and then *disable* > the transaction commit and rollback operations, i.e. a call to either does > nothing. Since we can't nest transactions in Postgresql (right?) this has > been the only sensible way to allow tests to proceed. The problem has been > -- I am positing -- that we're getting away with a lot of stuff as a result, > because constraint checks are *never* checked during testing. The small slew > of bugs I found is a demonstration of that, to me. No, sorry, nested transaction commands aren't supported. > Anyhow, thank you once again Simon for shedding light on this for me. No problem. Thanks to everybody working on Django. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Decision for ticket #6362 - Remove blank spaces with strip when validating the data
Hi all! I like patch 2 [1] for ticket #6362. It has docs and tests, but Yuri propose more interesting way. +1 for somthing like SomeField(validators=[...], processors=[...]) where `processors` is used for normalize value to common format, and `validators` is used for validate this value. [1] https://code.djangoproject.com/attachment/ticket/6362/6362.diff Cheers! On Tue, Jul 12, 2011 at 6:22 PM, Dmitry Gladkov wrote: > Hi Yuri, > > At first I thought that we should extend validators logic, but then I > realized that validation and cleaning (agree that 'cleaners' is a > better term than 'processors') are rather different tasks and mixing > them could be ambiguous, not mentioning backwards incompatibile as a > cleaner should return a value whether a validator should not. > > > -- > Best wishes, > Dmitry Gladkov, mailto:dmitry.glad...@gmail.com > > +380 91 303-37-46 > > > > On Tue, Jul 12, 2011 at 2:09 PM, burc...@gmail.com > wrote: > > Hi Dmitry, > > I think we could have combination of "validators" + "processors": > > they will return either exception or cleaned value. > > For example, > > SomeField(cleaners=[clean_and_validate_email()]) > > Did you mean exactly this or rather separated SomeField(validators=[...], > > processors=[...]) ? > > On Mon, Jul 11, 2011 at 6:31 PM, Dmitry Gladkov < > dmitry.glad...@gmail.com> > > wrote: > >> > >> I don't see what's wrong with 'strip' attribute for models.Field, > >> we've already non database-related 'blank' attribute, but I agree that > >> it does not solve the general problem. > >> > >> What I'm thinking about is something like django.core.validators [1], > >> but called 'processors' with the only and main difference that > >> processor returns value which gets passed to the next processor in > >> chain. > >> > >> I'm not sure that it plays nice with existing clean_[fieldname] > >> methods and that 'processor' is a good name either, but I'd like to > >> hear what do you think about this idea. > >> > >> Thanks. > >> > >> > >> [1] > >> > https://docs.djangoproject.com/en/dev/ref/forms/validation/#using-validators > >> > >> > >> -- > >> Best wishes, > >> Dmitry Gladkov, mailto:dmitry.glad...@gmail.com > >> > >> +380 91 303-37-46 > >> > >> > >> > >> On Mon, Jul 11, 2011 at 12:26 AM, Chris Beaven > >> wrote: > >> > To clarify, didn't even notice we were talking about models.Field, I'm > >> > +0 > >> > for a 'strip' attribute on the form's field, nothing on the model. > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Django developers" group. > >> > To view this discussion on the web visit > >> > https://groups.google.com/d/msg/django-developers/-/r9DLUCsK6rUJ. > >> > To post to this group, send email to > django-developers@googlegroups.com. > >> > To unsubscribe from this group, send email to > >> > django-developers+unsubscr...@googlegroups.com. > >> > For more options, visit this group at > >> > http://groups.google.com/group/django-developers?hl=en. > >> > > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Django developers" group. > >> To post to this group, send email to django-developers@googlegroups.com > . > >> To unsubscribe from this group, send email to > >> django-developers+unsubscr...@googlegroups.com. > >> For more options, visit this group at > >> http://groups.google.com/group/django-developers?hl=en. > >> > > > > > > > > -- > > Best regards, Yuri V. Baburov, Skype: yuri.baburov, MSN: bu...@live.com > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Django developers" group. > > To post to this group, send email to django-developers@googlegroups.com. > > To unsubscribe from this group, send email to > > django-developers+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/django-developers?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-developers@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > -- Alex Kamedov skype: kamedovwww: kamedov.ru -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Decision for ticket #6362 - Remove blank spaces with strip when validating the data
How is that supposed to interact with the `cleaning` mechanism of the field? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/BNMXjILUK7gJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Decision for ticket #6362 - Remove blank spaces with strip when validating the data
On 10/07/11 22:26, Chris Beaven wrote: > To clarify, didn't even notice we were talking about models.Field, I'm > +0 for a 'strip' attribute on the form's field, nothing on the model. Like Chris, I don't think we can put this feature anywhere on model definition. It is clearly an issue of how a form should be processed, not an issue of what data can exist in the model. strip=True means "when receiving input from user, strip leading/trailing whitespace" - and something similar for any 'text_filter' attribute. That really cannot belong on a field definition. With that said, the next option because an option on form fields. This isn't particularly attractive to me, because for ModelForm it isn't going to be automatically applied (for the reasons Jacob has given), and now you need some fairly hacky or non-DRY way to specify it. My solution to date has been this form mixin that need it: class StripStringsMixin(object): def clean(self): for field,value in self.cleaned_data.items(): if isinstance(value, basestring): self.cleaned_data[field] = value.strip() return self.cleaned_data This mixin is easy to use - just add it to a form's base class. This is even still easy to use in the context of the admin. Although this may not be perfect, it's probably the least of all the various evils. Luke -- A former CEO: "Some of you think that only half of the Board of Directors do all the work and the rest do nothing, actually the reverse is true." (True Quotes From Induhviduals, Scott Adams) Luke Plant || http://lukeplant.me.uk/ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
