date:20180830

Re: Help wanted testing proposal for the Migration Graph algorithm

2018-08-30 Thread Carlton Gibson

Hi Dan. 

Super! 

On Tuesday, 28 August 2018 17:28:32 UTC+2, d...@thread.com wrote:
>
> I've run it on our codebase with ~1100 migrations and ~380 apps.
>

Yes! This is what I was looking for. 

> There were no exceptions thrown - the script completed cleanly, although I 
> haven't actually migrated with it or checked that the migrations work. I 
> assume the script is enough to ensure consistent migrations? 
>

Perfect. The changes just relate to how the graph of migrations is built. 
This feeds into forward_plan, backwards_plan etc. (The individual 
migrations remain the same; theres no real need to migrate.) 

That the script executes without error on a large project is more or less 
what we want to see here. (There was additional input on the PR confirming 
the same, so that's good.) 

There's a second version of the script that writes the generated plans to a 
file. If you were to run this with v2.1 or master and the PR branch, you 
should find the generated plans to be same (i.e. the files to be 
identical). I'll paste this script at the bottom. If you have time to run 
it, as a extra check, that would be cool.

Around a 3.5% speedup.
>

OK. Perfect.

It looks like the kind of cycles that would lead to worst case performance 
don't really come up. (Most migrations just have a single dependency on the 
previous one, and that's it.) 
As such the performance boost of this patch is not massive. (Still worth 
having, other things being equal.)

We would expect slightly better improvement in real-life, since the script 
here recreates the loader (which regenerates the graph) each time through 
the loop, whereas `migrate` etc re-use the loader (which in the patch, vs 
master, only does the graph validation a single time).

However, still, performance will not be the key issue. 

Currently, with the feedback, my thinking is that the patch is a good 
refactoring (that will likely lead to further improvements). So I'm keen.
I just need to go over the code and tests again. 

Second version of script to follow. 

Thanks for the input. It's really helpful! 

Kind Regards,

Carlton

# Save as e.g. migration_graph.py
# Set DJANGO_SETTINGS_MODULE
# From project dir run: python migration_graph.py
# compare output from v2.1 or higher and PR. 
#
import sys

import django
django.setup()

from django.db import connection
from django.db.migrations.loader import MigrationLoader

f = open('plans-{}'.format(django.__version__), 'w')
print('Writing plans for v{}'.format(django.__version__))

loader = MigrationLoader(connection)
backwards = loader.graph.root_nodes()
forwards = loader.graph.leaf_nodes()

print('Calculating backwards plans:')
for root in backwards:
loader = MigrationLoader(connection)
sys.stdout.write('.')
plan = loader.graph.backwards_plan(root)
f.write(str(plan))
f.write('\n')

f.write('\n' * 3)
sys.stdout.write('\n')

print('Calculating forwards plans:')
for leaf in forwards:
loader = MigrationLoader(connection)
sys.stdout.write('.')
plan = loader.graph.forwards_plan(leaf)
f.write(str(plan))
f.write('\n')

f.close()
sys.stdout.write('\nRun Done\n\n')

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/24ad1be0-c2a4-4a24-9b6c-24a748a09dcb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fellow Reports -- August 2018

2018-08-30 Thread Carlton Gibson

Hi all,


Calendar Week 33 -- ending 19 August.


Triaged:

https://code.djangoproject.com/ticket/29672 -- Returns an empty model field 
that is filled with a trigger in the database. (Invalid)
https://code.djangoproject.com/ticket/29656 -- Range Fields do not support 
blank values via ModelForm (Duplicate of #27039.)
https://code.djangoproject.com/ticket/29671 -- Unable to modify UserAdmin 
to prevent editing/viewing passwords (invalid)
https://code.djangoproject.com/ticket/29669 -- admin.E033 incorrectly 
raised when adding a calculated field in the 'ordering' tuple of a 
ModelAdmin (Accepted)
https://code.djangoproject.com/ticket/29664 -- Need Proper error handling 
for CBV in urls. (wontfix)
https://code.djangoproject.com/ticket/29667 -- path converters don't 
handle spaces well. (Accepted)


Reviewed:

https://code.djangoproject.com/ticket/29681 -- Support XHTML5 (XML 
serialization of HTML5)
https://github.com/django/django/pull/9212 -- Fixed a race condition 
problem in migrations howto.
https://github.com/django/django/pull/10303 -- close # TODO: why can't 
I make this ..app2
https://code.djangoproject.com/ticket/29677 -- 
StaticFilesStorage.post_process does not seem to exist, contrary to 
documentation
https://code.djangoproject.com/ticket/29646 -- Document the validators that 
each model and form field uses
https://code.djangoproject.com/ticket/29529 -- Allow FilePathField path to 
accept a callable.
https://code.djangoproject.com/ticket/29243 -- Refactor migration graph code



Calendar Week 34 -- ending 26 August.


Triaged:

https://code.djangoproject.com/ticket/29707 -- New admin autocomplete 
widget ignores limit_choices_to filter in referring FK definition 
(Accepted.)
https://code.djangoproject.com/ticket/29704 -- manage.py test does not 
always accept test-runner specific command line options (Accepted.)
https://code.djangoproject.com/ticket/29705 -- RuntimeError while saving 
webp file to Image-Field (Accepted.)
https://code.djangoproject.com/ticket/29700 -- Document 
ModelAdmin.autocomplete_view() and AutocompleteJsonView (as customisation 
point). (Accepted.)
https://code.djangoproject.com/ticket/29680 -- Paginate start with last 
Page (wontfix)
https://code.djangoproject.com/ticket/29691 -- Support ForeignKey based 
model inheritance (wontfix)
https://code.djangoproject.com/ticket/29694 -- QuerySet.values_list() 
combined with .extra() or .annotate() may produce wrong .union() (Duplicate 
of #29229)


Reviewed:

https://code.djangoproject.com/ticket/29688 -- ModelAdmin: Add attribute to 
override manager used by ModelAdmin.get_queryset()
https://code.djangoproject.com/ticket/29243 -- Improve efficiency of 
migration graph algorithm



Kind Regards,

Carlton

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1494bc67-b57b-4d70-8812-754e08c3c548%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Proposal: relationships based on arbitrary predicates

2018-08-30 Thread Alexander Hill

Hi all,

I've run into many situations during my time using Django where I've wanted
to be able to express relations based on some other criteria than foreign
key equality. A few examples:
- descendants or children of a node in a tree structure
- saved search terms to search results
- a model containing a date range to timestamped items falling within that
date range

Currently to do this kind of thing, you might write a getter which returns
a queryset - think for example mptt's get_descendants(). But you don't get
any of the nice things a real relation field gives you - you can't use that
relationship in filters, you can't select/prefetch_related() or values(),
there's no reverse relationship, etc.

I've written a Relationship field[0] that lets you define relations in
terms of arbitrary Q filters containing objects of a new type, L. An L is
like an F, but represents a field on the "local" or "left" side of the
relation, where the Q is filtering against the remote "to" side of the
relation. For example, in a materialised path tree, this is how you might
express descendants:

class Node(models.Model):
path = models.TextField()
descendants = Relationship(
"self",
Q(path__startswith=L('path'), pk__ne=L('pk')),
multiple=True,
reverse_multiple=True,
related_name='ascendants',
)

Now you can use the descendants field like any other many-to-many field in
all the places I mentioned above, but the relationship is based purely on
prefix-matching on the path field. You also get an ascendants field on
Node, which represents the path back to the root and can be used in the
same way.

I think this could make a nice new feature for Django. It would give a
usability boost to anyone using MPTT or treebeard, for example. It works OK
as a third-party library, but the current implementation relies heavily on
undocumented ORM internals, and there are a few features I'd like to
implement that are impractical without making some ORM changes.

Thoughts/feedback/questions welcome!

Thanks,
Alex

[0] https://github.com/alexhill/django-relativity

--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CA%2BKBOKzXDHJnO09Trkws2MJDu4DnOUsgxgA634xpCAKk1O5xJw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal: relationships based on arbitrary predicates

2018-08-30 Thread charettes

Hello Alex!

Thanks for your work on this project, this is definitely something that I 
believe would be useful in Django's core based on the number of times I 
implemented a filtered queryset getter on Models.

I'm not totally sold on the API but having an analog of what ForeignObject 
is to ForeignKey for ManyToManyField would definitely be useful.

>From what I can see in relativity.fields[0] most of the additional logic 
revolves around the extra filtering capabilites through Restriction.

Do you have an idea of what the fields.related inheritance chain would look 
like if it was part of core? I feel like having Relation(RelatedField), 
ForeignObject(Relation), ManyToManyField(Relation) and adding the filtering 
logic to Relation could work but I'd be interested to hear what you think 
here. FWIW Anssi implemented something similar[1] for reverse unique 
relationship before FilteredRelation() was introduced.

In a sense Relation would be form of virtual field like ForeignObject since 
it's not in charge of any database field handling.

Simon

[0] 
https://github.com/AlexHill/django-relativity/blob/3802608c64e86c62ab6268efc37a3f5ca8539221/relativity/fields.py
[1] https://github.com/akaariai/django-reverse-unique



Le jeudi 30 août 2018 11:38:16 UTC-4, Alex Hill a écrit :
>
> Hi all,
>
> I've run into many situations during my time using Django where I've 
> wanted to be able to express relations based on some other criteria than 
> foreign key equality. A few examples:
> - descendants or children of a node in a tree structure
> - saved search terms to search results
> - a model containing a date range to timestamped items falling within that 
> date range
>
> Currently to do this kind of thing, you might write a getter which returns 
> a queryset - think for example mptt's get_descendants(). But you don't get 
> any of the nice things a real relation field gives you - you can't use that 
> relationship in filters, you can't select/prefetch_related() or values(), 
> there's no reverse relationship, etc.
>
> I've written a Relationship field[0] that lets you define relations in 
> terms of arbitrary Q filters containing objects of a new type, L. An L is 
> like an F, but represents a field on the "local" or "left" side of the 
> relation, where the Q is filtering against the remote "to" side of the 
> relation. For example, in a materialised path tree, this is how you might 
> express descendants:
>
> class Node(models.Model):
> path = models.TextField()
> descendants = Relationship(
> "self",
> Q(path__startswith=L('path'), pk__ne=L('pk')),
> multiple=True,
> reverse_multiple=True,
> related_name='ascendants',
> )
>
> Now you can use the descendants field like any other many-to-many field in 
> all the places I mentioned above, but the relationship is based purely on 
> prefix-matching on the path field. You also get an ascendants field on 
> Node, which represents the path back to the root and can be used in the 
> same way.
>
> I think this could make a nice new feature for Django. It would give a 
> usability boost to anyone using MPTT or treebeard, for example. It works OK 
> as a third-party library, but the current implementation relies heavily on 
> undocumented ORM internals, and there are a few features I'd like to 
> implement that are impractical without making some ORM changes.
>
> Thoughts/feedback/questions welcome!
>
> Thanks,
> Alex
>
> [0] https://github.com/alexhill/django-relativity
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c87af098-baa4-4d45-9a4b-757166b41734%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add autocomplete attribute to contrib.auth fields?

2018-08-30 Thread Adam Johnson

I wouldn't think such pentest tools are a major concern, as I'd guess not
many users have to meet such requirements, and it would always be possible
to subclass the forms and set autocomplete=off where appropriate. Also the
referred mdn docs

state
that autocomplete=off is often ignored.

Browser may be limited but I would think it's fine to add them now, they'll
ignore the values they don't know until they support them.



On Sat, 25 Aug 2018 at 20:11, Tom Forbes  wrote:

> I don’t have much to add other than it’s pretty common for pentests to
> flag autocomplete being enabled on sensitive fields (email/password) and
> recommend disabling it (autocomplete=off). While I’m not sure if I agree
> with that recommendation in some situations you have little choice but to
> follow it.
>
>
>
>
> On 25 August 2018 at 16:54:08, Tim Graham (timogra...@gmail.com) wrote:
>
> Browser support looks somewhat limited, so I wanted to ask if there are
> any concerns or drawbacks with adding
> autocomplete=username/email/current-password/new-password to contrib.auth's
> forms?
>
>
> Pull request: https://github.com/django/django/pull/9921
>
>
> From the ticket [https://code.djangoproject.com/ticket/29379]:
>
>
> The most useful one is autocomplete=new-password, which prevents browsers
> prefill with current password, Chrome will also suggest a random strong
> password for users who turned on account sync.
> Related docs:
> 
> https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#autofill
> 
> https://www.chromium.org/developers/design-documents/form-styles-that-chromium-understands
> 
> https://developer.mozilla.org/en-US/docs/Web/Security/Securing_your_site/Turning_off_form_autocompletion#The_autocomplete_attribute_and_login_fields
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/d398c554-3fe2-4e0f-9deb-a61dabc4cbf3%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAFNZOJMektU_6pQ6vXnRrBHEisGbUzp8sm-cE2m0-597dShHGA%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM2%3DPsWhjsdLr-EHOhGZOurCC1DKn2g%2BqF8e1qUFoOgm%2Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

New Password Validators

2018-08-30 Thread Mehmet Dogan

Hi Everybody,

Django currently ships with the following password validators:

UserAttributeSimilarityValidator
MinimumLengthValidator
CommonPasswordValidator
NumericPasswordValidator


However, it is typical nowadays to require uppercase, lowercase, at least a 
numeric character and a non-alphanumeric character, or a symbol. 

Is there any reason, why these not included in Django? I assume most 
serious Django applications use them. A sample implementation:

class HasLowerCaseValidator:
   def __init__(self):
  self.message = "The password must contain at least one lowercase 
character."

   def validate(self, password, user=None):
  if re.search('[a-z]', password) is None:
 raise ValidationError(
self.message,
code='missing_lower_case',
)

   def get_help_text(self):
  return self.message


class HasUpperCaseValidator:
   def __init__(self):
  self.message = "The password must contain at least one uppercase 
character."

   def validate(self, password, user=None):
  if re.search('[A-Z]', password) is None:
 raise ValidationError(
self.message,
code='missing_upper_case',
)

   def get_help_text(self):
  return self.message


class HasNumberValidator:
   def __init__(self):
  self.message = "The password must contain at least one numeric character."

   def validate(self, password, user=None):
  if re.search('[0-9]', password) is None:
 raise ValidationError(
self.message,
code='missing_numeric',
)

   def get_help_text(self):
  return self.message


class HasSymbolValidator:
   def __init__(self):
  self.message = "The password must contain at least one non-alphanumeric 
character (symbol)."

   def validate(self, password, user=None):
  if re.search('[^A-Za-z0-9]', password) is None:
 raise ValidationError(
self.message,
code='missing_symbol',
)

   def get_help_text(self):
  return self.message

Regards,


Mehmet 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/afd6d57c-ce8c-4d72-85d4-867cacc8ba9a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: New Password Validators

2018-08-30 Thread James Bennett

This type of enforced "complexity" does not increase security, and relevant
standards groups now recommend not trying to enforce these rules.

Quoting US NIST 800-63B, Appendix A:

> As noted above, composition rules are commonly used in an attempt to
increase the difficulty of guessing user-chosen passwords. Research has
shown, however, that users respond in very predictable ways to the
requirements imposed by composition rules [Policies]. For example, a user
that might have chosen “password” as their password would be relatively
likely to choose “Password1” if required to include an uppercase letter and
a number, or “Password1!” if a symbol is also required.

The NIST guidelines now say (800-63B §5.1.1.1):

> Memorized secrets SHALL be at least 8 characters in length if chosen by
the subscriber. Memorized secrets chosen randomly by the CSP or verifier
SHALL be at least 6 characters in length and MAY be entirely numeric. If
the CSP or verifier disallows a chosen memorized secret based on its
appearance on a blacklist of compromised values, the subscriber SHALL be
required to choose a different memorized secret. No other complexity
requirements for memorized secrets SHOULD be imposed.

I would be against adding validators to Django to try to enforce these
deprecated practices.

--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAL13Cg_%2BKMi2naSExPR0MVvBb0JnY%3DFV7A6goDHeaTWRoSpaJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Help wanted testing proposal for the Migration Graph algorithm

Re: Fellow Reports -- August 2018

Proposal: relationships based on arbitrary predicates

Re: Proposal: relationships based on arbitrary predicates

Re: Add autocomplete attribute to contrib.auth fields?

New Password Validators

Re: New Password Validators

7 matches

Site Navigation

Mail list logo

Footer information