Re: [GSoC] Switching to Jinja2 proposal
On Wednesday, March 18, 2015 at 4:44:05 PM UTC+1, Asif Saifuddin wrote: > > Hi, chris, > > So as django have the jinja 2 support out of the box what do you think the > direction of the ticket? https://code.djangoproject.com/ticket/15667 > > should the form rendering based on jinja2? or whats on our mind? If you > guide me with some direction then I might go for the ticket. looking > forward to hear from you. > > Regards > > On Sunday, March 23, 2014 at 5:24:05 PM UTC+6, Christopher Medrela wrote: >> >> On Sunday, March 23, 2014 11:51:06 AM UTC+1, Gwildor Sok wrote: >>> >>> So, there won't be a GSoC project for this, but it would be a shame to >>> let the lengthy discussion and momentum generated go to waste. Can a >>> decision on this matter be made and implemented regardless of the GSoC >>> projects? >>> >> >> I think that the initial decisions were made already -- we agreed on >> out-of-the-box support for Jinja2 and rewriting some templates into >> Jinja2. >> >> You are asking for implementing a big feature but who should do it? >> Everybody >> is a volunteer here, contributors are scare resource. As I said, I won't >> have >> enough time in the next six months to work at this feature. Maybe somebody >> will tempt to implement that for money? This feature seems to be a good >> kickstarter project -- it's an isolated issue with a clear objective. >> > Asif, I'm not developing Django since some time ago and I really don't know answer for your questions. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4f94db9c-8421-4706-a4db-63236cac40c0%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [GSoC] Switching to Jinja2 proposal
On Sunday, March 23, 2014 11:51:06 AM UTC+1, Gwildor Sok wrote: > > So, there won't be a GSoC project for this, but it would be a shame to let > the lengthy discussion and momentum generated go to waste. Can a decision > on this matter be made and implemented regardless of the GSoC projects? > I think that the initial decisions were made already -- we agreed on out-of-the-box support for Jinja2 and rewriting some templates into Jinja2. You are asking for implementing a big feature but who should do it? Everybody is a volunteer here, contributors are scare resource. As I said, I won't have enough time in the next six months to work at this feature. Maybe somebody will tempt to implement that for money? This feature seems to be a good kickstarter project -- it's an isolated issue with a clear objective. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/522d9885-3899-484a-b5ce-cb9299f51320%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [GSoC] Switching to Jinja2 proposal
Hello! I'm sorry for such long time without any feedback. Unfortunately, I'm not going to send Django proposal. My original plan was to send two proposals: one for Django and one for Breeze (Scala numerical processing library) but I lack time and I will focus on Breeze. Therefore, I won't have enough time to work at adding Jinja2 support this summer. Thank you very much for all responses! Lots of DTL aspects were discussed here and we reached consensus in some issues (like adding out-of-the-box support of Jinja2) so I hope that the time you spent at this thread is not lost time. Maybe there will be some other students eager to work at template system? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8b39a2c1-d00a-4f34-bada-8f3ccd669283%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [GSoC] Switching to Jinja2 proposal
The link is here: http://pastebin.com/DWjE1axV It times only rendering -- templates are parsed and cached in the setup. On Tuesday, February 25, 2014 8:34:14 AM UTC+1, Curtis Maloney wrote: > > Did I miss the link to your benchmark code? > Does it time just rendering, or parsing also? > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/eda8c6ea-79b6-4bb1-a4b5-7af8d2dfd3bf%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Switching to Jinja2 proposal
I've made some benchmarks. The results are:
jinja2 empty for 0.028 s
jinja2 include1.094 s
django empty for 0.435 s
django include2.134 s
Where "empty for" is an empty loop repeated 50 times:
{% for i in range_ %} {% endfor %}
And include is a similar template where the loop body includes another
template
(which is empty) inside for loop:
{% for i in range_ %}{% include "i" %}{% endfor %}
The full code is available here: http://pastebin.com/DWjE1axV
Note that including is only 1.5x faster in Jinja2 compared to Django.
Surprisingly, the main speedup is in executing for loop (19x), not it's
body.
On Friday, February 21, 2014 11:07:30 PM UTC+1, Kevin Christopher Henry
wrote:
>
> Hi Christopher,
>
>
>> ... checks the template extension and dispatch to
>> corresponding function from `django.dtl` or `jinja2`.
>>
>
> The mechanism for distinguishing the two kinds of template needs to be
> more flexible. For example, let's say I want to override a third-party
> template with my own Jinja template. In that case I need to use the same
> name (.html and all), but I want it to be processed by Jinja. The way
> django-jinja solves this is with a setting that provides a regular
> expression to determine which template names get processed by Jinja. That
> may not be the best way (I can imagine that regular expression getting
> hairy with a lot of "or" clauses), but just looking at file extensions
> isn't enough.
>
Unfortunately, you're right. So I will follow Aymeric Augustin's idea: at
the
beginning of every template there will be "{# syntax: django #}" or "{#
syntax:
jinja2 #}". If the template lacks such declaration, django template is
assumed.
I don't have a comment on the merits of your approach to inheriting from
> DTL templates, but personally this is not something I've ever needed to do.
> I'm sure there are use cases for this, that's just my experience.
>
> By contrast, something that I very often want to do is use third-party
> Django template tags in my Jinja2 templates. Right now there's just no way
> to do that (that I know of). So, if you're taking requests, please solve
> that problem. :-) And if you're forced to prioritize, I think that would be
> more useful than being able to inherit from DTL templates.
>
What kind of support do you except for third-party template tags? Suppose,
that
`cycle` tag is not builtin. Would it be acceptable to write sth like that:
dtl cycle '1' '2' as rows
It could be quite easily implemented as a Jinja2 extensions. Of course, I
guess
that you'd prefer this style:
cycle '1' '2' as rows
Unfortunately, it cannot be done easily. Each of template tags should map
to a
new Jinja2 extension. But the list of extensions must be given to
`Environment`
constructor and cannot be changed (at least this is what documentation
says).
What can we do?
1) We can tamper with Jinja2. Maybe it is possible to change list of
extensions
after environment creation? I will investigate it.
2) We need to know all template tags before rendering first template. All
tags
will be available in all templates. The drawback is that you have no
modules
that you can `load` when desired.
BTW, what about built-in template tags? Are there any issues?
>
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/ecf638ce-5811-4234-bd40-964178763090%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Switching to Jinja2 proposal
I'm sorry for such long time without any reply, but I was investigating
possible
approaches of mixing Django and Jinja2 templates.
On Sunday, February 16, 2014 1:08:49 AM UTC+1, Russell Keith-Magee wrote:
>
>
> On Sun, Feb 16, 2014 at 12:43 AM, Christopher Medrela > wrote:
>
>> My last post was pretty long and the most important questions and
>> statements
>> have left unanswered, so I will repeat them.
>>
>> What I'm proposing now is more conservative proposal. Firstly, Django will
>> support Jinja2 out-of-the-box, but DTL will remain the "blessed" option.
>>
>
> As a broad statement, this sounds fine; but what does this mean in
> practice? What does "out of the box" support look like?
>
The two main principles would be: 1) reusing existing interfaces and
contracts
2) making switching from Djinja/jingo/django-jinja/coffin as easy as
possible.
Last month, Djinja and jingo were downloaded 650 times while coffin and
django-jinja have each one 2k downloads. Coffin focus on imitating DTL.
Therefore, the "out of box" support should be similar to django-jinja.
Basically, the same functions will apply to Jinja2 as well as to Django
templates. If you want to render a template, you write
``render_to_response(template_name)``, whether the template is Jinja2 or
Django
template. Templates will be distinguished by extension.
How can it be achieved?
- We need to move all existing functions (like `render_to_response`) from
django.template to a new module. Let's call it `django.dtl`.
- Then, we can create dummy implementations of `render_to_response` (and
all
other functions) that checks the template extension and dispatch to
corresponding function from `django.dtl` or `jinja2`.
- It would be nice to use the same loaders for both django.dtl and jinja2.
This
requires to rewrite existing Django loaders so that they fulfill the
contract
of `jinja2.Loader`.
Secondly, Django will allow to mix DTL and Jinja2 templates (so you can
>> include/inherit DTL template from Jinja2 one and vice versa).
>
>
> Including doesn't sound like it would be any problem, but inheriting? Is
> that really going to be possible?
It seems to be possible. Consider parent Jinja2 template:
Parent
{% block overridden %}parent overridden block{% endblock %}
{% block nonoverridden %}parent nonoverridden block{% endblock %}
as well as child Jinja2 template:
{% extends "parent.jinja2" %}
{% block overridden %}child overridden block{% endblock %}
Jinja2 compiles templates by generating valid Python code and passing it to
`exec` function. The appropriate code for the former is:
from __future__ import division
from jinja2.runtime import LoopContext, TemplateReference, Macro,
Markup, TemplateRuntimeError, missing, concat, escape, markup_join,
unicode_join, to_string, identity, TemplateNotFound
name = 'parent'
def root(context, environment=environment):
if 0: yield None
yield u'Parent '
for event in context.blocks['overridden'][0](context):
yield event
for event in context.blocks['nonoverridden'][0](context):
yield event
def block_nonoverridden(context, environment=environment):
if 0: yield None
yield u'parent nonoverridden block'
def block_overridden(context, environment=environment):
if 0: yield None
yield u'parent overridden block'
blocks = {'nonoverridden': block_nonoverridden, 'overridden':
block_overridden}
debug_info = '1=8'
and for the child is:
from __future__ import division
from jinja2.runtime import LoopContext, TemplateReference, Macro,
Markup, TemplateRuntimeError, missing, concat, escape, markup_join,
unicode_join, to_string, identity, TemplateNotFound
name = 'child'
def root(context, environment=environment):
parent_template = None
if 0: yield None
parent_template = environment.get_template('parent.jinja2', 'child')
for name, parent_block in parent_template.blocks.iteritems():
context.blocks.setdefault(name, []).append(parent_block)
for event in parent_template.root_render_func(context):
yield event
def block_overridden(context, environment=environment):
if 0: yield None
yield u'child overridden block'
blocks = {'overridden': block_overridden}
debug_info = '1=8'
As you can see, child template loads parent template using
`environment.get_template` method; then iterates over its blocks; finally
delegates to it by calling `root_render_func` and passing the context.
We can write Jinja2 loader that checks the template extension; if it's
&qu
Re: [GSoC] Switching to Jinja2 proposal
My last post was pretty long and the most important questions and statements have left unanswered, so I will repeat them. What I'm proposing now is more conservative proposal. Firstly, Django will support Jinja2 out-of-the-box, but DTL will remain the "blessed" option. Secondly, Django will allow to mix DTL and Jinja2 templates (so you can include/inherit DTL template from Jinja2 one and vice versa). After doing it, I could focus on 3) decoupling DTL or/and 4) rewriting Django builtin templates in Jinja2 or/and 5) moving rendering form widgets from Python code to Jinja2 templates. After that all, we could start again the war DTL vs Jinja2, but please focus on the new proposal now. Questions are: 1) What do you think about the new proposal? Would it be useful? 2) Jinja2 doesn't support 3.2. Will Django 1.8 support 3.2? 3) Supporting Jinja2 out-of-the-box means introducing dependencies. Are we ready for this? On Tuesday, February 11, 2014 2:07:19 PM UTC+1, Aymeric Augustin wrote: > > 2014-02-11 13:42 GMT+01:00 Christopher Medrela : > > >> What did Armin said about Python 3 exactly? > > > He wrote an extensive argumentation about "why Python 2 [is] the better > language for dealing with text and bytes" [1] as well as a number of tweets > and a few other blog posts along the same lines. > > While his arguments are technically correct, I disagree with his > conclusions > because he's speaking with the point of view of an expert maintaining > libraries at the boundary between unicode and bytes (like werkzeug). > However, > most Python users aren't experts and aren't maintaining such libraries. In > my > experience working with Python programmers ranging from intern to veteran, > the > unicode model of Python 3 is a strict improvement over Python 2 in terms of > pitfalls hit in day-to-day programming. YMMV. > > [1] http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/ > > -- > Aymeric. > OK, so Armin finds Python 2 better than Python 3. But why is it at odds with Django? He didn't say that he is not going to support Python 3. So where is the risk that concerns you? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/79dbbf71-6b70-48d1-8510-cef471812677%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Switching to Jinja2 proposal
I'm really astonished how much feedback I got! On Sunday, February 9, 2014 1:11:19 AM UTC+1, Russell Keith-Magee wrote: > > On Sun, Feb 9, 2014 at 6:16 AM, Christopher Medrela > wrote: > >> Hello! GSoC 2014 is coming and I'm thinking about issue to work on. >> >> The template system is one of the components that are of special interest >> to me. >> One of the major issues is that rendering templates is slow. The problem >> could >> be solved by compiling template to Python bytecode, but it seems to be >> really >> hard to achieve, given that there was an unsuccessful attempt. >> > > This should set off a red flag for you. The GSoC project to byte code > compile Django's templates was implemented by Armin, the same person who > wrote Jinja2 - and yet the project didn't fully succeed. It's worth > investigating *why* this idea failed, because it flags one of the reasons > why "just adopt Jinja2" may not be a viable options. > Armin said in his proposal [1] that he wanted to "rewrite Jinja2 code generation to better support alternative Python implementations like PyPy" and I guess that was the reason why he didn't adopt "just switch to Jinja2" approach. Also note, that his proposal was about writing a backend for both Jinja2 and DTL. That was really ambitious so there is no risk like "he failed writing backend for two template engines so switching to Jinja2 will probably also fail". [1] https://groups.google.com/forum/#!searchin/django-developers/gsoc$20compilation/django-developers/amUQgS4QvOM The pros are obvious: 1) fast templates, 2) less code to maintain, 3) lot's >> of >> companies use Jinja2 (because of slowness of default template system) and >> builtin support for Jinja2 would be beneficial for them (thing about >> integrating Jinja2 with settings like TEMPLATE_DEBUG). >> >> Now the cons. First of all, one of the design decision is that Django has >> no >> dependencies. We can overwhelm it by "static linking" -- I mean copying >> Jinja2 >> code into Django. At the first glance, it may look like a horrible idea, >> but >> think about it in a different way. If we don't switch to Jinja2, we have >> to >> maintain entire template system and fix every bug as well as implement new >> features. If we switch, Jinja2 developers can do this job for us. We only >> need >> to forward tickets to Jinja2 developers and update the static linkage. >> > > We're unlikely to vendor a copy of Jinja2. If we went down this road, we'd > be much more likely to look at using dependencies defined in setup.py. > So we need to discuss introducing dependencies. Django design decision were made long time ago when python packaging mechanism was bad, so avoiding dependencies was obvious, but this argument is no longer valid. So are we ready to introduce Jinja2 dependency in Django 1.8? This is important question because introducing dependencies is necessary condition for switching/supporting Jinja2 and if we don't reach a consensus, it'll be better to stop discussion and focus on more productive activities (like considering other GSoC ideas). BTW, I'd like to have an internship in the late summer. It's impossible to >> work at GSoC and have an internship at the same time, but I really want >> to do >> both, so I need to start GSoC as early as possible, at 21 April or even >> earlier. Is it possible? >> > > It depends on exactly how long the overlapping period is. If it's a matter > of a week or two, and we have a good proposal from a student, I suspect > we'd be happy to internally shift the dates for the GSoC by a fortnight to > accommodate them. In the past, we've accommodated students who have a 2 > week exam period in the middle of the GSoC; I don't see why we wouldn't > extend the same courtesy to an overlap at the end of the GSoC. However, the > longer the overlap, the less likely we are to be accommodating. > Before approx. April, I can't say how much do I need to shift dates. I'm going to apply for these internships which won't collide with GSoC and more we shift the dates, more internships are available for me. Although Jinja2 and Django template share a common base syntax, Jinja2 > includes a bunch of features that I'm not wild about. Django's templates > are *deliberately* hobbled to prevent the injection of business logic into > templates. Jinja2 template allow for function calls, array subscripting, > and all sorts of other programming language structures. I'm not saying > these things are inherently bad; I'm saying t
[GSoC] Switching to Jinja2 proposal
Hello! GSoC 2014 is coming and I'm thinking about issue to work on. The template system is one of the components that are of special interest to me. One of the major issues is that rendering templates is slow. The problem could be solved by compiling template to Python bytecode, but it seems to be really hard to achieve, given that there was an unsuccessful attempt. Why not switching to Jinja2? I thought that somebody else proposed this idea but I couldn't find any discussion; so, please, point me to the discussion if the idea was discussed, otherwise let's discuss it! The pros are obvious: 1) fast templates, 2) less code to maintain, 3) lot's of companies use Jinja2 (because of slowness of default template system) and builtin support for Jinja2 would be beneficial for them (thing about integrating Jinja2 with settings like TEMPLATE_DEBUG). Now the cons. First of all, one of the design decision is that Django has no dependencies. We can overwhelm it by "static linking" -- I mean copying Jinja2 code into Django. At the first glance, it may look like a horrible idea, but think about it in a different way. If we don't switch to Jinja2, we have to maintain entire template system and fix every bug as well as implement new features. If we switch, Jinja2 developers can do this job for us. We only need to forward tickets to Jinja2 developers and update the static linkage. The second big problem is that switching is a big change and backward compatibility matters. We will need to support both the deprecated Django template system and the new one. However, it doesn't mean double work -- we don't need to implement new features for the deprecated system, only bug fixes will be required. Also note, that a lot of companies uses Jinja2 and switching from third-package Jinja2 to Jinja2-builtin-Django isn't an enormous change at all. I'd like to hear your opinion. Feel free to comment! BTW, I'd like to have an internship in the late summer. It's impossible to work at GSoC and have an internship at the same time, but I really want to do both, so I need to start GSoC as early as possible, at 21 April or even earlier. Is it possible? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a6a176b2-b380-4ba7-8bad-0f3d6719cb03%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: GSoC query
There is a list of ideas [1] and both improving aggregates and annotations as well as improving error messages are listed there, so I suppose these ideas are still open. Google doesn't allow you to submit your proposal before March 10, but the submission is only a formality and this doesn't mean you can't work at your proposal. The most important thing is that you should discuss your proposal with Django community before submission, because you will be given many constructive feedback that will help you make your proposal better and improve chances of being accepted. My experience is that it's better to start as early as possible and not to hesitate to publish proposal. AFAIK, there is no problem if there is more than one student working on the same idea. Disclaimer: I'm not a core dev neither GSoC admin, but I was GSoC participant (as a student) last year. [1] https://code.djangoproject.com/wiki/SummerOfCode2014 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0e3aaa96-0a5f-4948-b9d3-95522b9b1795%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Proposal to end the war with flake8 warnings
I've used autopep8 to make some additional cleanup -- see this pull request: https://github.com/django/django/pull/1875 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/93e61ad1-a47b-4bc3-8b6b-e393c3b7a79d%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Django 1.6 and migrating to the new test runner: note on user experience
I was working at system check framework [1] during this holiday. I've done some changes that will solve all problems considered in this thread. Maybe I'm a careless kind of guy, but when I test out new versions of > Django, reading all of the release notes is rarely the first thing that > I do... my bad. :) My first action is usually firing up the test suites > and looking at the number of failed tests and deprecation warnings. > As we all know, Django-1.6 is special in this regard, since it doesn't > run all the tests that the previous version would've discovered. Since > it doesn't raise any deprecation warnings, it becomes really easy to > assume that everything is fine and that the project is Django-1.6 ready, > when in fact it is completely broken. The first change is that when you run tests, all checks are performed, so you don't have to know that there is new check command in order to be notified about the test runner issue. The second problem is disabling the warning. The warning is issued when TEST_RUNNER is not defined in settings.py file. If you set it, there is no warning even if the new value is DiscoverRunner (the default one). So the warning can be silenced by setting TEST_RUNNER. And there is new SILENCED_SYSTEM_CHECKS setting so there is another way to disable the warning. I guess that my branch will be merged with 1.7 version and we cannot do much with 1.6 release. [1] https://groups.google.com/forum/#!topic/django-developers/_sde4eoNRyQ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7b19b6af-a972-47e5-b6da-af18f5a319d3%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
In order to get more attention, I've created a new thread [1]. [1] https://groups.google.com/forum/#!topic/django-developers/_sde4eoNRyQ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
System check framework
Hello!
This summer I'm a student of Google Summer of Code. I was working at
introducing a new system check framework (merging django-secure was a part
of
my proposal, but I didn't manage to do it due to my time overestimation).
The
project is coming to the end and there is a pull request [1] that is almost
ready for merging. I would like to present you with the capabilities of the
framework as well as a high-level overview. I would like to hear your
opinion
how the framework can be improved -- this is a last chance for that! This
thread is a continuation of old "[GSoC] Revamping validation framework and
merging django-secure once again" thread [2].
[1] https://github.com/django/django/pull/1364
[2] https://groups.google.com/forum/#!topic/django-developers/fEf21dtpqDE
Here I would like to say thank you to my mentor, Russell Keith-Magee, who
gave
me a lot of advices, kept an eye at my project, did reviews and supported
me.
Preston Holmes also spent a lot of time reviewing code. Many thanks to
numerous other contributors!
Now let me introduce you the framework.
*Overview*
I've introduced a new system check framework that replaces the old
validation
framework performing mainly model validation. The new framework is
responsible
for model validation, app-specific checks like ModelAdmin validation as well
as compatibility checks. It introduces concept of warnings. Warnings as well
as errors can be silenced. The framework is open-ended which means that you
can register your own check stuff.
*Messages*
The framework uses a concept of messages similar to messages from message
framework or logging framework. A message can be a light message like a
warning, info or debug; or a serious message, e.g. an error or critical
(error). Every message consists of four attributes:
- required error message (`msg`);
- optional `hint`; in order to force developers to think carefully if
they
cannot provide any useful hint, they need to explicitly pass hint=None in
that case;
- optional invalid object (`obj`);
- optional unique message identifier (`id`) used to refer to messages,
i.e.
"E001" (a core error) or "admin.W001" (a warning issued by admin app).
*Performing checks*
In order to perform all kinds of checks, type:
python manage.py check
This command does the same stuff as the old `validate` and `check` commands.
`validate` is now deprecated and delegates to `check` command.
You can still validate some specific apps:
python manage.py check admin auth
Or you can run a subset of checks labeled with a given tag:
python manage.py -t compatibility # alternative: --tag compatibility
*Silencing errors/warnings*
A new setting called SILENCED_SYSTEM_CHECKS was introduced. It's a list of
message identifiers. If you put on the list a light message like a warning,
it
won't be printed anymore. Putting on the list a serious message like an
error
won't hide the message, but will allow you to i.e. run server.
*Registering your own stuff*
If you want to add a new check, you need to write a function that accepts
`apps` and `**kwargs` arguments where `apps` is a list of applications that
should be validated (or None if all apps should be validated). The function
should return a list of messages, even if it's an empty list. Finally, you
need to register this entry point. Let's look at this snippet implementing
some security checks:
from django.conf import settings
from django.core import checks
# Label this entry point with 'security' tag. This tag can be used while
# invoking check command, e.g. you can write `./manage.py -t security`.
@checks.register('security')
def my_security_check(apps, **kwargs):
# This check does not perform any app-specific checks, so if we are
# asked to validate some specific apps, we can skip checks.
if apps is not None:
return []
if len(settings.SECRET_KEY) < 64:
return [
checks.Warning(
'Your SECRET_KEY is too short.',
hint=None, # need to explicitly pass hint even if you
cannot provide any hint
obj='settings.SECRET_KEY',
id='security.W001',
)
]
else:
return []
*Field/model/manager checks*
You don't have to register your check stuff if it's called from another
piece
of code that is registered (directly or indirectly). This is true especially
in the case of fields, models and managers. Each of these object have
`check`
method (or classmethod in the case of models) that can be overriden in order
to add some validation stuff. Consider that you implemented a field that
accepts only values from [min, max] range and you want to check if min <
max.
Here is a snippet of code:
from django.core import checks
from django.db import models
class RangedIntegerField(models.IntegerField):
def __init__(self, min=
Re: [GSoC] Revamping validation framework and merging django-secure once again
> > 4. More important changes in code: >> >> - Introduced RAW_SETTINGS_MODULE [1]. I use it in compatibility checks to >>> test >> >> if `TEST_RUNNER` setting was overriden [2]. >> >> > > Have a look at the internals of the diffsettings management command -- I'm >> not sure RAW_SETTINGS_MODULE is needed. > > > The problem is that when a test is decorated by `override_settings`, we > need to test `settings._wrapped._wrapped`, otherwise we need to check `settings._wrapped`. I and Russell agreed that this is not a huge problem since we can traverse through all "layers" of settings wrappers. However, there is another problem. By default, there are two settings file for tests: `django/conf/global_settings.py` and `tests/test_sqlite.py`. Settings from both files are mixed and indistinguishable. When no `override_settings` decorator is in use, there is only one 'layer' of settings. That means that we cannot test if `TEST_RUNNER` was overriden in `test_sqlite.py`. Any solutions? I've just noticed that we need to implement `__str__` method for ModelAdmin > *class* (not its instances) so errors involving ModelAdmins are printed > correctly. I will work at that. We decided to use "applabel.ModelAdminClassName: (admin.E033) error message" style for ModelAdmin issues, so its `__str__` method should return "applabel.ModelAdminClassName". However, ModelAdmin class doesn't know which app defines it. That means it's impossible to implement its `__str__` method. Proposals: (1) Use ModelAdmin instances instead of the class as `obj` parameter to `Error` constructor. The drawback is that we need to pass also `site`. (2) Refactoring. Now ModelAdmin class does not know anything about model it references to (is it a design decision?). After refactoring, ModelAdmin will have the model attached to itself. (3) Hardcode that case in `CheckMessage.__str__` [1]. All three doesn't look like a good solution. [1] https://github.com/chrismedrela/django/blob/c010cd06619503b7d0db914c04c6ed58c69a9d9c/django/core/checks/messages.py#L35 We probably cannot move checks of `primary_key` and `unique` living in >> `FileField.__init__`. We test if one of these two parameters was passed; >> we >> don't check their values. Consider that an user passes unique=False. This >> is >> the default value, but nevertheless, this should result in an error. We >> cannot check if the attributes where passed while performing system >> checks. I'm >> not sure if I make myself clear. > > > Why not just store a reference to the original arguments (or the relevant > subsets) in __init__(), and then validate them later in a system check? > That may seem a little indirect, but I think the validation system will be > much nicer if we can do everything in system checks instead of splitting > the work with __init__(). That's a good idea. Done in this commit [2]. [2] https://github.com/chrismedrela/django/commit/03fe680854b863020d865af5041172e4bd49943e -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
I've written four more tests. I've moved some of checks out of `__init__` methods. I've changed `tag` to `register`. I've rebased against master. I've improved documentation (added sections about filtering and silencing system checks), but I'm still polishing it. Code is ready for final review except for the issues I will mention in this post. We probably cannot move checks of `primary_key` and `unique` living in `FileField.__init__`. We test if one of these two parameters was passed; we don't check their values. Consider that an user passes unique=False. This is the default value, but nevertheless, this should result in an error. We cannot check if the attributes where passed while performing system checks. I'm not sure if I make myself clear. We must reject fk/m2m to neither a model nor model name in `__init__`. Some pieces of code (i.e. `RelatedField.contibute_to_class`) assume that `self.to` is a Model or a string. 4. More important changes in code: >> - Introduced RAW_SETTINGS_MODULE [1]. I use it in compatibility checks to >> test >> if `TEST_RUNNER` setting was overriden [2]. > > Have a look at the internals of the diffsettings management command -- I'm > not sure RAW_SETTINGS_MODULE is needed. The problem is that when a test is decorated by `override_settings`, we need to test `settings._wrapped._wrapped`, otherwise we need to check `settings._wrapped`. 2. I'm afraid that there is too little time to merge django-secure. >> September 16 is >> suggested 'pencils down' date, so there are only less than two weeks (12 >> days) + >> one buffer week, because firm 'pencils down' date is September 23. Merging >> django-secure in such little time is actually impossible -- we must >> through out >> this task from schedule. I really apologize for that. > > This is obviously disappointing, but it's better to deliver something > complete than a half-attempt. If we can get the core framework into a good > state, merging django-secure is a self-contained task that we can address > as a follow-up commit. > Also -- the GSoC will come to an end, but that doesn't mean your > contributions to Django have to… :-) Sure. However, after merging I would like to focus on other kinds of contributing like reviewing tickets, replying on mailing list and so on) -- it's easy to get bored if you do the same job all the time. - Introduced `BaseAppCache.get_models_from_apps` method [3]. This method >> returns >> all models of given set of apps (or of all apps if None is passed) and >> is used >> in `default_checks.py` [4]. > > I'm not sure I follow why this is needed -- or why it isn't just > duplicating functionality from loading.get_models()? > `get_models(app=someapp)` let you get all models from *one* app. I notice that I sometimes need to get all models from a *set* of apps. I've just noticed that we need to implement `__str__` method for ModelAdmin *class* (not its instances) so errors involving ModelAdmins are printed correctly. I will work at that. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
1. Progress: I've made improvements to admin checks. I've also finished
implementing filtering and silencing checks. I've rebased my branch against
master again.
2. I'm afraid that there is too little time to merge django-secure.
September 16 is
suggested 'pencils down' date, so there are only less than two weeks (12
days) +
one buffer week, because firm 'pencils down' date is September 23. Merging
django-secure in such little time is actually impossible -- we must through
out
this task from schedule. I really apologize for that.
My plan is that this week I will work at documentation (at this moment
there is
only a draft). I will also try to implement as much "nice to have" things as
possible. These are:
- Writing tests which would be nice to have. I mean three tests:
- [#20974],
- test for raising error when ImageField is in use but Pillow is
not installed,
- test for raising error in `BaseCommand.__init__`.
- Moving out checks performed in `__init__`.
- Checking clashes between accessors and GenericForeignKeys.
- Checks for grouped `choices`.
The second week and the backup week is for deep review. Regarding the amount
of comments I got during the first review, I guess I need *at least* one
week
for deep review.
[#20974] https://code.djangoproject.com/ticket/20974
3. As I said, I've implemented both filtering and silencing system checks.
You can run only a subset of checks like that:
./manage.py check auth -t tag1 --tag tag2
This command will run only these checks which are labeled with `tag1` or
`tag2`
tags and only `auth` app will be validated. If there is no app name, all
apps
are checked. If there is no tag name, all checks are performed.
Your check function can be labeled in this way:
from django.core.checks import register, tag
@tag('mytag')
def my_check_function(apps, **kwargs):
# apps is a list of apps that should be validated; if None, all apps
# should be checked.
return []
register(my_check_function)
The `tag` decorator works only for entry-point functions, e.g. these one
passed
to `register` function. It doesn't work for checks
functions/methods/classmethods which are called by entry-point functions
(directly or indirectly).
To silence a specific error/warning, you need to append its `id` to
SILENCED_SYSTEM_CHECKS setting. The `id` could be any string, but we use the
following convension: "E001" for core errors, "W002" for core warnings,
"applabel.E001" for errors raised by an custom app (including contrib apps,
i.e.
"sites.E001"). The error/warning number is unique to an app, e.g.
"sites.E001",
"admin.E001" and "E001" are all allowed, but "E001" and "W001" are not OK.
You
should use "E001" and "W002". To create an error/warning with given id,
pass it
as a keyword argument:
Error('Message', hint=None, obj=invalid_object, id='myapp.E001')
4. More important changes in code:
- Introduced RAW_SETTINGS_MODULE [1]. I use it in compatibility checks to
test
if `TEST_RUNNER` setting was overriden [2].
- Introduced `BaseAppCache.get_models_from_apps` method [3]. This method
returns
all models of given set of apps (or of all apps if None is passed) and is
used
in `default_checks.py` [4].
- Admin changes are backward compatible. Admin validators are deprecated in
favour of admin checks, i.e. `BaseValidator` is deprecated in favour of
`BaseModelAdminChecks`. `BaseValidator` is an almost empty class now.
`ModelAdmin.validator` class attribute is deprecated in favour of new
`checks`
attribute. If an ModelAdmin defines `validator`, we are not ignoring it.
[1]
https://github.com/chrismedrela/django/blob/gsoc2013-checks/django/conf/__init__.py#L139
[2]
https://github.com/chrismedrela/django/blob/gsoc2013-checks/django/core/checks/default_checks.py#L33
[3]
https://github.com/chrismedrela/django/blob/gsoc2013-checks/django/db/models/loading.py#L296
[4]
https://github.com/chrismedrela/django/blob/gsoc2013-checks/django/core/checks/default_checks.py
5. I've left one comment on the pull request. (I mean this one about moving
registering admin checks to `autodiscover`.)
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
1. One of my questions left unanswered on the pull request [1] (I mean this one about documentation and `__str__` use.). 2. I've finished rewriting admin checks. I've renamed `admin_validation` to `admin_checks`. I would like you to have a deep look at `fk_name` and `exclude` checks [2] as well as `inline_formsets` tests [3] (especially error messages). 2a. "applabel.modellabel[.fieldname]" is the format of error messages for models and fields. How should the one for admin look like? I propose "applabel.ModelAdminClassName". 2b. BaseModelAdmin, ModelAdmin and InlineModelAdmin classes and their checks live in separated files. `options.py` defines these classes, but checks lives in `checks.py`. We want to have these two issues separated, otherwise class definitions would become too long. Python does not support open classes, so we cannot just add some `_check_*` methods in `checks.py` to existing classes defined in `options.py`. The current approach is that `check` method is defined in `options.py`, but it delegates to appropriate functions in `checks.py` [4]. Yes, I use functions -- there is no need to have validator class, because we don't need to store anything in its instances. However, the code is really ugly. I wonder if there is any better approach. 3. I've created a new ticket [#20974] about lack of mysql-specific checks. 4. I've rebased my branch against master. On Fri, Aug 16, 2013 at 7:45 AM, Russell Keith-Magee > wrote: > >> 8. I've added a new check. If you're using a `GenericRelation` but there >>> is no >>> `GenericForeignKey` on the target model, an warning is raised. This >>> check was >>> implemented in this commit [9]. It uses `vars` builtin function to check >>> if the >>> target model has a `GenericForeignKey`. This is ugly, but I don't see a >>> better >>> approach. >> >> [9] >>> https://github.com/chrismedrela/django/commit/ab65ff2b6d6346407a11a72c072e358c7b518cf9#L1R397 >>> >> Hrm. I don't really like this, but I'm not sure I have a better option. A >> better approach would be to have GFKs turn up in get_fields, but it isn't >> your responsibility to fix the internal problems of Generic Foreign Keys. >> If we have to live with this, then we should at the very least document it >> as a FIXME, pointing at the underlying problem with _meta handling of GFKs. > > > > Michal Petruca just mailed the django-dev list with some discussion about > changes he wants to make in his own GSoC project, which drew my attention > to something I'd forgotten about. > Does _meta.virtual_fields -- contain the information you need? It looks > like it should. 5. Yes, you're right, `virtual_fields` is what I need. 6. Now I'm implementing filtering checks. [1] https://github.com/django/django/pull/1364 [2] https://github.com/chrismedrela/django/blob/gsoc2013-checks/django/contrib/admin/checks.py#L673 [3] https://github.com/chrismedrela/django/blob/gsoc2013-checks/tests/inline_formsets/tests.py#L113 [4] https://github.com/chrismedrela/django/blob/gsoc2013-checks/django/contrib/admin/checks.py [#20976] https://code.djangoproject.com/ticket/20976 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
Progress: - Converted `BaseCommand.verbosity` from bytestring into an unicode. - Integrated compatibility checks. - Deprecated "validate" command. This command delegates to "check" command now. Changed "check" command -- it performs all system checks, including model validation and compatibility checks. - Added test for check of uniqueness of field combination under ForeignObject. Now I'm working at rewriting admin checks (at gsoc2013-checks-admin branch [1]). [1] https://github.com/chrismedrela/django/tree/gsoc2013-checks-admin -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
*Progress.* - Deprecated `requires_model_validation` flag and `validate` method (both `BaseCommand` members) in favour of new `requires_system_checks` flag and `check` method. - Finished working at `contenttypes` tests. - Improved code thanks to Preston Holmes comments. Deleted dead code and added some new tests to improve code coverage. It'd be nice to have checks of clashes between GenericForeignKey and accessors. I didn't implemented it because little time left and I need to hurry up. When it was easy, I've added new tests to improve code coverage. However, there are still some tests that would be nice to have: - Test for raising an error while using an unique varchars longer than 255 characters under MySQL backend. [1] - Test for `ImageField`. When `Pillow` is not installed and `ImageField` is in use, an error should be raised. This should be tested. [2] - Test for raising warning/ImproperlyConfigured in `BaseCommand.__init__`. [3] [1] https://github.com/chrismedrela/django/blob/149800a8136ce839903f0fe9af7f548973da9244/django/db/backends/mysql/validation.py#L6 [2] https://github.com/chrismedrela/django/blob/149800a8136ce839903f0fe9af7f548973da9244/django/db/models/fields/files.py#L447 [3] https://github.com/chrismedrela/django/blob/149800a8136ce839903f0fe9af7f548973da9244/django/core/management/base.py#L202 *Filtering and silencing errors/warnings.* We need two features: ability to run only a subset of checks (aka filtering errors) and ability to silence some errors. Silencing is easy. Every warning will have a unique name like "W027". The format of the name is letter "W" followed by a unique number. The system check framework is open ended and third-party apps can register its own checks. For warnings raised by these apps, "Wnumber.applabel" (e.g. "W001.myapp") style will be used. Of course, everything can be changed and I'm open to yours suggestions, so feel free to comment and criticize it. There will be a new setting called `SILENCED_WARNINGS`. If you want to silence a warning, you put its name in this setting, e.g.: SILENCED_ERRORS = [ 'W027', ] Only light messages (warnings, infos and debugs) can be silenced. Running only a subset of check is a more complex task. We can associate every check with a set of tags (that can be done while registering checks). To run only checks tagged "mytag" or "mysecondtag", you need to type: manage.py check mytag mysecondtag However, we would like to run checks of an app (or a set of apps): manage.py check auth admin This is hard, because checks are no longer app-specific. I propose to solve this by passing an `apps` optional keyword argument to every check function. The function should only validate specified apps (or all apps if apps==None). The only problem is how to determine if we deal with a tag or app name? Consider this command: manage.py check auth mytag This should run all checks tagged "mytag". Only messages for `auth` app should be reported. I propose to collect all tags while registering check functions and if a string is one of them, then interpret it as a tag, otherwise assume it's an app name (and check if an app with given name exists). *Problems/questions.* 1. We decided to mimic message and logging framework, so every error is tagged with a level. Its value is an integer indicating how important and serious is a message. There are five predefined values: CRITICAL, ERROR, WARNING, INFO, DEBUG. However, Preston Holmes noticed that this is unpractical because we need only errors and warnings. I think we should discuss again if it's worth mimicing these frameworks. 2. On the pull request we started discussing names. "System check" is better than "check" but it suggest that it's connected with hardware. Preston proposed "Startup checks". I don't have strong opinion. To be honest, I don't thing "startup checks" is much more better than "system checks" so I will leave it as it is, but I'm still open to suggestions and I would like to see yours opinion. 3. There are some problems with moving custom User model checks. Their first source was that some tests override `INSTALLED_APPS` setting but don't override list of registered checks. So checks of `auth` app were registered (because the app was imported by some other tests), but this app wasn't installed. This ended in an error, because checks of `auth` try to load User model which is, by default, `auth.User` and `auth` is not installed. I've solved this by overriding list of registered checks (I've introduced `override_system_checks` decorator). However, there is still one red test -- `admin_scripts.test_complex_app` [4]. The problem is that this test spawns a new Django process and the decorator cannot affect this new process. This test installs two apps (`complex_app` and `simple_app`) and doesn't install `auth` app. However, `auth` app is imported, because `complex_app` imports `admin` app which imports `aut
Re: [GSoC] Revamping validation framework and merging django-secure once again
I'm still working at polishing after reviewing. I've deprecated
`requires_model_validation` and `validate`. I've started at adding tests for
contenttype fields: `GenericForeignKey` and `GenericRelation`.
I've updated gsoc2013-checks-review branch [1]. Now it's the same as
gsoc2013-checks branch [2]. I will push new work to the latter while the
former
will remain unchanged. I'm working at contenttype tests in
gsoc2013-checks-contenttypes [3] branch. The work is not finished yet and
there
are some failing tests.
@Shai Berger: Thank you for creating the ticket. I'm sorry that I
procrastinated accepting it -- I finally did it and proposed a patch. [4]
[1] https://github.com/chrismedrela/django/tree/gsoc2013-checks-review
[2] https://github.com/chrismedrela/django/tree/gsoc2013-checks
[3] https://github.com/chrismedrela/django/tree/gsoc2013-checks-contenttypes
[4] https://code.djangoproject.com/ticket/20814
Questions:
1. Output formatting. We decided that every error/warning will take one
line plus
additional one for a hint if it's provided. The justification is that a
Django
user may type "grep HINT" to filter all hints. But now I think it's
unpractical
since the lines with hints doesn't say which object is invalid. So we can:
(1)
put hint in the same line as the error message or (2) change the format to
sth
like this:
applabel.modellabel: Error message.
applabel.modellabel: HINT: Hint.
2. Is it allowed to use `GenericRelation` pointing to a model if the model
lacks
`GenericForeignKey`?
3. I've added unicode_literals import to django/core/management.py but this
affected `BaseCommand.verbosity` default value. In order not to break
commands,
I left the attribute as a bytestring [5]. Changing it to an unicode breaks
some admin_scipts tests, i. e. `CommandTypes.test_app_command` expects in
stdout:
... options=[('pythonpath', None), ... ('verbosity', '1')]
while the command prints:
... options=[('pythonpath', None), ... ('verbosity', u'1')]
[5]
https://github.com/chrismedrela/django/blob/09579fdafc05dea90db41f519b26655010a50ac2/django/core/management/base.py#L169
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
Unfortunately, I'm a bit late. I didn't suspected that polishing code after review takes so much time. Lots of my work from last Wednesday was small improvements, but there are some vital changes: The API will be consistent with API of logging module or message framework. Being consistent means that system check framework will be familiar to lots of developers. The change is that every message has its `level` now -- one of CRITICAL, ERROR, WARNING, INFO or DEBUG integer values. Django users may add their own levels (like SUPER_CRITICAL) since `level` is an integer. This change affected output format. All messages are sorted by level (and then by source of problem), so critical errors are first on the list and there is no opportunity to miss an important error in the middle of long list of warnings. There is some stuff that still need to be done as part of reviewing, i. e. deprecating `BaseCommand.validate` in favour of `check` or adding tests for `GenericForeignKey`. After finishing polishing I will focus on moving custom User model checks to auth apps, and then on rewritting admin checks. Note that there is a lot of discussion on the pull request [1]. [1] https://github.com/django/django/pull/1364 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
Progress: I've implemented registering entry points. Now there is `django.core.checks.register(callable)` function. There is no `run_in_development` and `run_in_production` arguments. I've also rewritten mechanism of triggering checking framework -- `BaseCommand.validate` calls `django.core.checks.run_checks` instead of `get_validation_errors` (which was deleted) now. Questions: 1. BaseCommand.validate have an `app` optional argument to run validation of a particular app (if app==None then all checks are performed). Unfortunately, with the current checking framework, we are not able to run checks for a particular app. The attribute isn't used anywhere in Django except for three tests [1] and I don't think that this is heavily used by third party commands. So I propose to deprecate this attribute if it's possible. [1] https://github.com/django/django/blob/master/django/contrib/auth/tests/test_management.py#L176 2a. Warnings are printed with bold, yellow foreground now (errors use red color). Is it a good choice? 2b. The rules of formatting error messages are that the error message (that may be multiline) is followed by the hint in the next line, i. e.: app_label.model_label: Error message. Error message can be multilined. In that case, the first line should be a short description and the rest should be a long description. HINT: Hints are printed in a new line. If hint is missing then there is no last line. If the invalid object is a field or a manager, then the error message starts with `app_label.model_label.field_or_manager_name: `. If it's neither a model, a manager nor a field, then '?: ' is printed. Do you have any opinion about this style? Now I will focus on moving custom user model checks from Model class to auth app (that should be easy), then rewriting admin checks and finally finishing first iteration (mainly polishing docs). -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
I've created a pull request [1] to make deep review easier. I've rolled out tests we were talking about and reverted `validate_field` -- there exist both `check_field` and `validate_field`, the first calling the second. [1] https://github.com/django/django/pull/1364 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
Progress: I've implemented manager checks. The schedule for the next 4 weeks is: 1. Manager checks (ref 4.1.7)(done)(0.5 week). 2. Entry point registering (ref 4.1.5) & rewriting mechanism of triggering checking framework (ref 4.1.9)(1.5 week). 3. Moving custom User model checks (ref 4.1.6)(0.5 week). 4. Rewriting AdminModel checks and tests (ref 4.1.8)(1 week). 5. Polishing doc (ref 4.1.10)(0.5 week). (ref 4.1.x are references to my original proposal [1]) [1] https://gist.github.com/chrismedrela/82cbda8d2a78a280a129) Let's talk about public API to register your own validation stuff. There will be `register(callable)` function in `django.core.checks`. It will return nothing. The callables will be stored in a global variable. They should obey the same contract as `check` methods: they allow for `**kwargs` and return list of errors and warnings. All registered callables will be called during checking phase (i. e. when you type `manage.py validate` or before running server). This API allows us to register, among other things, app-specific checks. But it's not necessary to write an app in order to do some checks. `register` function will have two optional parameters: `run_in_production` (default: False) and `run_in_development` (default: True) that let you specify when the callable should be called. Most checks should be performed only in development environment (which is equivalent to DEBUG==True). Security checks makes sense only in production environment (<==> DEBUG==False). I would be happy if I could hear your opinion! -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
Finally, I've rewritten all model and field tests and checks. Some new tests were written. First draft of checking framework documentation was written. All tests passes (at least at my computer). I've rebased my branch (`gsoc2013-checks` [1]) against master. The branch is ready to deep review. [1] https://github.com/chrismedrela/django/tree/gsoc2013-checks I've created pull request [2] for ticket [#19934]. The patch clarifies that importing `django.utils.image` cannot result in ImportError. I've also created ticket [#20735] about confusing ManyToManyField documentation. There is a pull request too [3]. Both pull requests were merged. [2] https://github.com/django/django/pull/1349 [3] https://github.com/django/django/pull/1350 [#19934] https://code.djangoproject.com/ticket/19934 [#20735] https://code.djangoproject.com/ticket/20735 1. I wrote some tests [4] where nose tests generators [5] would be really helpful. I tried to emulate them. Is the code I wrote a good approach? [4] https://github.com/chrismedrela/django/blob/1c14c674aa52c666ec8742c72657d5dad2efe0cb/tests/invalid_models/tests.py#L690 [5] https://nose.readthedocs.org/en/latest/writing_tests.html#test-generators 2. Two m2m through the same model are forbidden. The new error message is: applabel.modelname: Two m2m relations through the same model. The model has two many-to-many relations through the intermediary Membership model, which is not permitted. (No hint.) Is there any reasonable hint we can provide except for suggesting duplicating the intermediary model? 3. I've deleted `app_errors` [6] because it wasn't used anywhere (inside Django). Was it left by accident or was there a reason for that? [6] https://github.com/chrismedrela/django/commit/3b221caefc151f7750a322e18798c22343daba6f 4. I've renamed DatabaseValidation.validate_field to check_field and changed its arguments. They were: `errors` -- ModelErrorCollection [7], `opts` -- meta of the model of the field and `f` -- the field; The method returned None. Now there is only `field` argument and the method returns list of errors. It's not public API, but we should go through deprecation process. How can we do that regarding changed arguments? We can call old `validate_field` method where the backend specific validation is triggered [8]. By default, the old method will call `check_field` method. The old one need to transform list of errors into strings and add them to `errors` list. The problem is that at [8] we need to convert the strings again to errors. That cannot be done in a graceful way. Is there any other way to achieve deprecation? [7] https://github.com/django/django/blob/master/django/core/management/validation.py#L11 [8] https://github.com/chrismedrela/django/blob/1c14c674aa52c666ec8742c72657d5dad2efe0cb/django/db/models/fields/__init__.py#L710 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
I'm finishing making field tests green. I'm a bit late, because the schedule says that I should finish this job by the end of the previous week. My excuse is that I had exams in the previous week so I couldn't focus on GSoC. Fortunately, I passed all exams and now I can work full time on GSoC and catch up. This week I will finish field tests (there are still 5 red tests and documentation needs to written) and I will start working on model tests. Today I had a conversation with Russell and we decided to simplify some issues. For example, `check_*` methods won't yield errors -- they will just return list of errors. What's more, `check` method will call all `check_*` methods explicitly (now it uses introspection to find all methods starting with `check_`). -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
Hello! I've deleted old `gsoc2013-verification` branch. Follow the new `gsoc2013-checks` branch [1]. [1] https://github.com/chrismedrela/django/tree/gsoc2013-checks What did I do? I've rewritten field tests living in `django.tests.invalid_models` package [2], inside `tests.py` file [3]. I've renamed `invalid_models.invalid_models` submodule into `old_invalid_models`. I've created new `invalid_models` submodule; it contains models required by the rewritten tests. [2] https://github.com/chrismedrela/django/tree/gsoc2013-checks/tests/invalid_models [3] https://github.com/chrismedrela/django/blob/gsoc2013-checks/tests/invalid_models/tests.py#L65 Now I'm working on rewritting field checks. Now all checks are done in `get_validation_errors` function [4]. They will be moved to relevant `Field` subclasses during the current week. I've introduced `Warning` and `Error` classes inside `django.core.checks` package. [4] https://github.com/chrismedrela/django/blob/gsoc2013-checks/django/core/management/validation.py#L22 Now I'm taking exams at university. I will have last exam on Friday (I hope), so next week it will be easier to focus on GSoC and I will work full time. Good-bye! -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
Hello! **Making distinction between form validation and static validation.** I named the process of static checks of apps, models etc. "validation". Unfortunately, novices may confuse it with form validation. So I propose to rename it to "verification". Functions/methods/classmethods (referred to as just functions) starting with `validate_` will start with `verify_` now. And functions discovering all other verification functions will be called `verify` instead of `validate_all`. This is shorter than `validate_` (only 6 letters) and there is no risk of confusing novices. I considered alternatives: "check" is too general and "static_check" as well as "static_validate" are too long. **Disabling/enabling parts of verification** > Hi Christopher, > > Overall, this looks like a great project and I look forward to the more > flexible validation hooks. As the author of django-secure, I do have one > concern with the implementation plan: > > Django's existing validation checks are intended to be run in > development, as a debugging aid, but not in production (to avoid any > slowdown). Thus, by default they are run prior to many management > commands (such as syncdb, runserver), but are generally not run on > production deployments. > > The checks in django-secure, on the other hand, are intended to be run > only under your production configuration, and are mostly useless/wrong > in development. Since runserver doesn't handle HTTPS, most people don't > use HTTPS in development, which means you can't set e.g. > SESSION_COOKIE_SECURE to True in development or your sessions will > break, which means that django-secure check will fail; same goes for > most of the other checks. Running django-secure's checks every time you > syncdb or runserver in development would be at best a waste of time and > at worst extremely annoying. > > So I think the validation framework you build has to incorporate some > kind of distinction between these two types of validation, which have > very different purposes and should be run at different times: > development/debugging validation and production-configuration > validation. I'm not sure exactly what form this distinction should take: > perhaps validators are simply tagged as one type or the other and then > there are two different management commands? > > Interested in your thoughts, > > Carl DEBUG setting is a hint about the environment; if you are working in development environment, then it's likely that DEBUG is set to True; and you have DEBUG set to False in production, I hope. By default, `verify_*` functions will be run only when DEBUG is set to True. django-secure checks will be decorated with new `run_only_in_production` decorator and will run only when DEBUG is set to False. However, I'm not sure if this is the best solution. There are some disadvantages, i. e. I can imagine people saying "what to do if I want to run security checks while DEBUG is set True?". Of course, they can set DEBUG to False, but it's only a workaround. I afraid that it's only tip of the iceberg -- we will probably need more control over which checks to run. An another solution, much more flexible, is to pass around a "filter function" which takes a `verify_*` function and says if the function should be called. A drawback is that this forces developers to add an argument to every `verify_*` function even though they don't want it. In order to avoid that we can introspect the function and check if the function requires `filter` argument and in that case call it without any argument, otherwise call it passing the filter function as `filter` argument, but this is hacky, ugly and non-pythonic; and if anybody wants to rewrite `verify` function (which collect all `verify_` functions and run them), they need to know about that trick and how to use it. So we can split `verify_` functions into two groups: one which does the checks and doesn't require the filter function and another one which finds all `verify_` functions and run them. An example of the former is `verify_upload_to_attribute` of `FieldFile`; and an example of the latter is `verify` function. The latter requires the filter function as an argument. We need to come out with name pattern for the former (I have no idea now, I will propose something tomorrow). This is the most flexible solution, but it increases complexity. I prefer the last, most flexible solution. **Breaking points.** As Russell suggested me during private conversation, I will try to merge my branch as often as possible. That should increase chance of success. There is one obvious breaking points: after finishing revamping validation. Merging while working on the framework shouldn't be too hard. That is because I will refactor bottom-up and keep all code and tests running and because I won't touch any public API. Actually, merging can be done after every week. First question: Django 1.6 is being released now and there is no 1.
[GSoC] Revamping validation framework and merging django-secure once again
Hello! I'm really happy that I've been accepted as a Google Summer of Code student. I will work at revamping validation framework and merging django-secure this summer. I created this thread to start discussion (especially about public API) and get some feedback as well as to share progress. Any comments are welcomed! You can find my proposal as a gist here [1], but I'm going to describe my ideas in short here. [1] https://gist.github.com/chrismedrela/82cbda8d2a78a280a129 My proposal consists of two parts: the first one is revamping validation framework. Don't be confused with form validation - I mean validation of (mainly) apps and models triggered before launching server (or directly by `python manage.py validate` command). The main drawback of the existing framework is that it's monolithic and therefore hard to customize. The second part of my project is merging django-secure using the new refactored framework. That will be a proof that the framework is customizable. Let's start with validation. The validation is done every time you run your Django application. It prevents from misconfigurations (for example missing `upload_to` attribute of FieldFile) or invalid models (a model with a `id` field without `primary_key` set to True). Some apps need custom validation, for example admin app need to check all registered ModelAdmin instances. The new validation framework will cover some other use cases. For example, your apps will be able to check dependencies. Let's start with validating a field. Consider existing FieldFile field. It requires an `upload_to` attribute. How could you check if it's set using the new framework? class FieldFile(Field): (... lots of stuff ...) def validate_upload_to(self): if not self.upload_to: yield Error(obj=self, msg="required 'upload_to' attribute", explanation="Django need to know the directory where uploaded files should be stored.", hint="Add \"upload_to = 'path/to/directory/on/server'\" to model %(model)s in file %(model_file)s.") All methods starting with `validate_` will be called automagically at validation stage. You don't have to trigger it manually. The methods shouldn't have any parameters (excluding `self`). They should return list of errors and warnings or they should yield each of them (like in the example). You can see a new `Error` structure with the following fields: `obj`, `msg`, `explanation` and `hint`. The last two attributes are separated from `msg` in order to force Django developers to give its users really useful, user-friendly error messages with a list of solutions and explanation of a problem. I think that Django should be an intelligent framework that make a conversation with its users. The new framework will introduce a concept of warnings which are really similar to errors expect that they allow you to run server. A warning have the same fields: `obj`, `msg`, `explanation` and `hint`. Warnings will be used while merging django-secure. Not only fields can be validated, but also: models, managers and apps. In case of models, a developer has to write a *class method*, because we validate a model which is a class, not an instance. In case of apps, you can create `validation.py` file which should contain functions. After refactoring validation framework, I will focus on merging django-secure. django-secure is a set of tests improving security of django applications. I won't implement new features, that would be only fitting it to the new framework. I will publish proposal of its new public API in the future. Feel free to comment and criticize! -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC 2013] Revamping validation functionality proposal
> > On Mon, Apr 29, 2013 at 6:25 AM, Christopher Medrela > wrote: > > Jacob, as you said, I deleted django-updates part, but I ended with a > > proposal taking only 9 weeks. Did you mean also scheduling more than 6 > weeks > > for the first part of the proposal (writing the new framework) and > > "stretching" the schedule to 12 weeks? I agree that 6 weeks for writing > the > > framework is slightly too few, but on the other side we have to send > > evaluations after first six weeks, so I think it's a good idea to have > > something that works done. > > Look, it's up to you; my goal is to encourage you to come up with a > plan that's realistically deliverable. Nearly ever summer of code > student so far has overestimated how much they can get done. So I'm > trying to help you come up with a proposal that matches what you can > actually do. Don't forget to take Hofstadter's Law > (http://en.wikipedia.org/wiki/Hofstadter%27s_law) into account! > > Ultimately, we're much more likely to reject an overly-ambitious > proposal than an underly-ambitious one. Our goal is to see you get > done what you promise to get done, so that starts with having > realistic expectations. > > Jacob > I looked deeper into what I have to do and reestimated my proposal. And you're right -- the result is that the first part (introducing the new validation framework) will take more time (about 8 weeks instead of 6). Merging django-secure will take one week longer. The proposal is still available here [1]. I rewritten "schedule and milestone" part -- that was mainly adding more details, so there is no need to review it again. I've published my proposal as a gist. Can I point to it in the submit proposal form? Repasting it and formatting again might be boring job. [1] https://gist.github.com/chrismedrela/82cbda8d2a78a280a129 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC 2013] Revamping validation functionality proposal
> > 2. I think your proposal is a bit too big. I'd general prefer to see a > less ambitious proposal with a high probability of success over a high > risk/high reward. I'd like to see you drop the "django-updates" part > of the proposal, and focus on validation and django-secure. If you end > up with extra time, you can always use it to write more checks. Jacob, as you said, I deleted django-updates part, but I ended with a proposal taking only 9 weeks. Did you mean also scheduling more than 6 weeks for the first part of the proposal (writing the new framework) and "stretching" the schedule to 12 weeks? I agree that 6 weeks for writing the framework is slightly too few, but on the other side we have to send evaluations after first six weeks, so I think it's a good idea to have something that works done. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC 2013] Revamping validation functionality proposal
Thank you for your feedback. I really appreciate every comment because that let me improve my proposal. 1. First of all, I noticed that the license of django-secure is copyright. Google forces us to release code under one of approved license [1], so a question to Carl Meyer: do you mind changing license? On a practicality issue, I'd also like to see one more piece of detail in > your "why me" section. Since your from Poland, English is (I'm guessing) a > second language. Your proposal is very clear and shows you have good > communications skills. However, what we don't know is how long it took you > to write this proposal, and how comfortable you are working in English. If > you're not comfortable working in English in "real time" (e.g., on a spoken > phone call, or in IRC), then that will alter the way that you and your > mentor will interact - you may wish to only communicate via email, for > example; and this may slow down the rate of feedback that you can get from > your mentor. > > 2. Last year and this year I attended FCE conversation class, so my English level isn't worse than FCE. I guess that I'm not ready to speak in real time, but I can talk in real time at IRC. But you know, it's hard to judge your own skills. 1. We've had some discussions about bringing django-secure into core > as part of a more general "checkdeploy" command. The idea being > something you can run shortly before deployment that'd check for all > the stuff that django-secure checks for -- but also more (outdated > dependencies, debug mode, exposed admin, etc). I think this dovetails > nicely with your proposal: it seems that all these "checks" > (validation, deployment, security) could use a single discovery and > running mechanism. I'd love to see you think about modifying your > proposal to include this sort of unification as well as the bringing > of django-secure into core. > > 3. I tried to find discussions about the additional functionality expected from django-secure (you mentioned checking outdated dependencies, debug mode and exposed admin), but I didn't succeed. Could you point me to any discussion on this mailing list or in the ticket tracker or anywhere else? I'd possibly add one additional point - the potential for confusion between > validation of the type you're describing, and "model validation". This > isn't a problem of your making - the ambiguity already exists in Django. > However, if we're adding API points on fields, which already support the > idea of "validators", we need to be careful we don't confuse issues more. > To that end, I'd be interested in seeing at least some initial thoughts on > how we can structure the API or change the naming so that this issue is > controlled. > > 4. It looks like changing validate into validate_all solves the problem. We also have to emphasise the difference between those two mechanisms in documentation. 5. I said that I won't have any trip during GSoC, but I forget about my holiday after September 6. This is not backpacking-trip, I will live in a hotel with net access, but I won't be able to work full time. I hope that you won't disqualify my proposal on this basis -- that can be considered as an advantage because I will be highly motivated to finish before time. 6. I've updated my proposal. I'm not repasting it here -- that would be too much mess. If you want to see the new version, look at the gist [2]. Changes to the proposal in short: - *Rewriting the second part of the proposal*. The proposal is not finished, it lacks the "improving django-secure" part. - *Changing schedule of the first half* -- removing "write documentation" week. - *Minor changes.* Changing solution -> hint. Changing validate -> validate_all. Changed app validation rules: the file validation.py won't be created by default for new apps; if the file exists but it misses validate_all or validate_models functions, then the default ones are used. [1] http://opensource.org/licenses/ [2] https://gist.github.com/chrismedrela/82cbda8d2a78a280a129 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[GSoC 2013] Revamping validation functionality proposal
I would like to participate in Google Summer of Code this year. I've written proposal for "revamping validation functionality" idea. It's available as a gist: https://gist.github.com/chrismedrela/82cbda8d2a78a280a129 . Below, I'm pasting the proposal. Feel free to criticize it. *Table of content* 1. *Abstract* 1.1 Drawbacks of the existing framework 1.2 Goals 1.3 Benefits 2. *The new framework *2.1 Overview 2.2 Advantages 3. *New features *3.1 django-secure 3.2 django-update 4. *Schedule and milestones* 5. *Who am I?* *1. Abstract* *1.1 Drawbacks of the existing framework* Django currently has a validation framework, but there are a lot of problems with it. First of all, it is monolithic and developers cannot write custom validation (see [#16905]) or modify the existing functionality (see [#12674]); validation lives in a few functions like django.core.management.validation.get_validation_errors [1] or django.contrib.admin.validation.validate [2]. The validation functionality is not separated from other stuff like printing found errors during validating models in get_validation_errors or registering models in admin app [3] (see [#8579]); it is sometimes done during first call to an important method, i. e. CurrentSiteManager [4] is validated in its get_queryset [5] method. [#12674] https://code.djangoproject.com/ticket/12674 [#16905] https://code.djangoproject.com/ticket/16905 [1] https://github.com/django/django/blob/master/django/core/management/validation.py#L22 [2] https://github.com/django/django/blob/master/django/contrib/admin/validation.py#L14 [3] https://github.com/django/django/blob/master/django/contrib/admin/sites.py#L52 [#8579] https://code.djangoproject.com/ticket/8579 [4] https://github.com/django/django/blob/master/django/contrib/sites/managers.py#L5 [5] https://github.com/django/django/blob/master/django/contrib/sites/managers.py#L38 There are few tests of the validation framework and it is not easily testable because validation functions return concatenated error messages instead of list of errors (see django.tests.invalid_models.invalid_models.models [6]). It also lacks some features like warnings (see [#19126]). Due to this disadvantages lots of apps do not have any validation, i. e. they do not check inter-app dependencies. [6] https://github.com/django/django/blob/master/tests/invalid_models/invalid_models/models.py#L366 [#19126] https://code.djangoproject.com/ticket/19126 *1.2 Goals* This proposal is about revamping current validation framework. First of all, we need to write more tests and rewrite existing ones. Then we need an consistent API of validating different kinds of objects like models, fields, managers or whole apps so it will be easy to add new kind of object. Validation functionality should be separated from other stuff and it should minimize dependencies. We should allow developers to add validation to their apps and any other kind of objects, so custom validation is a must. We will not break backward compatibility. This proposal is not only about refactoring but also introducing new features. Introducing warnings is first feature that allows us to add security warnings based on django-secure [7] app. The second main feature is making switching to newer version of Django easier by developing django-update app which should inspect settings as well as models and predict and warn about some problems. [7] https://github.com/carljm/django-secure *1.3 Benefits* There are a lot of benefits. Cleaning code, removing unwanted dependencies and adding more tests are the most obvious ones. We will also benefit from long term solution which will be easy to maintain since it is extendable. Switching Django version will be piece of cake. We will improve security of Django projects thanks to security warnings. This also implies that Django will be considered as a safe and secure framework. Better opinion is always desired. *2. The new framework* *2.1 Overview* The API is based on Honza Kral idea from his patch [8]. An developer can add new validation functionality by writing a callable piece of code. It will be automatically called during validating whole project (triggered by python manage.py validate) and it must fulfill the following contract: it has no arguments and returns a list of warnings and errors or yields each of them. [8] https://github.com/HonzaKral/django/compare/master...ticket-12674 The validated object may be a model, a manager, a field or an app. In case of a field or a manager, the callable piece of code is a method. In case of a model, it is a classmethod. In case of an app, we will put functions in validation.py file. Let's see an example:: class FieldFile(File): > (lots of stuff) > > def validate_upload_to_attribute(self): > if not self.upload_to: > yield Error(obj=self, msg="required 'upload_to' attribute", > explanation="Django need to know the dir
Re: Ticket #17093 -- quarantine global state in django.template
Hello, I made all tests green finally (all commits are tested under Python 2.7.3 and 3.2.3). The problem that took a few days to solve was that the default engine should be recomputed after changing TEMPLATE_LOADERS setting since its _template_source_loaders internal needs to be recalculated. I tried to solve it by adding receiver in template/base.py and then circular-imports problem occured. I tried to solve it (that took a few days). Fortunately, I asked apollo13 and he helped me and told me that I should put the receiver to test/signals.py so I did it. I also updated master which is important since the hierarchy of test files was restructured. I kept old branch and created new called ticket_17093v3 [new-branch]. The patch is ready to first review (however it still lack documentation). After gathering all pieces of global state into one, I started to do some clean up. I noticed that the cache loader unnecessary procrastinates resolving loaders list, so I moved the resolving to __init__. Now there is still a lot of mess in the template system. The mess is about passing on origin of template and some APIs. For example, TemplateEngine.find_template and BaseLoader.load_template return tuple instead of a template. The second element of the tuple is display_name which is necessary to compute origin of template (and the origin is computed in many places! [computing-origin]). The first element may be not a Template instance but... its source. My plan is that TemplateEngine.find_template and BaseLoader.load_template will return only a Template instance (and no display_name!). The Template will be created in BaseLoader.load_template and that will be place for computing origin of template. That will be much more simpler API than the current one. I tried to do this, but there are some problems. The main problem is somehow buggy (in my opinion) behaviour of TemplateEngine.find_template [buggy-find-template]. If a loader raises TemplateDoesNotExist, then we catch this exception and check next loader. However, this is not what should be done. Look at the case when the template exists, but it includes a missing template. Including missing template raises the exception and it shouldn't be caught by find_template. The problem occured in many tickets [tickets] but the patches didn't solve the problem -- they're only a workarounds in my opinion. I think that find_template should distinguish why the TemplateDoesNotExist was raised (because the current template is missing or because the current template exists but it includes some missing templates). The simplest solution is to ensure that the TemplateDoesNotExist is created with template name as its argument (with the exception of TemplateEngine.select_template case). Then find_template should check if the template name in exception is equal to `name` argument. If it's not true then reraise the exception loudly. Otherwise make it silent. I partially did the changes (however, I didn't pushed it into the branch), and it works. My question is if the changes of API are acceptable since they are backward incompatible. The changes includes: TemplateDoesNotExist (added template_name argument in constructor), TemplateEngine.find_template and BaseLoader.load_template (and its subclasses)(changed return values). Now I would like to ask about GSOC. Last time I told about my idea of introducing rendererers. Now I think that it doesn't introduce many improvements -- now you can use Django REST Framework (as Tom said) for JSON or you can just write your own render_to_response delegating to mako template system. So I dropped this idea. Now I wonder what are priorities? - Continuing refactoring (and adding new features) of template system. Better error reporting. Logging invalid variable lookups. Compiling Django templates into python functions so they will render faster. - Refactoring another part of Django. Is there any wiki page about how Django should be refactored and what principles should I base on when I restructure Django? - Improving error reporting. - Revamping of validation functionality. - Test framework cleanup. Speeding up tests. Splitting long tests migrated from doctests. Testing Django application "in a void" (independently of Django project and everything else). Running all tests against different databases backends in sequence? Please tell me which of these are most interesting for you (and which one have the greates chance to be a GSoC project). Which ones are important and which ones negligible? The list of ideas from 2012 doesn't contain information which ones are priorities. I would like to know to focus on the most important first. [new-branch] https://github.com/chrismedrela/django/tree/ticket_17093v3 [computing-origin] Run `git grep -n make_origin` [buggy-find-template] https://github.com/chrismedrela/django/blob/ticket_17093v3/django/template/base.py#L1362 [t
Re: Ticket #17093 -- quarantine global state in django.template
Hello. I would like to report progress of my work (not too much new things, rather cleaning and polishing), discuss some issues and propose an idea for next Google Summer Of Code. Progres from last time: - Solved problem with TemplateResponse -- it receives optional keyword argument called `engine` (if omitted, then it calls get_default_engine). During pickling the engine is lost. During repickling get_default_engine() is called. - Dropped idea of invalidating caches: base.invalid_var_format_string and context._standard_context_processors. These settings are used in about a dozen places in tests and it's not really necessary now. - Renamed _Template back to Template. The Template class accepts optional keyword argument called `engine`. If omitted, then it calls get_default_engine(). I've also rebased commits so the branch forks from updated master. I've reordered and squashed commits and rewritten description of commits so all work ended in less then twenty well-described commits. I tried to make all tests green before next weekend (Django Sprint is coming!), but I failed. There are some failing tests (exactly 450 xD); they are heisenbugs -- they appear only when you run all tests; if you run only one test suit, then they disappear. I guess that the problem is that the default_engine should be invalidated between some tests. I also bisected commits and I found out that the commit introducing failures is [1]. I will try to fix these problems before the sprint on Saturday, because the current state is almost ready to review (and to merge, I hope) and I would like to have something that can be reviewed at the weekend. I will sum up what I've done. I've not introduced new features. I've gathered all pieces of global state into one (the default_engine). I've introduced TemplateEngine class and I've rewritten most tests so they create their own TemplateEngine instances instead of using the global one. I tried not to introduce backward incompatibilities. I've rewritten history so in order not to break existing links I won't use ticket_17093 branch any more. Instead, see ticket_17093v2 branch [1]. [1] https://github.com/chrismedrela/django/commit/ed578d64fff8c8eb58898548d5ef1c0815c25f24 [2] https://github.com/chrismedrela/django/tree/ticket_17093v2 > I agree that it doesn't mean less global state. But there is good global > state and bad global state. For example, we could make DjangoApplication > class to avoid global state but there is no need. Global settings is OK > since we don't need the ability to create more than one django > application instance. Even tests doesn't require more than one instance > at the same time, so we can just reinitialize the application when there > is need for that. That's done by change_settings decorator and it works > well. And there is bad global state, like current template system. ... > I didn't know about Jinja2 and that it's more suitable for use outside > Django. So it seems like the only goal is refactoring -- we need to > quarantize global state, but not dependencies on settings. We also don't > need to split the template system into django-dependent and > django-independent parts. > I'm not sure the distinction between "good global state" and "bad global state" is so clear, or even exists. A major part of the motivation for #17093 was that it could be a step along the path towards a global-state-free Django (using something like a DjangoApplication or ApplicationConfig object), which would allow running multiple Django sites in a single process (which is indeed something that some people, though not most, need). This is mentioned in the ticket description. > If we are not working towards that larger goal (and I'd love to hear other opinions on whether we should be or not), and external library use of Django templates is not a strong motivator, and the testing issues are manageable via override_settings, then I think the rationale for #17093 begins to look rather weak. No-global-state would be a better from-scratch design, certainly, but will it bring enough concrete benefit now to be worth the transition? OK, I've changed my mind. I agree that the goal of refactoring is to introduce something like DjangoApplication class whose instances will own components like default_engine. So everything will be global-state-free. However, it would be annoying if you had to pass the application instance everywhere and type "trains" like "request.application.query(MyModel).objects.all()". It would break backward compability in almost every line of code. So there must be one global instance of DjangoApplication. OK, let mi introduce the idea for Google Some Of Code of this year. The idea is stolen from Pyramid [3]. The main new concept is a renderer. A renderer may be current TemplateEngine as well as JSON serializer or mako renderer. When I'm typing a view I would like to wri
Re: Ticket #17093 -- quarantine global state in django.template
Hello. I'm really astonished how fast I get feedback and I really appreciate that. I apologize that you had to wait for my reply so long, but I cannot push myself to stop programming and to share results. :) I did some progress since creating this topic: - Almost all tests doesn't rely on the default engine, including tests of custom tag tests. That excludes tests in response.py (this is problematic one) and loaders.py:RenderToStringTest which tests render_to_string function; this function doesn't delegate to TemplateEngine instance in a simple way, so it should be tested. - The template engine is passed to loaders now, so I was able to restore caching in cache-loader. - I also did some small things like python 3.x support, making default_engine lazy (by using get_default_engine instead of default_engine) and adding unicode_literals imports. By the way, I added also absolute_import to almost every file. Is that desirable? I don't want to discourage you from working on the project if you find > it interesting and informative, and it may well be that the results are > worth the effort and end up being merged into Django; but it is also > possible that we look at the patch and say "sorry, that's just too much > disruption for the benefits" - so you should be prepared for that > possibility. > I would like to avoid the situation like that, so what is the source of that disruption, apart from backward incompatibility? I really don't have a good sense of where the other core devs stand on how much this type of deep refactoring is worth. I would like also to know the opinion of other core devs, so... Core devs, is it worth? > 14. Is this plan good? What issues do you see? Could you look at my work > > and review it (I wonder especially if I'm using a python feature not > > supported by python version supported by Django?). Best way to notice this is to fix up the test suite into a running > state, and run it with the supported Python versions (minimum is 2.6 > currently, the more likely area for problems is Python 3, since Django > master now supports Python 3). That's the approach I adopted -- red-green refactor loop. I tested under 2.7.3 and 3.2.3 python, all tests pass. > 7. Then there is the issue of loaders. At the first, I thought that > passing the engine will be not necessary if I change behaviour of > loader.BaseLoader.load_template method so it will always return source > of template instead of Template instance. So I did it for BaseLoader. > Few days ago I did the same for loaders.cached.Loader, but I realised > that this change means no caching at all -- the loader cached compiled > templates. Now it caches nothing. The current architecture let us to > cache template sources but not compiled templates. > > 8. There are two solutions. First, we can just pass the engine to every > loader. Second, we can remove cache-loader and implement caching inside > TemplateEngine. Both solutions are ugly. Do you see any better one? I think loaders will need to have access to the template engine that is using them (though for backwards-compatibility they may need to default to default_engine if not given another engine). There was chicken and egg situation. You had to provide list of loaders to create an template engine instance and to create a loader you had to provide an engine. I solved this issue by introducing two-step creating engine. First you create an template engine, then create a loader (and pass on the engine), eventually call engine.set_loaders(...). This is not the simplest one, but it works. It also made possible solving problem with custom inclusion tags. > 3. The goal is to write TemplateEngine class to encapsulate global state. > For backward-compatibility, there will be one global instance of > TemplateEngine and all functions like loader.get_template should delegate > to the instance. Tests need to be rewritten. > A global instance is probably unavoidable, but by itself doesn't result in > less global state - it just wraps it all in one global object. Functions > that need Engine state should be passed an engine instance explicitly > wherever possible - not importing default_engine at module level. I agree that it doesn't mean less global state. But there is good global state and bad global state. For example, we could make DjangoApplication class to avoid global state but there is no need. Global settings is OK since we don't need the ability to create more than one django application instance. Even tests doesn't require more than one instance at the same time, so we can just reinitialize the application when there is need for that. That's done by change_settings decorator and it works well. And there is bad global state, like current template system. I think that Preston Holmes made it plain so I will cite him: The shim point for all of these IMO should be your get_default_engine > fu
Ticket #17093 -- quarantine global state in django.template
Hello everybody. 1. I'm working on ticket #17093 [1]. You can see progress of work at github [2]. It's about rewritting django.template so it won't use global state. I wrote this post to tell you what I did so far, what issues I deal with and to discuss them. I hope that we will gain valuable experience in splitting Django into smaller parts. 2. This package has two kinds of global state. The first one are global attributes defined at module level of template package. These are base.builtins, base.libraries, loader.template_source_loaders and base.templatetags_modules. The second one are dependencies on settings and some other module-level attributes in template package like base.invalid_var_format_string (this is cache; it's a flag which means "is there '%s' in settings.TEMPLATE_STRING_IF_INVALID?") or context._standard_context_processors, which is cache set in context.get_standard_processors function which transform data from settings in some way. I decided to focus on the first kind of global state at first. 3. The goal is to write TemplateEngine class to encapsulate global state. For backward-compatibility, there will be one global instance of TemplateEngine and all functions like loader.get_template should delegate to the instance. Tests need to be rewritten. 4. What I did so far? I created base.TemplateEngine class and base.default_engine global instance. Some functions (mostly from loader module: get_template, find_template, select_templates and from base module: add_to_builtins, get_library, compile_string) delegates to appropriate methods of default_engine. I also partially rewrote tests (only tests/regressiontests/templates/tests.py) so they don't use default_engine directly. 5. The one issue is that the engine is needed everywhere. For example, consider creating base.Template instance. At the time of creation the template source is being compiled. The template may load a library, so the compilation cannot be done without list of all libraries defined in the engine. To sum up, we cannot just type Template(source), because this depends on template engine. I solved this issue by passing engine instance to Template constructor. In order not to break everything, I renamed Template class to _Template and created function called Template that does "return _Template(default_engine, **kwargs)". 6. The same issue occures when you consider including another template. loader_tags.IncludeNode needs to be able to load another template. I resolved this by passing an engine to base.Parser constructor. A parser passes itself to constructors of nodes, so IncludeNode can load another template in this way: "parser.engine.find_template(...)". 7. Then there is the issue of loaders. At the first, I thought that passing the engine will be not necessary if I change behaviour of loader.BaseLoader.load_template method so it will always return source of template instead of Template instance. So I did it for BaseLoader. Few days ago I did the same for loaders.cached.Loader, but I realised that this change means no caching at all -- the loader cached compiled templates. Now it caches nothing. The current architecture let us to cache template sources but not compiled templates. 8. There are two solutions. First, we can just pass the engine to every loader. Second, we can remove cache-loader and implement caching inside TemplateEngine. Both solutions are ugly. Do you see any better one? 9. Now, there is another issue. Changing API. It's obvious that changing base.Parser signature is OK since it's an internal structure. It's also easy to notice that base.Template is used everywhere so we cannot change API of Template. But there are many places where it's unclear if a method or class is public (it means that we promise we won't change its signature) or internal. For example, I don't know how much we can change loaders. I don't know if Django developers types "isinstance(obj, Template)" -- it won't work know. I don't know if I can change signature of SsiNode as I did. By the way, I think that we should make clear distinction between these two methods by using underscore convention. 10. As I said I'm trying to rewrite tests from tests.py file so they won't use the default engine. There left some sources of dependency on default engine. One of them are include tags in tests which load templates at the time of importing module [3] (or the source of problems is in base.Library.inclusion_tag?). Another one are template.response module [4] and all other test files (i. e. callables.py, custom.py). I didn't dig into these issues deeply yet since I wanted to publish my results and take some feedback. 11. A small issue is test regressiontests.views.tests.debug.DebugViewTests.test_template_loader_postmortem. The problem is in method django.views.debug.ExceptionReporter.get_traceback_data. It uses template_source_loaders global state which was removed. We c
