Re: slow migrations

[Peter Baumgartner]

Reporting back on some additional findings for what it's worth.
SmileyChris dumped and recreated the project in question's migrations,
manually ordering them to minimize dependencies. It was not a major
reduction in total migrations: 82 to 58 (mostly third-party migrations
and initials), but the time to create an empty migration dropped
massively. It went from 260s on 1.8.6 (160s on 1.9.1) to 2s on both
versions.

I'm not sure what to make of it, but the slowness I was seeing doesn't
seem to be as much related to number of migrations, but perhaps the
contents of the migrations files. Is it possible that having lots of
cross-app migration dependencies could be the cause?

Thanks as always!

-- Pete

On Thu, Jan 7, 2016 at 3:12 PM, Carl Meyer  wrote:
> On 01/07/2016 03:03 PM, Aymeric Augustin wrote:
>> As far as I understand, the CPU cost comes from generating a full set of
>> model classes for each step of the migration history. That’s consistent
>> with the profile sent by Florian.
>>
>> I usually end up throwing away the migration history and regenerating a
>> new set of migrations when I get to that point. This requires truncating
>> the django_migrations table manually and faking the new set of migrations.
>>
>> If the project doesn’t use data migrations, squashmigrations may achieve
>> the same effect. Sadly real-life projects tend to have data migrations
>> whose only purpose is to run once in production. They prevent full
>> squashing.
>
> FWIW, I've also done a hybrid of these two options, where I generate
> fresh initial migrations rather than actually using squashmigrations
> (for the same reason, to avoid problems with data migrations), but then
> I still keep the old migrations around for a transition period and use
> the `replaces` attribute (the same one added automatically by
> `squashmigrations`) on the new initial migrations. Then later (once the
> new migrations are deployed everywhere) delete the old migrations and
> the `replaces` attr.
>
> Effectively this is similar to what you're doing, it just takes
> advantage of the `replaces` feature to avoid manually fiddling with the
> migrations table on each deployment.
>
> If I (or anyone else) ever gets around to it, I think
> https://code.djangoproject.com/ticket/24109 could make the actual
> squashmigrations command usable for real projects, by letting you just
> mark certain migrations as "please ignore when squashing".
>
> Carl
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/568EE2B2.1040501%40oddbird.net.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAAoxf4sacLi4faHx0qXuP3WF8ZXT-4nU9qcub_QGDBaHJ_em%3DA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Vote on Jira as bugtracker

[Joe Tennies]

I'm not a lawyer, but I care about licensing.

The MIT License would allow you to relicense it, but you must keep the
original copyrights in tact. (From license: Redistribution and use in
source and binary forms, *with or without modification*, are permitted...)

It does have a list of "buts" too.

You must keep the copyright and disclaimer in the source files you used,
and your documentation must also do that.

Typical answer is to add a header above it in the source and then state
that all modifications from the original are AGPL... Original code is:
leave theirs alone.

You typically add "Portions of the code are based on Django Rest Framework
which is *paste copyright and lack of warranty etc disclaimer*"
 to your documentation.

On Thu, Jan 7, 2016 at 7:12 PM, Carl Meyer  wrote:

> Hi Andrey,
>
> On 01/07/2016 04:23 PM, Andrey Antukh wrote:
> > On Fri, Jan 8, 2016 at 12:42 AM, Florian Apolloner
> > The DRFv2 (as DRFv3) as far as I know is licensed using MIT permissive
> > license that does not impide take the source and re-license it under
> > other license.
> >
> > taiga.io  initially have started using the DRF just
> > like a dependency but over time the dev-team found the needs monky-patch
> > many and many parts of it... Later, the dev-team have decided just
> > include it in the code base and remove useless (for taiga) code and add
> > additional features.
> >
> > Much of code/improvements is contributed back to the DRF (v2), other as
> > third party packages and some other is no, because the approaches are
> > diverged and the changes are just to much opinionated/coupled to the
> > taiga usage.
> >
> > I hope I have solved your question.
>
> I am not a lawyer, but it seems to me the problem is not the use of the
> code (which the license does allow), but the re-licensing and the lack
> of the original DRF license in your codebase.
>
> You are allowed to use the code and re-distribute it, but AFAIK you are
> not allowed to re-license it unless you are the copyright owner. And the
> terms of the DRF license do require that the DRF license text be
> preserved along with any redistribution of the DRF code, which (as far
> as I can see) Taiga currently doesn't do.
>
> So if I'm right (which I may well not be), I think the issue is easily
> resolved by including the original DRF license text in your base/api/
> directory, and clarifying that that license (not the AGPL) is the
> license that applies to all code taken from DRF.
>
> Carl
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/568F0D05.3050901%40meyerloewen.net
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Joe Tennies
tenn...@gmail.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CACiOJ6sFEP%3D-i3z8ge3JDXm7%3Dn0xDgg3bOE%2BjHJDJruuvz8ONw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: re-thinking middleware

[Ryan Hiebert]

Sent from my iPhone

> On Jan 7, 2016, at 20:50, Carl Meyer  wrote:
> 
> Hi all,
> 
> Back at the Django Under the Hood sprints in November, Florian and I
> started batting around some ideas for fixing some of the problems with
> the existing middleware abstraction. Florian put together an initial
> pull request with a POC for some of those ideas. I've now updated the
> proposal (incorporating some implementation ideas from Pyramid's
> middleware equivalent) and written it up in the form of a DEP. You can
> find the DEP at
> 
> https://github.com/django/deps/blob/master/draft/0005-rethinking-middleware.rst
> 
> and I'm also pasting the full text below, to ease in-line commenting via
> email.
> 
> Comments welcome!

Only one comment from me. This is awesome!

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7E2126FA-0178-4F4F-B8AB-F49B9BD7B684%40ryanhiebert.com.
For more options, visit https://groups.google.com/d/optout.

re-thinking middleware

[Carl Meyer]

Hi all,

Back at the Django Under the Hood sprints in November, Florian and I
started batting around some ideas for fixing some of the problems with
the existing middleware abstraction. Florian put together an initial
pull request with a POC for some of those ideas. I've now updated the
proposal (incorporating some implementation ideas from Pyramid's
middleware equivalent) and written it up in the form of a DEP. You can
find the DEP at

https://github.com/django/deps/blob/master/draft/0005-rethinking-middleware.rst

and I'm also pasting the full text below, to ease in-line commenting via
email.

Comments welcome!


DEP 0005: Re-thinking middleware


:DEP: 0005
:Author: Carl Meyer
:Implementation Team: Florian Apolloner
:Shepherd: Carl Meyer
:Status: Draft
:Type: Feature
:Created: 2016-01-07
:Last-Modified: 2016-01-07

.. contents:: Table of Contents
   :depth: 3
   :local:


Abstract


The existing Django "middleware" abstraction suffers from an absence of
strict layering and balanced in/out calls to a given middleware. This
DEP proposes an improved abstraction for wrapping the request cycle in
strictly layered pre-view and post-view actions.


Motivation
==

In theory, and per `the documentation`_, ``process_request`` will be
called for each incoming request, ``process_response`` will be called
for each outgoing response, and ``process_exception`` will be called in
case of an uncaught exception.

This description seems to imply the invariant that if
``process_request`` is called, either ``process_response`` or
``process_exception`` will later be called on that same middleware in
that same request cycle. Django itself has in the past included
middleware (the now-defunct ``TransactionMiddleware``) that implicitly
relied on this invariant.

In fact, due to the short-circuiting and exception-handling behavior of
various middleware methods, this invariant does not hold. It is possible
for a middleware to have its ``process_request`` method called, but then
never see its ``process_response`` or ``process_exception`` called for
that request (e.g. in case of an uncaught exception in a "later"
middleware method).

It is also possible for a middleware to never see its
``process_request`` method called for a given request (because an
earlier middleware's ``process_request`` returned a response), but still
have its ``process_response`` or ``process_exception`` method called on
that response.

This lack of strict in/out layering makes it impossible to safely
implement some types of middleware (such as ``TransactionMiddleware``),
and requires verbose defensive programming: e.g. even if
``process_request`` sets a certain attribute on the request,
``process_response`` on that same middleware can't assume that that
attribute will be present on the request it receives.

This is the primary problem that this DEP intends to solve.

.. _the documentation:
https://docs.djangoproject.com/en/stable/topics/http/middleware/


Acknowledgment
==

The proposed API in this DEP is modelled on Pyramid's `Tween`_ concept
(the author and implementor of this DEP developed a very similar idea
independently at a Django sprint before reading about Tweens).

.. _Tween:
http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/hooks.html#registering-tweens

Specification
=

This DEP introduces a new setting, ``MIDDLEWARE``, which contains an
ordered list of dotted paths to middleware factories.

A middleware factory can be written as a function that looks like this::

def simple_middleware(get_response):
# one-time configuration and initialization

def middleware(request):
# code to be executed for each request before
# the view is called

response = get_response(request)

# code to be executed for each request/response after
# the view is called

return response

return middleware

Or it can be written as a class with a ``__call__`` method, like this::

class SimpleMiddleware(object):
def __init__(self, get_response):
self.get_response = get_response

# one-time configuration and initialization

def __call__(self, request):
# code to be executed for each request before
# the view is called

response = self.get_response(request)

# code to be executed for each request/response after
# the view is called

return response

In prose instead of examples: a middleware factory is a callable that
takes a ``get_response`` callable and returns a middleware. A middleware
is a callable that takes a ``request`` and returns a ``response``. (Just
like a view! Turtles all the way down!)

The ``get_response`` callable provided by Django might be the actual
view (if this is the last listed middleware), or it might be the next
middleware in 

Re: Vote on Jira as bugtracker

[Carl Meyer]

Hi Andrey,

On 01/07/2016 04:23 PM, Andrey Antukh wrote:
> On Fri, Jan 8, 2016 at 12:42 AM, Florian Apolloner
> The DRFv2 (as DRFv3) as far as I know is licensed using MIT permissive
> license that does not impide take the source and re-license it under
> other license. 
> 
> taiga.io  initially have started using the DRF just
> like a dependency but over time the dev-team found the needs monky-patch
> many and many parts of it... Later, the dev-team have decided just
> include it in the code base and remove useless (for taiga) code and add
> additional features. 
> 
> Much of code/improvements is contributed back to the DRF (v2), other as
> third party packages and some other is no, because the approaches are
> diverged and the changes are just to much opinionated/coupled to the
> taiga usage.
> 
> I hope I have solved your question.

I am not a lawyer, but it seems to me the problem is not the use of the
code (which the license does allow), but the re-licensing and the lack
of the original DRF license in your codebase.

You are allowed to use the code and re-distribute it, but AFAIK you are
not allowed to re-license it unless you are the copyright owner. And the
terms of the DRF license do require that the DRF license text be
preserved along with any redistribution of the DRF code, which (as far
as I can see) Taiga currently doesn't do.

So if I'm right (which I may well not be), I think the issue is easily
resolved by including the original DRF license text in your base/api/
directory, and clarifying that that license (not the AGPL) is the
license that applies to all code taken from DRF.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/568F0D05.3050901%40meyerloewen.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: Vote on Jira as bugtracker

[Andrey Antukh]

HI Florian

On Fri, Jan 8, 2016 at 12:42 AM, Florian Apolloner 
wrote:

> Hi Yamila,
>
> On Thursday, January 7, 2016 at 2:22:21 PM UTC+1, Yamila Moreno wrote:
>>
>> I just wanted to let you know that me or anyone in our team would be glad
>> to answer any question about the product, saas or installation, migrating
>> issues or anything related. No strings attached, of course!!!
>>
>
> I personally do actually have one question. I was looking through your
> source code and realized that you are apparently importing DRF and changing
> a few parts and relicensing it as AGPL. I am not a lawyer and you might
> actually be allowed to do that, but what about contributing to DRF back
> instead? Your AGPL code (again I am not a lawyer and might be wrong),
> actually forbids Tom to take your changes back into DRF… That said, the DRF
> license requires you to retain the copyright information, either I missed
> it or you did not retain it.
>

The DRFv2 (as DRFv3) as far as I know is licensed using MIT permissive
license that does not impide take the source and re-license it under other
license.

taiga.io initially have started using the DRF just like a dependency but
over time the dev-team found the needs monky-patch many and many parts of
it... Later, the dev-team have decided just include it in the code base and
remove useless (for taiga) code and add additional features.

Much of code/improvements is contributed back to the DRF (v2), other as
third party packages and some other is no, because the approaches are
diverged and the changes are just to much opinionated/coupled to the taiga
usage.

I hope I have solved your question.

Regards.
Andrey

>
> Do not get me wrong, but if half what I wrote is true; I'd personally be
> very worried if Django switched to Taiga, given the signal that this would
> send to our userbase (might be just me though).
>
> Cheers,
> Florian
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/4f189a3c-9011-4b06-994f-ec4b1a346c9c%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Andrey Antukh - Андрей Антух - 
http://www.niwi.nz
https://github.com/niwinz

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAOQnsABOv9-o8h1Cstv4UU4SEFpebaiQoJf3ADA9PdX1QaiQsQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Vote on Jira as bugtracker

[Russell Keith-Magee]

Weighing in on Jira specifically - I’ve had to use it on a number of
projects, and I’ve never had a good experience with it. That might be
because the Jira instances were badly configured - bit if that’s the case,
it suggests to me that there’s a deeper problem with Jira being too complex
for it’s own good.

On the more broad topic of adopting a new ticket system - historically,
I’ve been very much of the same opinion as James. All ticket trackers suck.
Trac sucks in ways we understand.

However, I think the integration between Trac and Github is sufficiently
kludgey that it’s at least worth looking at options. If we’re going to
switch (and it’s a *BIG* if),

* Open source, and written in Django.
* Able to support our workflow, and customisable to adapt to other
workflows if we need to
* Able to integrate with our authentication mechanisms
* Able to deal with our “everyone can contribute” permissions structure
(more on this below)
* *Completely* seamless integration with the Github PR workflow.

On Thu, Jan 7, 2016 at 10:18 PM, Yamila Moreno  wrote:

> Hi,
>
>
>> Are you already familiar with Trac and the way we use it?
>>
>
> I'm familiar with trac (we were using it some year ago), and I'm a bit
> familiar with django workflow because I attended a couple of sprints.
> Anyway, I'm not an expert in trac.
>
>
>>
>> Key things are:
>>
>> * integration with GitHub (tickets, login)
>>
>
> Yes, we have.
> - Everything is based in a documented API, so with webhooks you can get
> new issues in github automatically created in your instance of taiga.io,
> as well as comments
> - Login with github is a contrib plugin, check it here:
> https://github.com/taigaio/taiga-contrib-github-auth
> 
> ; there are also other contrib plugins to login with other platforms:
> https://github.com/agolden/taiga-contrib-google-auth
> 
> https://github.com/fedora-infra/taiga-contrib-fas-openid-auth
> 
>
>
>
>> * integration with Django Project login
>>
>
> It's not developed, but it should be easy to create a new contrib plugin
> for your instance
>
>
>> * integration with site styling
>>
>
> Yes!
> https://taigaio.github.io/taiga-doc/dist/#themes
> 
>
>
>> * workflow
>>
>
> The filosophy behind Taiga is the agile frame (Individuals and
> interactions over processes). Thus, workflow is completely free to set; in
> a fresh project, administrators can configure the workflow, statuses, name
> and colours; which of them are "open statuses" and which one are "closed
> statuses".
>

Related to out workflow is permissions. This is one of the biggest reasons
why Github Issues doesn’t work for us. We need to be able to give access to
*anyone* to update issues. This doesn’t just mean adding comments - it
means progressing a ticket along the workflow, adding flags (or whatever
categorisation mechanism exists), even closing issues.

Being able to identify members of the core team when they participate in
conversations is highly desirable, but not absolutely essential. This is a
feature that we’ve added to Trac.

Providing the ability for core team members to “lock” certain issues would
be a nice-to-have that we don’t currently have.

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAJxq84_eeorkYZmWwm6F4H18pQ7rVy%3D5vAGsv%2BTqv-dK7PFQ%3DA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Vote on Jira as bugtracker

[Chris Cogdon]

I used Jira at my previous company. It is a great tool, but it is 
_extremely_ heavyweight. Unless you need to high level of customisation of 
workflows and integrations it can provide, and have someone intimately 
familiar with it and/or have the nearly-full-time job of learning and 
fiddling with it [1], it will be an alienating, frustrating experience for 
all involved.


[1] Just like about every Java-based tool out there. ;)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/92a03cbc-fb14-4f89-ad5d-45ab77fcb207%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Feature Request] Orderable ArrayAgg and StringAgg in contrib.postgres.aggregates

[Josh Smeaton]

Seems reasonable enough to me. Expressions already support generating an 
ORDER BY clause by calling .asc() or .desc() on them. That'd allow your 
proposed API to support:

Model.objects.aggregate(ArrayAgg(some_field, order_by=F('some_field').asc()
))

Or any other expression. Consider that any ordering added within *may* need 
to be contributed to GROUP BY, but I haven't read the documentation to say 
that is a requirement.

Feel free to open a ticket in Trac to track the feature. If you're also up 
for implementing it (..and testing.. and documenting..) I'm quietly 
confident that it'll be fairly straight forward to add, given familiarity 
with Expressions API.

Additionally, if you think json_agg and json_object_agg should be 
implemented, you can open a feature ticket on Trac for those too. 

Regards,

On Friday, 8 January 2016 00:46:52 UTC+11, Floris den Hengst wrote:
>
> The excellent ArrayAgg and StringAgg Postgres-specific aggregates were 
> introduced in contrib.postgres in Django 1.9 and I've been quite happy 
> using them here and there.
> Thanks for the keeping Django awesome!
>
> The documentation 
>  of 
> Postgres 9.0 first mentions the possiblity of ordering the results within 
> such aggregations.
> This could be useful in some cases.
> For example: it could make sense to perform the StringAgg in 
> lexicographical order in some cases.
>
> The simple format of ordering the aggregation result in SQL is quite 
> simple:
> SELECT ARRAY_AGG(some_field ORDER BY some_field ASC) FROM table;
> SELECT ARRAY_AGG(some_field ORDER BY some_field DESC) FROM table;
> SELECT ARRAY_AGG(some_field ORDER BY other_field ASC) FROM table;
>
> It would be nice if the above would be supported as follows:
> Model.objects.aggregate(ArrayAgg(some_field, order_by='some_field'))
> Model.objects.aggregate(ArrayAgg(some_field, order_by='-some_field'))
> Model.objects.aggregate(ArrayAgg(some_field, order_by='other_field'))
> where order_by is an optional parameter. If it not specified, behavior can 
> remain unchanged from the current implementation.
>
> Note that the documentation for Postgres >= 9.3 also mentions 
> json_agg ordering and Postgres >= 9.4 mentions json_object_agg as well.
> I'm unsure whether these should be included as well as I couldn't find the 
> matching implementation in the ORM atm and have no knowledge on plans in 
> that direction.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/310b45a6-d1b2-4241-96ce-60912a62c562%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Vote on Jira as bugtracker

[Florian Apolloner]

Hi Yamila,

On Thursday, January 7, 2016 at 2:22:21 PM UTC+1, Yamila Moreno wrote:
>
> I just wanted to let you know that me or anyone in our team would be glad 
> to answer any question about the product, saas or installation, migrating 
> issues or anything related. No strings attached, of course!!!
>

I personally do actually have one question. I was looking through your 
source code and realized that you are apparently importing DRF and changing 
a few parts and relicensing it as AGPL. I am not a lawyer and you might 
actually be allowed to do that, but what about contributing to DRF back 
instead? Your AGPL code (again I am not a lawyer and might be wrong), 
actually forbids Tom to take your changes back into DRF… That said, the DRF 
license requires you to retain the copyright information, either I missed 
it or you did not retain it.

Do not get me wrong, but if half what I wrote is true; I'd personally be 
very worried if Django switched to Taiga, given the signal that this would 
send to our userbase (might be just me though). 

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4f189a3c-9011-4b06-994f-ec4b1a346c9c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: slow migrations

[Carl Meyer]

On 01/07/2016 03:03 PM, Aymeric Augustin wrote:
> As far as I understand, the CPU cost comes from generating a full set of
> model classes for each step of the migration history. That’s consistent
> with the profile sent by Florian.
> 
> I usually end up throwing away the migration history and regenerating a
> new set of migrations when I get to that point. This requires truncating
> the django_migrations table manually and faking the new set of migrations.
> 
> If the project doesn’t use data migrations, squashmigrations may achieve
> the same effect. Sadly real-life projects tend to have data migrations
> whose only purpose is to run once in production. They prevent full
> squashing.

FWIW, I've also done a hybrid of these two options, where I generate
fresh initial migrations rather than actually using squashmigrations
(for the same reason, to avoid problems with data migrations), but then
I still keep the old migrations around for a transition period and use
the `replaces` attribute (the same one added automatically by
`squashmigrations`) on the new initial migrations. Then later (once the
new migrations are deployed everywhere) delete the old migrations and
the `replaces` attr.

Effectively this is similar to what you're doing, it just takes
advantage of the `replaces` feature to avoid manually fiddling with the
migrations table on each deployment.

If I (or anyone else) ever gets around to it, I think
https://code.djangoproject.com/ticket/24109 could make the actual
squashmigrations command usable for real projects, by letting you just
mark certain migrations as "please ignore when squashing".

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/568EE2B2.1040501%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: slow migrations

[Andrew Godwin]

Yes, it's basically a fundamental design flaw of having the migrations
represented this way - it makes autodetection and code generation very
accurate, but at the expense of calculation time. There is some
optimisation work that can be done to try and avoid building intermediate
states, but it's a pretty complex code optimisation problem.

I think it's an acceptable flaw providing you take the time to squash every
so often, and I prefer it to South's flaw of missing dependencies out and
making tables incorrectly sometimes, but I can understand that it's kind of
ridiculously slow sometimes.

Andrew

On Thu, Jan 7, 2016 at 2:03 PM, Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> As far as I understand, the CPU cost comes from generating a full set of
> model classes for each step of the migration history. That’s consistent
> with the profile sent by Florian.
>
> I usually end up throwing away the migration history and regenerating a
> new set of migrations when I get to that point. This requires truncating
> the django_migrations table manually and faking the new set of migrations.
>
> If the project doesn’t use data migrations, squashmigrations may achieve
> the same effect. Sadly real-life projects tend to have data migrations
> whose only purpose is to run once in production. They prevent full
> squashing.
>
> --
> Aymeric.
>
> On 7 janv. 2016, at 22:36, Florian Apolloner 
> wrote:
>
> Okay, my base time is currently 95 seconds, where we are spending >60
> seconds in ModelBase.__new__. I've attached a picture of the profile run,
> there are not really any obvious spots, so shaving time off there will be
> hard I fear.
>
> Cheers,
> Florian
>
> On Thursday, January 7, 2016 at 9:49:11 PM UTC+1, Pete wrote:
>>
>> It looks like there are some other fixes in 1.9 that weren't covered
>> by my monkeypatch. I upgraded the project and included a new cProfile
>> in the ticket https://code.djangoproject.com/ticket/22608#comment:23.
>> It shaved 100s off, but still takes 2.5 minutes to create an empty
>> migration.
>>
>> Florian, I'll contact you off-list about the project.
>>
>> -- Pete
>>
>> On Thu, Jan 7, 2016 at 1:11 PM, Florian Apolloner 
>> wrote:
>> > Also, is there any chance that I can get access to this project to
>> profile a
>> > little bit more and identify some hotspots?
>> >
>> > On Thursday, January 7, 2016 at 9:10:26 PM UTC+1, Florian Apolloner
>> wrote:
>> >>
>> >> Just looking at the topmost three entries:
>> >> _expire_cache spends 18 second in delattr, can you try replacing that
>> with
>> >> del self.__dict__[cache_key]. Maybe also try moving self.__dict__ in a
>> local
>> >> variable since it will get referenced a lot
>> >> subclass_exception seems awfully slow, maybe there are some creative
>> ways
>> >> to speed it up?
>> >> It would also be interesting to know where the entries from
>> collections.py
>> >> come and replace it with more efficient structures.
>> >>
>> >> The rest is as Tim said.
>> >>
>> >> Cheers,
>> >> Florian
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups
>> > "Django developers (Contributions to Django itself)" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an
>> > email to django-develop...@googlegroups.com.
>> > To post to this group, send email to django-d...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/django-developers.
>> > To view this discussion on the web visit
>> >
>> https://groups.google.com/d/msgid/django-developers/7b687031-050f-4de3-854c-afd6cbd00ca5%40googlegroups.com.
>>
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/b2af60ae-c071-4f7c-94e7-7e3c2adb8e63%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
> 
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web 

Re: slow migrations

[Aymeric Augustin]

As far as I understand, the CPU cost comes from generating a full set of model 
classes for each step of the migration history. That’s consistent with the 
profile sent by Florian.

I usually end up throwing away the migration history and regenerating a new set 
of migrations when I get to that point. This requires truncating the 
django_migrations table manually and faking the new set of migrations.

If the project doesn’t use data migrations, squashmigrations may achieve the 
same effect. Sadly real-life projects tend to have data migrations whose only 
purpose is to run once in production. They prevent full squashing.

-- 
Aymeric.

> On 7 janv. 2016, at 22:36, Florian Apolloner  wrote:
> 
> Okay, my base time is currently 95 seconds, where we are spending >60 seconds 
> in ModelBase.__new__. I've attached a picture of the profile run, there are 
> not really any obvious spots, so shaving time off there will be hard I fear.
> 
> Cheers,
> Florian 
> 
> On Thursday, January 7, 2016 at 9:49:11 PM UTC+1, Pete wrote:
> It looks like there are some other fixes in 1.9 that weren't covered 
> by my monkeypatch. I upgraded the project and included a new cProfile 
> in the ticket https://code.djangoproject.com/ticket/22608#comment:23 
> . 
> It shaved 100s off, but still takes 2.5 minutes to create an empty 
> migration. 
> 
> Florian, I'll contact you off-list about the project. 
> 
> -- Pete 
> 
> On Thu, Jan 7, 2016 at 1:11 PM, Florian Apolloner  > wrote: 
> > Also, is there any chance that I can get access to this project to profile 
> > a 
> > little bit more and identify some hotspots? 
> > 
> > On Thursday, January 7, 2016 at 9:10:26 PM UTC+1, Florian Apolloner wrote: 
> >> 
> >> Just looking at the topmost three entries: 
> >> _expire_cache spends 18 second in delattr, can you try replacing that with 
> >> del self.__dict__[cache_key]. Maybe also try moving self.__dict__ in a 
> >> local 
> >> variable since it will get referenced a lot 
> >> subclass_exception seems awfully slow, maybe there are some creative ways 
> >> to speed it up? 
> >> It would also be interesting to know where the entries from collections.py 
> >> come and replace it with more efficient structures. 
> >> 
> >> The rest is as Tim said. 
> >> 
> >> Cheers, 
> >> Florian 
> > 
> > -- 
> > You received this message because you are subscribed to the Google Groups 
> > "Django developers (Contributions to Django itself)" group. 
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to django-develop...@googlegroups.com . 
> > To post to this group, send email to django-d...@googlegroups.com 
> > . 
> > Visit this group at https://groups.google.com/group/django-developers 
> > . 
> > To view this discussion on the web visit 
> > https://groups.google.com/d/msgid/django-developers/7b687031-050f-4de3-854c-afd6cbd00ca5%40googlegroups.com
> >  
> > .
> >  
> > 
> > For more options, visit https://groups.google.com/d/optout 
> > . 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-developers@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/b2af60ae-c071-4f7c-94e7-7e3c2adb8e63%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1A91799C-A133-4F71-8FBE-2451F619CE5F%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: slow migrations

[Florian Apolloner]

Also, is there any chance that I can get access to this project to profile 
a little bit more and identify some hotspots?

On Thursday, January 7, 2016 at 9:10:26 PM UTC+1, Florian Apolloner wrote:
>
> Just looking at the topmost three entries:
> *_expire_cache* spends 18 second in delattr, can you try replacing that 
> with del self.__dict__[cache_key]. Maybe also try moving self.__dict__ in a 
> local variable since it will get referenced a lot
> *subclass_exception *seems awfully slow, maybe there are some creative 
> ways to speed it up?
> It would also be interesting to know where the entries from collections.py 
> come and replace it with more efficient structures.
>
> The rest is as Tim said.
>
> Cheers,
> Florian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7b687031-050f-4de3-854c-afd6cbd00ca5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: slow migrations

[Tim Graham]

As far as I know, no one is actively working on this area at the moment.

There is at least one open issue:
https://code.djangoproject.com/ticket/22608

You can look through all open migrations tickets too:
https://code.djangoproject.com/query?status=assigned=new=Migrations=Accepted=id=summary=status=owner=type=version=1=id

On Thursday, January 7, 2016 at 2:07:24 PM UTC-5, Pete wrote:
>
> Hi there, I know this has been discussed in the past, but I'm not sure 
> where things stand today regarding the the speed of migrations. On 
> non-trivial projects, it's common to hear about migrations that take 
> 5-10 minutes to run. I've even heard reports about them taking over an 
> hour. 
>
> I know about a couple issues [1] that have been closed around 
> optimizing the process, but it still feels like there's a serious 
> issue here. The project I'm working on has ~80 migrations and ~300 
> tables. It uses Django 1.8 and I backported the 1.9 optimization [2] 
> to test it. It's still ~5 minutes to run a makemigrations or migrate 
> command. For those coming from South and previous Django versions, 
> this is a major degradation. It makes development pretty painful and 
> regular testing untenable. 
>
> Here is the cProfile of a makemigrations command: 
> https://gist.github.com/ipmb/e65159f7315b9645841b 
>
> Is there an issue open for this at the moment? Is there any other info 
> I can provide to help? 
>
> Thanks! 
>
> [1] https://code.djangoproject.com/ticket/23745 
> https://code.djangoproject.com/ticket/24743 
> [2] 
> https://github.com/django/django/commit/5aa55038ca9ac44b440b56d1fc4e79c876e51393
>  
>
> -- Pete 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/145681d1-55d7-433c-9ef5-dd45d931d2e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Vote on Jira as bugtracker

[Yamila Moreno]

Hi,  


> Are you already familiar with Trac and the way we use it? 
>

I'm familiar with trac (we were using it some year ago), and I'm a bit 
familiar with django workflow because I attended a couple of sprints. 
Anyway, I'm not an expert in trac.
 

>
> Key things are: 
>
> * integration with GitHub (tickets, login) 
>

Yes, we have. 
- Everything is based in a documented API, so with webhooks you can get new 
issues in github automatically created in your instance of taiga.io, as 
well as comments
- Login with github is a contrib plugin, check it here: 
https://github.com/taigaio/taiga-contrib-github-auth 

 
; there are also other contrib plugins to login with other platforms:
https://github.com/agolden/taiga-contrib-google-auth 

https://github.com/fedora-infra/taiga-contrib-fas-openid-auth 


 

> * integration with Django Project login 
>

It's not developed, but it should be easy to create a new contrib plugin 
for your instance
 

> * integration with site styling 
>

Yes!
https://taigaio.github.io/taiga-doc/dist/#themes 

 

> * workflow 
>

The filosophy behind Taiga is the agile frame (Individuals and interactions 
over processes). Thus, workflow is completely free to set; in a fresh 
project, administrators can configure the workflow, statuses, name and 
colours; which of them are "open statuses" and which one are "closed 
statuses".
 

>
> (I expect I've missed something.) 
>
> What scope is there for automated migration? 
>

Right now there isn't an automated migration from trac, it's in the backlog 
(https://tree.taiga.io/project/taiga/us/108) but it's not planned in the 
short term. Nevertheless, it could be done with the Taiga API 
(http://taigaio.github.io/taiga-doc/dist/api.html) or with the import 
export feature.
 

>
> And perhaps users familiar both with Taiga and our Trac might be able to 
> comment too? 
>

It would be very interesting to read these comments. 

Best,
Yamila

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/05772af1-2f40-4ee9-a7e6-3b3ba6f101b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Vote on Jira as bugtracker

[Daniele Procida]

On Thu, Jan 7, 2016, Yamila Moreno  wrote:

>I'm Yamila, part of Taiga Team and Kaleidos (the company that develops 
>Taiga.io). We're very happy to be as a "completely hypothetical" candidate 
>for a new bugtracker. Taiga (a Django and open source project) being used 
>by django-dev would be huge :)
>
>I just wanted to let you know that me or anyone in our team would be glad 
>to answer any question about the product, saas or installation, migrating 
>issues or anything related. No strings attached, of course!!!

Thanks Yamila.

Are you already familiar with Trac and the way we use it?

Key things are:

* integration with GitHub (tickets, login)
* integration with Django Project login
* integration with site styling
* workflow

(I expect I've missed something.)

What scope is there for automated migration? 

And perhaps users familiar both with Taiga and our Trac might be able to 
comment too?

Daniele

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20160107134625.2110950811%40mail.wservices.ch.
For more options, visit https://groups.google.com/d/optout.

Re: Vote on Jira as bugtracker

[Yamila Moreno]

Hi there, 

I'm Yamila, part of Taiga Team and Kaleidos (the company that develops 
Taiga.io). We're very happy to be as a "completely hypothetical" candidate 
for a new bugtracker. Taiga (a Django and open source project) being used 
by django-dev would be huge :)

I just wanted to let you know that me or anyone in our team would be glad 
to answer any question about the product, saas or installation, migrating 
issues or anything related. No strings attached, of course!!!

Best,
Yamila

On Thursday, January 7, 2016 at 12:55:26 AM UTC+1, Tim Graham wrote:
>
> To search Trac, use "site:code.djangoproject.com your query" in a Google 
> search box. Works great in my experience.
>
> On Wednesday, January 6, 2016 at 6:42:44 PM UTC-5, Josh Smeaton wrote:
>>
>> FWIW I actually like Jira (much more than Trac) and find it a lot easier 
>> to use. I think the trick is configuring very basic workflows so users 
>> don't have to fight through transitions. Open, Closed, Assigned/In Progress 
>> and transitions to and from each state would get us really close to the 
>> current Trac workflow. My number 1 gripe with Trac is that search SUCKS, so 
>> I'd actually be in favour of a migration *if someone else were to do all 
>> the work* :D. But that's the rub isn't it, nothing comes for free. We'd 
>> also lose login with Github (I think) and psuedo-anonymous triage because 
>> Jira requires an email account, so there's no way it could be a full parity 
>> transition.
>>
>> Taiga looks very nice, but arguments made above also apply. There's a 
>> cost in setup + migration, overhead of learning a new system, and a lack of 
>> knowledge about the problems that will evidently exist.
>>
>> I really do think Trac is awful though, just wanted to be clear about 
>> that.
>>
>> On Thursday, 7 January 2016 03:34:13 UTC+11, Victor Sosa wrote:
>>>
>>>
>>> Looks like it is a NO to the proposition.
>>>
>>> Daniele
>>>
>>> I like what I saw in taiga, that's a way better bug tracking UI; you can 
>>> check here:
>>>
>>> https://tree.taiga.io/project/taiga/issues?page=1
>>>
>>> On Wednesday, January 6, 2016 at 12:18:13 PM UTC-4, Daniele Procida 
>>> wrote:

 On Wed, Jan 6, 2016, Daniele Procida  wrote: 

 >By all means it's useful to get votes on something like this, even 
 >before we consider those questions, because if enough people want 
 >something it's always possible - but be aware that simply getting lots 
 >of votes for it would only ever be the first and easiest step. 

 While we're in the realm of the completely hypothetical, if I were 
 going to find myself enthusiastic about moving to a new development 
 tracking platform, it would be Taiga.  

 It's written in Django, by a company that actively supports the Django 
 community. It's open-source.  The people who develop it are approachable 
 and friendly. It has a nice name. 

 Daniele 



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b4b8d31a-605f-4ae8-b8d7-372965de4f69%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.