Re: Django's build: Try Earthly?

[Vlad A. Ionescu]

Hi Tom,

I believe that a lot of what django-docker-box achieves is what Earthly was
created for too. Now that you already have django-docker-box, it’s possible
that you might not need Earthly. Or at least not for the purpose of
reproducing CI integration testing locally, specifically.

In any case, here’s how Earthly *might* help. Emphasis on might, as I have
limited understanding of the complete Django dev ecosystem - I’m pretty
much shooting in the dark here :-)

   - Beyond CI integration testing, if there are other CI scripts, Earthly
   can help containerize those too in order to achieve reproducibility. It can
   help debug builds faster on your own computer, rather than wait for CI to
   kick in again and again.
   - If django-docker-box is placed in Earthly, then Earthly could run
   these tests isolated and in parallel. e.g. Run Python 3.7 tests in a docker
   compose stack distinct and isolated from a Python 3.8 tests stack.
   Similarly for different DB versions. Every matrix tuple combination can be
   its own isolated execution that runs in Earthly in parallel.
   - In terms of yaml spaghetti: it may be possible to decouple some of the
   integration tests into distinct docker-compose yaml files which are
   executed separately through Earthly. It may or may not be beneficial to
   trade-off yaml spaghetti with some Earthfile scripting to help tame the
   complexity. On one end of the spectrum you could get rid of all the
   docker-compose scripting and use docker run inside Earthly. On the other
   end of the spectrum, you can just run compose stacks in Earthly itself as
   we support that pretty well. Elixir’s Phoenix framework, for
example uses docker
   run‘s for DB waiting, docker compose for dependencies and mix (Elixir’s
   build tool) for running the tests. This is what this looks like in
   practice.
   
   .
   - In general, when migrating from one CI vendor to another, migrating to
   Earthly first helps abstract away the CI. Earthly runs the same on either
   Jenkins or GitHub Actions, or your own laptop. This helps with having a
   common CI language that works the same regardless of the vendor.
   - Compute intensive steps (if there are any) can be cached and not
   repeated if the inputs have not changed. In Earthly this can work
with a remote
   cache  too.
   - Earthly’s multiplatform features could run tests on other
   architectures too (e.g. arm64), in parallel (uses QEMU underneath).

If any of this feels interesting, I’d be happy to dig deeper and put
together a demo PR to POC a particular benefit.

Cheers,
Vlad.

On Thu, Mar 18, 2021 at 3:19 PM Tom Forbes  wrote:

> Hey!
> I've recently been trying to improve our local/CI experience
> 
> so this is quite a nice and timely message!
>
> I really like the look of Earthly and it has some uses at $WORK that I'm
> excited about, but for Django the most annoying thing is the large build
> matrix
> 
> rather than a complex dockerfile
> . I
> really dislike docker-compose for this, but I can't see anything in Earthly
> that would help manage or reduce this yaml-spaghetti?
>
> Tom
>
> On Wed, Mar 17, 2021 at 8:10 PM Vlad A. Ionescu  wrote:
>
>> Hi Django devs,
>>
>> Nick Pope directed me to this list. Wondering if anyone is interested in
>> trying https://github.com/earthly/earthly for Django's build.
>>
>> Earthly could help with reproducing CI failures locally (via containers),
>> testing against multiple versions of Python in parallel, or for migrating
>> between CI vendors.
>>
>> It works well on top of Jenkins and GH Actions.
>>
>> Happy to put together a demo PR if this is of interest. Let me know!
>>
>> Cheers,
>> Vlad.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/CAO_OUmM%2BMFU1nPMzFaEsGcD4rh6tFh-wQ3T_hK4GH5-4%2BCxgxQ%40mail.gmail.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/BXxm2h-9ECg/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an 

Re: Django's build: Try Earthly?

[Tom Forbes]

Hey!
I've recently been trying to improve our local/CI experience

so this is quite a nice and timely message!

I really like the look of Earthly and it has some uses at $WORK that I'm
excited about, but for Django the most annoying thing is the large build
matrix

rather than a complex dockerfile
. I
really dislike docker-compose for this, but I can't see anything in Earthly
that would help manage or reduce this yaml-spaghetti?

Tom

On Wed, Mar 17, 2021 at 8:10 PM Vlad A. Ionescu  wrote:

> Hi Django devs,
>
> Nick Pope directed me to this list. Wondering if anyone is interested in
> trying https://github.com/earthly/earthly for Django's build.
>
> Earthly could help with reproducing CI failures locally (via containers),
> testing against multiple versions of Python in parallel, or for migrating
> between CI vendors.
>
> It works well on top of Jenkins and GH Actions.
>
> Happy to put together a demo PR if this is of interest. Let me know!
>
> Cheers,
> Vlad.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAO_OUmM%2BMFU1nPMzFaEsGcD4rh6tFh-wQ3T_hK4GH5-4%2BCxgxQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJM1SEpJiJdsq0AFmOefZFRBPmaDXZseE3e4J7QALw%2BsKw%40mail.gmail.com.

Re: Bug in migrations when testing on Django 3.2b1

[Carlton Gibson]

Looking at the history for the migrations code 
, the 
was 
https://github.com/django/django/commit/110001d0bbbabe2a5b57b14a59bd0e4b71bf2712#diff-e7df880bdc17c719e0332fa0cfbd4eff49bd481f638e34335a6311cfcd0ebc26
 
recently which adjusted AppConfigStub, so maybe related. 
(You could see if reverting that helps.)

On Thursday, 18 March 2021 at 15:39:04 UTC+1 Carlton Gibson wrote:

> Hi. 
>
> So, first off, thanks for testing!
>
> Initial thoughts: 
>
> * We might need a bit more to be able to reproduce — are you able to 
> narrow down the problem? 
> * "Which migration it is varies between runs" — that sounds fun 
> * "Changing AppConfig.name..." — The AppConfig loading was reworked, it 
> could be that:  
> https://docs.djangoproject.com/en/dev/releases/3.2/#automatic-appconfig-discovery
>
> Can I ask you to get as close as you can to directions for a reproduce and 
> open a ticket on Trac?
> https://code.djangoproject.com/newticket
>
> If you can't get to a reproduce, it's still worth opening a ticket with as 
> much info as you can:
>   * Maybe someone is able to say "Oh, it's this"
>   * Provides info for anyone else hitting this. 
>
> Thanks again! 
>
> Kind Regards,
>
> Carlton
>
>
> On Thursday, 18 March 2021 at 15:26:23 UTC+1 HM wrote:
>
>> Changing AppConfig.name to just "app" leads to ModuleNotFoundError: No 
>> module named 'app'. 
>>
>> Changing the path in INSTALLED_APPS to "prefix.app.apps.AppConfig" 
>> instead leads to the same NodeNotFoundError as before. 
>>
>> On Thu, 18 Mar 2021 at 15:06, Hanne Moa  wrote: 
>> > 
>> > I have some migrations that runs/tests fine on Django 3.0 and 3.1, but 
>> > not on 3.2b1. 
>> > 
>> > There's a specific app whose migrations fail with: 
>> > 
>> > django.db.migrations.exceptions.NodeNotFoundError: Migration 
>> > prefix_app.000N_fooFoo dependencies reference nonexistent parent node 
>> > ('app', '000M_bar'). 
>> > 
>> > Which migration it is varies between runs, and note that the first app 
>> > label looks different than the second. Is there some new way to 
>> > auto-generate app labels that leads to this? 
>> > 
>> > In INSTALLED_APPS the app in question is by path: "prefix.app". In the 
>> > AppConfig, "name" is identical to the path, "prefix.app", while 
>> > "label" is "prefix_app". The models do not have an explicit tablename 
>> > set and in the database they have been created with 
>> > "app_modelnameinlowercase", not "prefix_app_modelnameinlowercase". 
>> > 
>> > 
>> > HM 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4424f465-3643-4a39-ac9b-f52bacbe48a8n%40googlegroups.com.

Re: Bug in migrations when testing on Django 3.2b1

[Carlton Gibson]

Hi. 

So, first off, thanks for testing!

Initial thoughts: 

* We might need a bit more to be able to reproduce — are you able to narrow 
down the problem? 
* "Which migration it is varies between runs" — that sounds fun 
* "Changing AppConfig.name..." — The AppConfig loading was reworked, it 
could be that: 
 
https://docs.djangoproject.com/en/dev/releases/3.2/#automatic-appconfig-discovery

Can I ask you to get as close as you can to directions for a reproduce and 
open a ticket on Trac?
https://code.djangoproject.com/newticket

If you can't get to a reproduce, it's still worth opening a ticket with as 
much info as you can:
  * Maybe someone is able to say "Oh, it's this"
  * Provides info for anyone else hitting this. 

Thanks again! 

Kind Regards,

Carlton


On Thursday, 18 March 2021 at 15:26:23 UTC+1 HM wrote:

> Changing AppConfig.name to just "app" leads to ModuleNotFoundError: No
> module named 'app'.
>
> Changing the path in INSTALLED_APPS to "prefix.app.apps.AppConfig"
> instead leads to the same NodeNotFoundError as before.
>
> On Thu, 18 Mar 2021 at 15:06, Hanne Moa  wrote:
> >
> > I have some migrations that runs/tests fine on Django 3.0 and 3.1, but
> > not on 3.2b1.
> >
> > There's a specific app whose migrations fail with:
> >
> > django.db.migrations.exceptions.NodeNotFoundError: Migration
> > prefix_app.000N_fooFoo dependencies reference nonexistent parent node
> > ('app', '000M_bar').
> >
> > Which migration it is varies between runs, and note that the first app
> > label looks different than the second. Is there some new way to
> > auto-generate app labels that leads to this?
> >
> > In INSTALLED_APPS the app in question is by path: "prefix.app". In the
> > AppConfig, "name" is identical to the path, "prefix.app", while
> > "label" is "prefix_app". The models do not have an explicit tablename
> > set and in the database they have been created with
> > "app_modelnameinlowercase", not "prefix_app_modelnameinlowercase".
> >
> >
> > HM
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3f0f2034-62a8-4886-8e0d-f010e20c19b2n%40googlegroups.com.

Re: Bug in migrations when testing on Django 3.2b1

[Hanne Moa]

Changing AppConfig.name to just "app" leads to ModuleNotFoundError: No
module named 'app'.

Changing the path in INSTALLED_APPS to "prefix.app.apps.AppConfig"
instead leads to the same NodeNotFoundError as before.

On Thu, 18 Mar 2021 at 15:06, Hanne Moa  wrote:
>
> I have some migrations that runs/tests fine on Django 3.0 and 3.1, but
> not on 3.2b1.
>
> There's a specific app whose migrations fail with:
>
> django.db.migrations.exceptions.NodeNotFoundError: Migration
> prefix_app.000N_fooFoo dependencies reference nonexistent parent node
> ('app', '000M_bar').
>
> Which migration it is varies between runs, and note that the first app
> label looks different than the second. Is there some new way to
> auto-generate app labels that leads to this?
>
> In INSTALLED_APPS the app in question is by path: "prefix.app". In the
> AppConfig, "name" is identical to the path, "prefix.app", while
> "label" is "prefix_app". The models do not have an explicit tablename
> set and in the database they have been created with
> "app_modelnameinlowercase", not "prefix_app_modelnameinlowercase".
>
>
> HM

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CACQ%3Drrf6gLTENCnoiUmuM7KAwBefbGD7Db4LU1ume2bbEODfWg%40mail.gmail.com.

Django 3.2 release candidate 1 released.

[Carlton Gibson]

Details are available on the Django project weblog: 

https://www.djangoproject.com/weblog/2021/mar/18/django-32-rc1/ 


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/146303B6-65EE-40DE-84E3-C5D955159F64%40gmail.com.

Bug in migrations when testing on Django 3.2b1

[Hanne Moa]

I have some migrations that runs/tests fine on Django 3.0 and 3.1, but
not on 3.2b1.

There's a specific app whose migrations fail with:

django.db.migrations.exceptions.NodeNotFoundError: Migration
prefix_app.000N_fooFoo dependencies reference nonexistent parent node
('app', '000M_bar').

Which migration it is varies between runs, and note that the first app
label looks different than the second. Is there some new way to
auto-generate app labels that leads to this?

In INSTALLED_APPS the app in question is by path: "prefix.app". In the
AppConfig, "name" is identical to the path, "prefix.app", while
"label" is "prefix_app". The models do not have an explicit tablename
set and in the database they have been created with
"app_modelnameinlowercase", not "prefix_app_modelnameinlowercase".


HM

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CACQ%3DrreGyzuJ2wuJPsdcFA_yUckFKU%3DvQouAcm4k2F2GTK1r6g%40mail.gmail.com.