Re: Making startproject's settings more 12-factor-y

[Kit La Touche]

Wow, `distutils.util.strtobool` is great to know about!

So, can we refocus this conversation? This is starting to look like
previous conversations on this topic, which pull in a lot of possibilities
but don't lead to a change. How do we go about generating a DEP or other
consensus-building tool on what we want here?

It seems to me this conversation has historically gotten stuck by trying to
bite off a bigger bite. Therefore, I would recommend a minimal change that
gestures towards the direction we want to explore.

Personally, I think that *at minimum* providing Django-builtin "get from
env"  helpers would be great; beyond that, I'd love to have them be
included around `DEBUG` and `SECRET_KEY` with the current values as
defaults, so they're optional. Once we see how this gets used, we can see
about passing it a file instead of `os.environ`, or borrowing other ideas
from any of the various supporting projects that have been suggested.

It's clear that different people have different use-cases and different
needs, but regardless, I think that it's clear also that including values
like DEBUG and SECRET_KEY as *hard coded values in settings* by default
does not point people towards good practices. What "good practices" are is
likely to differ in each person's case, but I think that suggesting one
option (again, my vote is "look in the environment") will at least help
newer devs understand that this is a topic they should learn more about.

Thanks,

--Kit

On Thu, Jun 25, 2020 at 11:16 AM Javier Buzzi 
wrote:

> Hi Bobby, yes, thank you, this looks around the line of what i would like
> us to implement in Django.
>
> Side note: i saw this config('DEBUG', default=False, cast=bool) and
> thought "there is NO WAY that works", that led me to from distutils.util
> import strtobool, absolute mind blown! Thanks!
>
> -Buzzi
>
> On Thursday, June 25, 2020 at 1:03:19 PM UTC-4, Bobby Mozumder wrote:
>>
>> There’s also python-decouple that I use that I haven’t seen mentioned in
>> this thread. It lets you set specific environment variables in a separate
>> .env file or INI file: https://github.com/henriquebastos/python-decouple
>>
>> -bobby
>>
>> On Jun 25, 2020, at 4:47 AM, Javier Buzzi  wrote:
>>
>> Hey Tom, cool project haven't heard of it, looks to me more inline with
>> validating and converting user input from an api/form. I could really see
>> myself using this in my personal projects, however this looks like we'd be
>> going back to the class based configuration that im trying to avoid.
>> Nonetheless thank you for the share!
>>
>> - Buzzi
>>
>> On Thursday, June 25, 2020 at 4:34:11 AM UTC-4, Tom Carrick wrote:
>>>
>>> Javier, I just wanted to point out another option for configuration:
>>> pydantic <https://pydantic-docs.helpmanual.io/> - it offers a very
>>> slick and intuitive interface for settings management across environments,
>>> seamless handing of environment variables by using type hints, and so on. I
>>> wouldn't recommend it for anything other than large sites with complex
>>> configurations, but it does work well for those, once you grapple with how
>>> to integrate it with django's settings so they're all exposed as
>>> `settings.FOO`, and so on.
>>>
>>> I don't think I would want to integrate anything like this into Django
>>> proper, but it might deserve a mention in the documentation.
>>>
>>> Tom
>>>
>>> On Wed, 24 Jun 2020 at 23:52, Javier Buzzi  wrote:
>>>
>>>> This makes sense, I have a project that has a lot of settings files
>>>> that get activated depending on the value of DJANGO_SETTINGS_MODULE. The
>>>> solution i outlined above takes your reservations under consideration, if
>>>> you want to use it, great, if not also great -- its a supplement not a
>>>> requirement.
>>>>
>>>> - Buzzi
>>>>
>>>> On Wednesday, June 24, 2020 at 5:24:35 PM UTC-4, Dan Davis wrote:
>>>>>
>>>>>  tMost of the world is not as seamless as heroku.  My DevOps won't
>>>>> give me any more than a handful of environment variables.  I wanted
>>>>> something like DATABASE_URL, but all I have is DJANGO_LOG_DIR and
>>>>> DJANGO_SETTINGS_MODULE, and so I need many, many settings files. I think
>>>>> that happens a lot, and maybe a common pattern.
>>>>>
>>>>> From a 12factor perspective, I would like to get it down to local
>>>>> settings (development) and production settings - yet for a lot of users,
>>>>> DevOps is not really suppo

Re: Making startproject's settings more 12-factor-y

[Kit La Touche]

Those are valid concerns, and I want to especially come back to "all of
this has to be optional, which seems to rule out making it part of the
starting project template".

But first, I want to reiterate that what's vital to me (and this may be
only my concern, of course!) is that newer developers, especially folks
without an ops team and without ops experience, have a clearer path to
deployment. Partly this is selfish: I want Django to grow and flourish, and
I think that giving new folks a good experience with it their first time
will facilitate that. Folks with an ops team can already be assumed to have
people to talk with and work with to negotiate and understand their
particular deployment context.

So, in light of that, coming back to "has to be optional", I agree—I very
much agree. However, I think it also has to be *visible*. That was my
motivation for, in my proposed PR, including it in the default
`settings.py` template, but making sure defaults were set to the values
they would be without the addition of `from_env`. I'm not sure at all that
that's the best way, but that was my thought process! "Optional" means a
few things, and I think that there's a difference between "setting env vars
should be optional" and "using `from_env` should be optional" and "having
`from_env` ever be in your `settings.py` should be optional", and I'd love
to clarify which part of that spectrum most closely aligns to your intent.

Cheers,

--Kit

On Wed, Jun 24, 2020 at 3:24 PM Dan Davis  wrote:

>  tMost of the world is not as seamless as heroku.  My DevOps won't give me
> any more than a handful of environment variables.  I wanted something like
> DATABASE_URL, but all I have is DJANGO_LOG_DIR and DJANGO_SETTINGS_MODULE,
> and so I need many, many settings files. I think that happens a lot, and
> maybe a common pattern.
>
> From a 12factor perspective, I would like to get it down to local settings
> (development) and production settings - yet for a lot of users, DevOps is
> not really supporting a full PaaS-like experience any way.
>
> So - all of this has to be optional, which seems to rule out making it
> part of the starting project template.  For sure, I've got my personal
> template, and work has an on-premise template and a Cloud template as well
> - but the department of developers doesn't always use these.  I find
> databases containing the tables for other projects, long after the models
> and migrations are gone, indicating a start by copy mode.
>
> On Wed, Jun 24, 2020 at 1:35 PM Kit La Touche  wrote:
>
>> Carlton—thanks very much for the feedback. Javier—likewise. In
>> particular, the imagined API you describe above is very appealing to me:
>> start with `from_env` and then if you learn more about this and want to,
>> add in some `EnvFileLoader`.
>>
>> I want to make clear my motivation and agenda here: I have recently had
>> some conversations with newer devs about their experiences with deployment
>> of apps they're working on, and with a friend at Heroku about his informal
>> research into the problems people have with the same. One recurring
>> friction point (and this is not just on Heroku at all, to be clear) is that
>> there are a number of things that people *don't know they need to
>> configure* for a working deployment.
>>
>> There are four settings that are recurring particular gotchas that people
>> miss: the secret key, debug, static files, and databases. Static files
>> seems big and out of scope, databases seems adequately handled by
>> dj-database-url for most cases, and if your case is more complex, you'll
>> learn it, but the other two (secret key and debug) seemed easy enough to
>> flag as "you probably need to configure these!" with this sort of change to
>> settings. This would be a first step towards shortening the distance from
>> `startproject` to a working deployment.
>>
>> Newer devs in particular have, based on my conversations and this
>> friend's research, been unlikely to (a) know that there are different
>> `startproject` templates, and (b) feel equipped to choose one, if they do
>> know.
>>
>> My hope is to make the smallest possible change to just start us moving
>> towards more clearly flagging, especially for newer devs, "these are things
>> that will need additional configuration in order to move from 'works on my
>> machine' to 'deployed'."
>>
>> Towards that end, I thought that adding a "you might want to get this
>> from the env" helper would be a clear indication to a new dev that this is
>> a matter to even consider. Adding other configuration-getting options like
>> different secret-store file backends se

Re: Making startproject's settings more 12-factor-y

[Kit La Touche]

Carlton—thanks very much for the feedback. Javier—likewise. In particular,
the imagined API you describe above is very appealing to me: start with
`from_env` and then if you learn more about this and want to, add in some
`EnvFileLoader`.

I want to make clear my motivation and agenda here: I have recently had
some conversations with newer devs about their experiences with deployment
of apps they're working on, and with a friend at Heroku about his informal
research into the problems people have with the same. One recurring
friction point (and this is not just on Heroku at all, to be clear) is that
there are a number of things that people *don't know they need to configure*
for a working deployment.

There are four settings that are recurring particular gotchas that people
miss: the secret key, debug, static files, and databases. Static files
seems big and out of scope, databases seems adequately handled by
dj-database-url for most cases, and if your case is more complex, you'll
learn it, but the other two (secret key and debug) seemed easy enough to
flag as "you probably need to configure these!" with this sort of change to
settings. This would be a first step towards shortening the distance from
`startproject` to a working deployment.

Newer devs in particular have, based on my conversations and this friend's
research, been unlikely to (a) know that there are different `startproject`
templates, and (b) feel equipped to choose one, if they do know.

My hope is to make the smallest possible change to just start us moving
towards more clearly flagging, especially for newer devs, "these are things
that will need additional configuration in order to move from 'works on my
machine' to 'deployed'."

Towards that end, I thought that adding a "you might want to get this from
the env" helper would be a clear indication to a new dev that this is a
matter to even consider. Adding other configuration-getting options like
different secret-store file backends seems like a good next step.

Thanks,

--Kit

On Wed, Jun 24, 2020 at 11:13 AM Javier Buzzi 
wrote:

> I looked at the libs that do what we want:
>
> django-configurations - it looks like they use environment variables /
> either via loading them from the environ or a key/value pair file. Having
> classes inside the settings.py might be weird to people.. at the least very
> different.
> confucius - very simplistic, only supports environ and is classed based,
> similar to django-configurations.
> django-environ - supports env file and environ, non-class based.
> dynaconf - supports all kinds of loading options (toml, json, ini,
> environ, .env +) non-class based.
>
> In my opinion, django-environ and dynaconf would be the easiest to sell to
> the community, it would require the least changes/paradigm shifts from how
> everyone is already using django.
>
> Personally, i would really like to see something like this inside my
> settings.py:
>
> from django.conf import from_env  # using standard os.environ
>
> DEBUG = from_env.bool("DEBUG", default=False)
>
> DATABASES = {
> "default":  from_env.db("DATABASE_URL")  # crash if it cant find it
> }
>
> ...
>
> for more complex examples:
>
> from django.conf import EnvFileLoader
>
> from_env = EnvFileLoader("path/to/.secret")
>
> ...
>
> We can have how ever many loaders we want: toml, json, ini ..
>
> This is both borrowing heavily from dynaconf and django-environ, making
> the fewest changes to how people are accustomed to doing things.
>
> .. what do you guys think?
>
> - Buzzi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/41cea044-ffe1-445d-ab7a-a65f69e6d85ao%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFcS-hDn0AFW4dj_-secrMGq-iWYz-TWKG-vw9%3DVmZ3L6j04aA%40mail.gmail.com.

Re: Making startproject's settings more 12-factor-y

[Kit La Touche]

I saw what you added; I tried to make this a minimal change, and was
conscious also Carl's expressed preference (which I share, tbh) in the
linked discussion for less-magic. I figured a function that just parsed
values out of the env would be the smallest possible first step, so I'm
inclined to keep with this, for now.

On Tue, Jun 23, 2020 at 4:15 PM Javier Buzzi  wrote:

> I think its great, but id add a little more customizability. I've added an
> example to your PR.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/8dc94197-fa0c-442a-8f97-3d7990a3a2c2o%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFcS-hD4RFMt%2BwQCCkt6TB5P3rEjarD9Bc9WcArvdaKBzxiYEw%40mail.gmail.com.

Re: Making startproject's settings more 12-factor-y

[Kit La Touche]

I have put in a PR (https://github.com/django/django/pull/13029) that I
think matches pretty well with the comments from Carl Meyer in the
conversation you linked. Please let me know what you think of it.

It adds a small helper function that will read an env var, optionally
coercing it to a type, optionally falling back to a default. It also makes
two of the settings in the default settings.py template (DEBUG and
SECRET_KEY) use this function, with their previous values set as defaults.

Thanks!

On Fri, Jun 5, 2020 at 4:39 PM Kit La Touche  wrote:

> I have, thank you! I tried to do my research before I wrote this email and
> this PR, but there's a ton of context to all those conversations that I'm
> sure I'd be missing, so I'm never gonna reach the point of feeling
> confident that I have a handle on what's gone before, alas.
>
> On Fri, Jun 5, 2020 at 4:03 PM Tim Graham  wrote:
>
>> Have you read past discussions on the mailing list?
>>
>> For example:
>>
>> Making Django more PaaS-friendly
>> https://groups.google.com/d/topic/django-developers/BAGhOKXGj4I/discussion
>>
>> Searching the archives for "12-factor" yields some other results.
>>
>> That may help you to craft your proposal.
>>
>> On Friday, June 5, 2020 at 5:45:07 PM UTC-4, Kit La Touche wrote:
>>>
>>> I'm interested in reducing the distance from running `startproject` to
>>> having a deployable Django app, having recently had a number of
>>> interactions with junior devs interested in using Django but finding
>>> deployment a nightmare. Some of this is of course on the other side, on the
>>> PaaS of choice, but some of it can (and I think should) be done on the
>>> Django side.
>>>
>>> I'd like to start small. If I made a PR to replace a few key settings
>>> with "get from the env, or fall back to this default", would people be
>>> interested in approving that? I hope so!
>>>
>>> --Kit
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/a45204c6-5475-4ee9-a467-78c66736cfado%40googlegroups.com
>> <https://groups.google.com/d/msgid/django-developers/a45204c6-5475-4ee9-a467-78c66736cfado%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFcS-hAfFWsbtjYQP%3DD92yjqNUx%2BBthffQWWkP2%2BWnGHG9__UA%40mail.gmail.com.

Re: Making startproject's settings more 12-factor-y

[Kit La Touche]

I have, thank you! I tried to do my research before I wrote this email and
this PR, but there's a ton of context to all those conversations that I'm
sure I'd be missing, so I'm never gonna reach the point of feeling
confident that I have a handle on what's gone before, alas.

On Fri, Jun 5, 2020 at 4:03 PM Tim Graham  wrote:

> Have you read past discussions on the mailing list?
>
> For example:
>
> Making Django more PaaS-friendly
> https://groups.google.com/d/topic/django-developers/BAGhOKXGj4I/discussion
>
> Searching the archives for "12-factor" yields some other results.
>
> That may help you to craft your proposal.
>
> On Friday, June 5, 2020 at 5:45:07 PM UTC-4, Kit La Touche wrote:
>>
>> I'm interested in reducing the distance from running `startproject` to
>> having a deployable Django app, having recently had a number of
>> interactions with junior devs interested in using Django but finding
>> deployment a nightmare. Some of this is of course on the other side, on the
>> PaaS of choice, but some of it can (and I think should) be done on the
>> Django side.
>>
>> I'd like to start small. If I made a PR to replace a few key settings
>> with "get from the env, or fall back to this default", would people be
>> interested in approving that? I hope so!
>>
>> --Kit
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/a45204c6-5475-4ee9-a467-78c66736cfado%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/a45204c6-5475-4ee9-a467-78c66736cfado%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFcS-hDDidkfNgDLuvLFegXUSQMAiouPU8t055qGW2KOx%3DKBSA%40mail.gmail.com.

Making startproject's settings more 12-factor-y

[Kit La Touche]

I'm interested in reducing the distance from running `startproject` to
having a deployable Django app, having recently had a number of
interactions with junior devs interested in using Django but finding
deployment a nightmare. Some of this is of course on the other side, on the
PaaS of choice, but some of it can (and I think should) be done on the
Django side.

I'd like to start small. If I made a PR to replace a few key settings with
"get from the env, or fall back to this default", would people be
interested in approving that? I hope so!

--Kit

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFcS-hA_YbUvsZJQNd-MD-4DfmvHwtWJfubOeOSb-46nkLPGmg%40mail.gmail.com.