Re: Proposal: FileSystemFinder and AppDirectoriesFinder shall serve unminimized assets in DEBUG mode

[Carlton Gibson]

It seems this thread ties in with the current discussion about static
assets. For instance if we were to include a compression step, you’d always
just reference the raw assets, these being processed, with source maps if
we had that, for production.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAJwKpyQn4Ai88OFhgF1QERDvO94F7Xe%3DihStt-nG2bDB9Dq4QA%40mail.gmail.com.

Re: Proposal: Allow stopwords in slugs generated by ModelAdmin.prepopulated_fields

[Andy Chosak]

Thanks, Adam, for your reply. I've opened a ticket at 
https://code.djangoproject.com/ticket/31511, which includes a link to a PR 
that makes this change.

Any advice on documenting how to wrap window.URLify?

Thanks,
Andy

On Thursday, April 9, 2020 at 1:41:30 PM UTC-4, Adam Johnson wrote:
>
> I for one am quite surprised to learn the admin has this behaviour.
>
> I'm extra surprised it assumes it's in English if only ASCII letters are 
> used. This is quite a naïve assumption 😂 (See what I did in that sentence?)
>
> Was removal of these words introduced for SEO reasons?
>>
>
> Seems likely.
>
> Personally, for the reasons you've presented I think it would make sense 
> to remove this behaviour. We can probably document how to wrap 
> window.URLify to preserve the old behaviour.
>
> On Wed, 8 Apr 2020 at 20:38, Andy Chosak > 
> wrote:
>
>> Automatic slug generation in ModelAdmin via prepopulated_fields 
>> 
>>  
>> uses a urlify.js 
>> 
>>  
>> file which, among other behaviors, removes certain stop words 
>> 
>>  
>> from the slug. For example, a string like "To be or not to be, that is the 
>> question" will generate a slug "be-or-not-be-question", not 
>> "to-be-or-not-to-be-that-is-the-question" as one might expect. I’d like to 
>> solicit feedback on the idea of removing this logic so that slugs can 
>> contain these words.
>>
>> For reference, the current list is: a, an, as, at, before, but, by, for, 
>> from, is, in, into, like, of, off, on, onto, per, since, than, the, this, 
>> that, to, up, via, with.
>>
>> Django ticket #30538  
>> mentions this behavior as part of a more general comparison between 
>> urlify.js and Python slugify 
>> .
>>  
>> It was closed as wontfix due to reasons of backwards compatibility. Per the 
>> triaging 
>> guidelines 
>> ,
>>  
>> I’m making this post to solicit feedback on the more specific question of 
>> addressing stopword removal in the JS code only -- not to try to address 
>> any other differences in behavior between these two methods. There’s been 
>> quite a bit of discussion on generating slugs for non-English languages 
>> (for example #2282 ), and 
>> this post is not intended to reopen that discussion.
>>
>> The current list of stopwords being removed seems to have been the same 
>> since 
>> at least 2005 
>> 
>>  
>> (the earliest code I can find including this logic). Some of these words 
>> feel a little unexpected, for example “before” and “since”. After 15 years 
>> it seems reasonable to revisit the list and consider whether it still makes 
>> sense.
>>
>> Was removal of these words introduced for SEO reasons? If so, is this 
>> still a recommended default behavior? In 2020, search engines like Google 
>> seem smart enough to interpret them properly. Here's 
>>  an arbitrary page that 
>> discusses this and includes a much longer list of what might be considered 
>> stopwords. As another datapoint, the popular WordPress Yoast SEO plugin 
>> used to remove stopwords, but stopped doing so 
>>  a few years back.
>>
>> Potentially outdated SEO concerns aside, does this behavior still align 
>> well with the needs and desires of Django users? Is this something this 
>> community would be open to revisiting? Thanks for your consideration.
>>
>> (One minor point on language support: allowing these words would help to 
>> resolve at least some of the unequal treatment given to English over other 
>> languages, for example #12905 
>> . See also wagtail#4899 
>> , from which much of 
>> this post has been copied, for an example of how this logic impacts a 
>> Django-based CMS.)
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-d...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/fb6c9596-951d-4102-91b5-b5fd9c8c6340%40googlegroups.com
>>  
>> 

Re: Proposal: FileSystemFinder and AppDirectoriesFinder shall serve unminimized assets in DEBUG mode

[Adam Johnson]

Maybe we can move to always serving minimized assets with corresponding
source maps? I was under the impression this is the "best practice" these
days.

On Thu, 23 Apr 2020 at 10:09, Jacob Rief  wrote:

> When specifying paths to assets like JavaScript files, but also CSS, the
> Django documentation states:
>
>> Any links to the file in the codebase should point to the compressed
>> version.
>> 
>
>
> This statement however does not properly reflect Django's internal
> handling of such assets, for instance in
>
> https://github.com/django/django/blob/master/django/contrib/admin/helpers.py#L83-L84
>
> https://github.com/django/django/blob/master/django/contrib/admin/options.py#L639-L648
>
> https://github.com/django/django/blob/master/django/contrib/admin/options.py#L2026-L2028
>
> https://github.com/django/django/blob/master/django/contrib/admin/widgets.py#L446-L460
> we see, that Django only serves the minimized version of the file, when
> not in DEBUG mode.
>
> This inconsistent handling of assets can cause other problems for third
> party apps, if they follow the documentation as show in the first link.
> If in their Media definition they refer to the minimized version of a
> Django asset, say 'admin/js/vendor/jquery/jquery.min.js', then
> the automatic sorting does not work anymore in DEBUG mode. If on the other
> side they refer to 'admin/js/vendor/jquery/jquery.js',
> then automatic sorting fails in production.
>
> There are two solutions to this problem.
>
> *Either*
>
> The documentation explicitly states that when a Media definition refers to
> internal Django assets, one *must* distinguish between the
> minimized and unminimized version, just as Django's internal Media
> definitions do (see code examples above).
> This however can easily be forgotten and errors therefore become visible
> only after deployment.
>
> *Or*
>
> We rewrite the FileSystemFinder and AppDirectoriesFinder so that in DEBUG
> mode, Django looks if an unminimized version
> of the same asset exists, and if so it then serves that. Doing that
> automatically is easy, because the general convention is, that the .min
> infix is always placed immediately before the .js- or .css file extension.
>
> In many of my projects I therefore use these two alternative finders,
> which do exactly that:
>
> import os
> from django.conf import settings
> from django.contrib.staticfiles.finders import (FileSystemFinder as 
> FileSystemFinderBase, AppDirectoriesFinder as AppDirectoriesFinderBase)
>
>
> class FileSystemFinder(FileSystemFinderBase):
> """
> In debug mode, serve /static/any/asset.min.ext as /static/any/asset.ext
> """
> locations = []
> serve_unminimized = getattr(settings, 'DEBUG', False)
>
> def find_location(self, root, path, prefix=None):
> if self.serve_unminimized:
> # search for the unminimized version, and if it exists, return it
> base, ext = os.path.splitext(path)
> base, minext = os.path.splitext(base)
> if minext == '.min':
> unminimized_path = super().find_location(root, base + ext, 
> prefix)
> if unminimized_path:
> return unminimized_path
> # otherwise proceed with the given one
> path = super().find_location(root, path, prefix)
> return path
>
>
> class AppDirectoriesFinder(AppDirectoriesFinderBase):
> serve_unminimized = getattr(settings, 'DEBUG', False)
>
> def find_in_app(self, app, path):
> matched_path = super().find_in_app(app, path)
> if matched_path and self.serve_unminimized:
> base, ext = os.path.splitext(matched_path)
> base, minext = os.path.splitext(base)
> if minext == '.min':
> storage = self.storages.get(app, None)
> path = base + ext
> if storage.exists(path):
> path = storage.path(path)
> if path:
> return path
> return matched_path
>
>
> In my opinion this approach makes Django's internal and 3rd party package
> code more readable, because it removes the clutter of
> case distinction between DEBUG and production mode for each referred
> JavaScript and CSS asset. It also might be less error prone.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/dd904eeb-a877-48d3-81fa-e72fe71a6735%40googlegroups.com
> 

Re: Remove automatic date-naming of migrations (00XX_auto_YYYMMDD)

[Adam Johnson]

Thank you all for the feedback.

Re: Andrew:

> (60 is a bit long though, maybe we can bump it down to something a bit
> shorter?)
>

Sure, how about 42? 👽

Re: Jon:

> Did you have a suggestion for this situation? Revert back to auto-naming
> or request the user to name the migration?
>

The patch as-is accumulates suggested names from operations in order, until
the length is >=60 characters. Later operations just aren't mentioned.

Re: Claude:

> An alternative could be to ask the user in interactive mode (and keep the
> current behaviour in non-interactive mode).
>

I'd be up for this if we "pre-fill" the input with the auto-generated name,
to make it easy to continue when we can suggest a reasonable name. Forcing
users to type a name from scratch could be annoying especially when
iterating on a new feature and dropping/rebuilding the respective migration.

Re: Caio:

> I work on a project where migration names are fixed to “auto”. We use a
> similar technique to those mentioned in the blog post to do that. The
> reason is that we want to force developers to get conflicts (git) on
> migration names during the review process, rather than having to handle
> migration merging manually during deploy


Yes, this is a secondary problem with migrations, trying to keep the
history linear via git.

I've used an alternative solution where a second file is created in the
migrations folder called "latest" or similar, that simply contains the name
of the latest migration file. This forces a conflict.

Although that's an alternative proposal, I think adding something like that
to Django could be a good idea. It's better than forcing all migrations to
have meaningless names.


On Wed, 22 Apr 2020 at 16:47, Andrew Godwin  wrote:

> I am a little mixed on this change - the behaviour during the initial
> development was indeed to concatenate names like this, albeit only for
> adding fields or models; I thought it looked unwieldy, which is why I added
> the "auto" name.
>
> That said, the number is the only part that actually matters, and while
> the date is sometimes useful for people to resolve merge conflicts, I don't
> think it's better than more informative autogenerated names, so I'm happy
> to go with the change.
>
> (60 is a bit long though, maybe we can bump it down to something a bit
> shorter?)
>
> Andrew
>
> On Wed, Apr 22, 2020 at 7:06 AM Adam Johnson  wrote:
>
>> Hi djangonauts,
>>
>> In a blog post earlier this year I outlined my technique to prevent
>> Django creating migration files with automatic date names. I had a lot of
>> response with other techniques and ended up adding two more techniques to
>> the post. It's at
>> https://adamj.eu/tech/2020/02/24/how-to-disallow-auto-named-django-migrations/
>> .
>>
>> The problem with such migration names:
>>
>> When you run Django’s manage.py makemigrations, it will try to generate
>>> a name for the migration based upon its contents. For example, if you are
>>> adding a single field, it will call the migration
>>> 0002_mymodel_myfield.py. However when your migration contains more than
>>> one step, it instead uses a simple ‘auto’ name with the current date +
>>> time, e.g. 0002_auto_20200113_1837.py. You can provide the -n/--name
>>> argument to makemigrations, but developers often forget this.
>>>
>>> Naming things is a known hard problem in programming. Having migrations
>>> with these automatic names makes managing them harder: You can’t tell which
>>> is which without opening them, and you could still mix them up if they have
>>> similar names due to being generated on the same day.
>>>
>> Django *currently* sets the migration name to something other than the
>> automatic date name in two cases:
>>
>>- If the migration contains a single operation, it uses a name based
>>on that operation, e.g. 00023_remove_model_field, or 0024_add_model_field
>>(but not for all operation types)
>>- If the migration consists *only* of model creation operations, it
>>combines their operation names names, which come as just the lower-cased
>>model names e.g. 0025_book_author_sale
>>
>> I opened a PR to expand the operation naming for the single case to cover
>> all built-in operations: https://github.com/django/django/pull/12131
>>
>> I'd like to propose using this new full coverage of operation naming to
>> remove the "auto_MMDD" behaviour, and instead always combine
>> operations' "suggested migration names" up until a limit of say 60
>> characters. I made a commit for that here:
>> https://github.com/adamchainz/django/commit/c2bc6893a05c2c8099e1fb68e688618446641ed6
>>
>> This would lead to migration names such as:
>>
>>- 0026_remove_book_title_add_book_description
>>
>> Whilst perhaps long, they're explict and imo easier to work with than the
>> auto_MMDD behaviour.
>>
>> Mariusz wrote on the PR:
>>
>> Personally, I'm against removing auto named migrations. IMO chaining
>>> operation names is even more confusing.
>>>
>> --
>> Adam
>>
>> 

Re: Deadlock bug in logging? Reproducible case

[Brian Tiemann]

True, I did notice it needed some more tweaking. I've got a long evening of 
rewriting years' worth of wsgi.py's and Apache configs ahead of me. I'll 
probably just jump straight to the envparse approach because (as you noted) 
my celery env var handling was making the WSGI approach moot anyway.

Now if there were only a way to avoid having to put 
os.environ["DJANGO_SETTINGS_MODULE"] = "myapp.settings" into wsgi.py (since 
I'm sharing lots of vhosts in one Apache) ...

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9bab5139-ea38-418e-8b97-c49463be65e3%40googlegroups.com.

Re: Deadlock bug in logging? Reproducible case

[Adam Johnson]

Actually that might not work entirely, since your settings file will be
imported before the environment variables are copied over.

Instead you can lazily construct the django application on the first
request:

django_application = None

def application(environ, start_response):
global django_application

if django_application is None:
# pass the WSGI environment variables on through to os.environ
for var in env_variables_to_pass:
os.environ[var] = environ.get(var, '')
django_application = get_wsgi_application()

return django_application(environ, start_response)

And yes, using a real environment variable solution is better. You can only
use WSGI environment for configuration during requests - not during
background processes such as management commands or background task queues
like Celery.

On Thu, 23 Apr 2020 at 15:00, Brian Tiemann  wrote:

> Beautiful. That does the trick. Thank you!
>
> And I certainly can see there's plenty of other approaches such as
> envparse or django-environ that I could be using, that keeps the vars out
> of my Apache config. Quick fix and a slighly longer better fix. This'll
> change how I do all my new projects from now on, plus retroactively for all
> the ones I used this approach with...
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/97ea9ad1-7f9d-4648-8eb5-c3b8a42f479b%40googlegroups.com
> 
> .
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM1Ti0inv_5xrc-OSJJSKPWCzvpp0qy7OeB%2B2JpOy1iwBA%40mail.gmail.com.

Re: Deadlock bug in logging? Reproducible case

[Brian Tiemann]

Beautiful. That does the trick. Thank you!

And I certainly can see there's plenty of other approaches such as envparse 
or django-environ that I could be using, that keeps the vars out of my 
Apache config. Quick fix and a slighly longer better fix. This'll change 
how I do all my new projects from now on, plus retroactively for all the 
ones I used this approach with...

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/97ea9ad1-7f9d-4648-8eb5-c3b8a42f479b%40googlegroups.com.

Re: Deadlock bug in logging? Reproducible case

[Adam Johnson]

https://adamj.eu/tech/2019/05/27/the-simplest-wsgi-middleware/

django_application = get_wsgi_application()

def application(environ, start_response):
# pass the WSGI environment variables on through to os.environ
for var in env_variables_to_pass:
os.environ[var] = environ.get(var, '')
return django_applicationenviron, start_response)

But I think it's questionable to push some environment variables into wsgi
environ, you can read the environment variables elsewhere.


On Thu, 23 Apr 2020 at 14:25, Brian Tiemann  wrote:

> Ah, interesting. Let me show you what my non-trimmed-down wsgi.py looked
> like --
>
> import os, sys
> from django.core.wsgi import get_wsgi_application
>
> env_variables_to_pass = [
> 'SECRET_KEY',
> ... other env vars ...
> ]
>
> def application(environ, start_response):
> # pass the WSGI environment variables on through to os.environ
> for var in env_variables_to_pass:
> os.environ[var] = environ.get(var, '')
> return get_wsgi_application()(environ, start_response)
>
> That's why I had it set up that way. This was how someone once told me env
> vars should be passed in via the Apache config. Is there a better way to do
> that? Like I want to define an env var in the Apache  block,
> but I want it to then be available to the app via os.environ. Is there a
> best-practices way to make that happen?
>
> Thank you for digging into this!
>
>
> On Thursday, April 23, 2020 at 9:05:20 AM UTC-4, Adam Johnson wrote:
>>
>> I ran it on Python 3.7.6 and Python 3.8.2 and found the deadlock issue on
>> both.
>>
>> I then adjusted the settings to *only* have the console logger, and added
>> these lines at the end of the settings file:
>>
>> import faulthandler
>> faulthandler.enable()
>> print(os.getpid())
>>
>> faulthandler allows getting a stacktrace in dire situations such as hangs
>> like this: https://docs.python.org/3/library/faulthandler.html
>>
>> Then used `runserver`, ran until the deadlock, and killed the *second*
>> printed PID with `kill -ABRT` . This kills the actual Django server
>> process, rather than the outer "watch & restart" process.
>>
>> This ended with a traceback with lots of threads all trying to acquire a
>> lock. Some are trying to log at the end of the request:
>>
>> Thread 0x7cecf000 (most recent call first):
>>   File
>> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/__init__.py",
>> line 221 in _acquireLock
>>   File
>> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/__init__.py",
>> line 1687 in isEnabledFor
>>   File
>> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/__init__.py",
>> line 1433 in info
>>   File
>> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
>> line 157 in log_message
>>   File
>> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/http/server.py", line
>> 544 in log_request
>>   File
>> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/wsgiref/simple_server.py",
>> line 34 in close
>>   File
>> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
>> line 114 in close
>>   File
>> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/wsgiref/handlers.py",
>> line 196 in finish_response
>>   File
>> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/wsgiref/handlers.py",
>> line 138 in run
>>   File
>> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
>> line 197 in handle_one_request
>>   File
>> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
>> line 174 in handle
>>   File
>> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/socketserver.py", line
>> 720 in __init__
>>   File
>> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/socketserver.py", line
>> 360 in finish_request
>>   File
>> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/socketserver.py", line
>> 650 in process_request_thread
>>   File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/threading.py",
>> line 870 in run
>>   File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/threading.py",
>> line 932 in _bootstrap_inner
>>   File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/threading.py",
>> line 890 in _bootstrap
>>
>> One stood out to me as trying to reconfigure logging through wsgi.py:
>>
>> Thread 0x7eed5000 (most recent call first):
>>   File "", line 163 in _get_module_lock
>>   File "", line 148 in __enter__
>>   File "", line 988 in _find_and_load
>>   File "", line 1014 in _gcd_import
>>   File
>> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/importlib/__init__.py",
>> line 127 in import_module
>>   File
>> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/utils/module_loading.py",
>> line 17 in import_string
>>   File
>> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages

Re: Deadlock bug in logging? Reproducible case

[Brian Tiemann]

FYI, here's where that approach came from 
-- 
http://ericplumb.com/blog/passing-apache-environment-variables-to-django-via-mod_wsgi.html
>
>
It's the top hit for "django apache environment variable" Which is 
literally an old friend of mine from grade school. Maybe that's a red flag 
...

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9b119d25-31fb-4eca-a116-abfd61886777%40googlegroups.com.

Re: Deadlock bug in logging? Reproducible case

[Brian Tiemann]

Ah, interesting. Let me show you what my non-trimmed-down wsgi.py looked 
like --

import os, sys
from django.core.wsgi import get_wsgi_application

env_variables_to_pass = [
'SECRET_KEY',
... other env vars ...
]

def application(environ, start_response):
# pass the WSGI environment variables on through to os.environ
for var in env_variables_to_pass:
os.environ[var] = environ.get(var, '')
return get_wsgi_application()(environ, start_response)

That's why I had it set up that way. This was how someone once told me env 
vars should be passed in via the Apache config. Is there a better way to do 
that? Like I want to define an env var in the Apache  block, 
but I want it to then be available to the app via os.environ. Is there a 
best-practices way to make that happen?

Thank you for digging into this!


On Thursday, April 23, 2020 at 9:05:20 AM UTC-4, Adam Johnson wrote:
>
> I ran it on Python 3.7.6 and Python 3.8.2 and found the deadlock issue on 
> both.
>
> I then adjusted the settings to *only* have the console logger, and added 
> these lines at the end of the settings file:
>
> import faulthandler
> faulthandler.enable()
> print(os.getpid())
>
> faulthandler allows getting a stacktrace in dire situations such as hangs 
> like this: https://docs.python.org/3/library/faulthandler.html
>
> Then used `runserver`, ran until the deadlock, and killed the *second* 
> printed PID with `kill -ABRT` . This kills the actual Django server 
> process, rather than the outer "watch & restart" process.
>
> This ended with a traceback with lots of threads all trying to acquire a 
> lock. Some are trying to log at the end of the request:
>
> Thread 0x7cecf000 (most recent call first):
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/__init__.py", 
> line 221 in _acquireLock
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/__init__.py", 
> line 1687 in isEnabledFor
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/__init__.py", 
> line 1433 in info
>   File 
> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
>  
> line 157 in log_message
>   File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/http/server.py", 
> line 544 in log_request
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/wsgiref/simple_server.py", 
> line 34 in close
>   File 
> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
>  
> line 114 in close
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/wsgiref/handlers.py", 
> line 196 in finish_response
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/wsgiref/handlers.py", 
> line 138 in run
>   File 
> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
>  
> line 197 in handle_one_request
>   File 
> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
>  
> line 174 in handle
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/socketserver.py", line 
> 720 in __init__
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/socketserver.py", line 
> 360 in finish_request
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/socketserver.py", line 
> 650 in process_request_thread
>   File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/threading.py", 
> line 870 in run
>   File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/threading.py", 
> line 932 in _bootstrap_inner
>   File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/threading.py", 
> line 890 in _bootstrap
>
> One stood out to me as trying to reconfigure logging through wsgi.py:
>
> Thread 0x7eed5000 (most recent call first):
>   File "", line 163 in _get_module_lock
>   File "", line 148 in __enter__
>   File "", line 988 in _find_and_load
>   File "", line 1014 in _gcd_import
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/importlib/__init__.py", 
> line 127 in import_module
>   File 
> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/utils/module_loading.py",
>  
> line 17 in import_string
>   File 
> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/utils/log.py",
>  
> line 89 in __init__
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/config.py", line 
> 744 in configure_handler
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/config.py", line 
> 563 in configure
>   File 
> "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/config.py", line 
> 808 in dictConfig
>   File 
> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/utils/log.py",
>  
> line 71 in configure_logging
>   File 
> "/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/_

Re: Deadlock bug in logging? Reproducible case

[Adam Johnson]

I ran it on Python 3.7.6 and Python 3.8.2 and found the deadlock issue on
both.

I then adjusted the settings to *only* have the console logger, and added
these lines at the end of the settings file:

import faulthandler
faulthandler.enable()
print(os.getpid())

faulthandler allows getting a stacktrace in dire situations such as hangs
like this: https://docs.python.org/3/library/faulthandler.html

Then used `runserver`, ran until the deadlock, and killed the *second*
printed PID with `kill -ABRT` . This kills the actual Django server
process, rather than the outer "watch & restart" process.

This ended with a traceback with lots of threads all trying to acquire a
lock. Some are trying to log at the end of the request:

Thread 0x7cecf000 (most recent call first):
  File
"/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/__init__.py",
line 221 in _acquireLock
  File
"/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/__init__.py",
line 1687 in isEnabledFor
  File
"/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/__init__.py",
line 1433 in info
  File
"/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
line 157 in log_message
  File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/http/server.py",
line 544 in log_request
  File
"/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/wsgiref/simple_server.py",
line 34 in close
  File
"/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
line 114 in close
  File
"/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/wsgiref/handlers.py",
line 196 in finish_response
  File
"/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/wsgiref/handlers.py",
line 138 in run
  File
"/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
line 197 in handle_one_request
  File
"/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
line 174 in handle
  File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/socketserver.py",
line 720 in __init__
  File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/socketserver.py",
line 360 in finish_request
  File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/socketserver.py",
line 650 in process_request_thread
  File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/threading.py",
line 870 in run
  File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/threading.py",
line 932 in _bootstrap_inner
  File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/threading.py",
line 890 in _bootstrap

One stood out to me as trying to reconfigure logging through wsgi.py:

Thread 0x7eed5000 (most recent call first):
  File "", line 163 in _get_module_lock
  File "", line 148 in __enter__
  File "", line 988 in _find_and_load
  File "", line 1014 in _gcd_import
  File
"/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/importlib/__init__.py",
line 127 in import_module
  File
"/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/utils/module_loading.py",
line 17 in import_string
  File
"/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/utils/log.py",
line 89 in __init__
  File
"/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/config.py", line
744 in configure_handler
  File
"/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/config.py", line
563 in configure
  File
"/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/logging/config.py", line
808 in dictConfig
  File
"/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/utils/log.py",
line 71 in configure_logging
  File
"/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/__init__.py",
line 19 in setup
  File
"/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/wsgi.py",
line 12 in get_wsgi_application
  File "/Users/chainz/Documents/Projects/wedge/wedge/wsgi.py", line 6 in
application
  File
"/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/contrib/staticfiles/handlers.py",
line 68 in __call__
  File
"/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/wsgiref/handlers.py",
line 137 in run
  File
"/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
line 197 in handle_one_request
  File
"/Users/chainz/Documents/Projects/wedge/venv38/lib/python3.8/site-packages/django/core/servers/basehttp.py",
line 174 in handle
  File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/socketserver.py",
line 720 in __init__
  File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/socketserver.py",
line 360 in finish_request
  File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/socketserver.py",
line 650 in process_request_thread
  File "/Users/chainz/.pyenv/versions/3.8.2/lib/python3.8/threading.py",
line 870 in run
  F

Re: New Merger nomination.

[Carlton Gibson]

Hi all, thanks. 

I think that's everyone, so I'll add Claude to the team. 

💃

Kind Regards,

Carlton

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f3f9f4ba-ab67-40b3-882c-985faff9b826%40googlegroups.com.

Re: Deadlock bug in logging? Reproducible case

[Brian Tiemann]

It happens with Apache/mod_wsgi in prod.


On Thursday, April 23, 2020 at 7:59:45 AM UTC-4, Tom Forbes wrote:
>
> Just to confirm - are you running into this issue while using the Django 
> development server in production, or does this occur with Gunicorn/uwsgi as 
> well?
>
> Tom
>
> On 23 Apr 2020, at 12:40, Brian Tiemann > 
> wrote:
>
> It also happens in 3.6.9, which is my prod environment.
>
> FWIW, by way of background context, this is not a heavily used app, not 
> enough so to warrant logging to a custom buffered consumer or remote agent. 
> It's just a small office using it. But the main home view's UI has 5 AJAX 
> calls that cause the site to lock up on a regular basis, such that I have 
> to have a watchdog in place to bounce Apache whenever it happens.
>
> It's not the logging itself that's the issue; none of the AJAX views log 
> anything. The app only actually logs maybe 100 lines per day. My 
> investigation however turned up that if I commented out LOGGING altogether, 
> it worked fine. Further trimming down of the codebase showed that I could 
> remove all the loggers and it would still happen as long as any handlers 
> were defined. So that's how I set up the demo. The logging itself isn't 
> important, in other words; it's that the whole app is wedging just because 
> I happen to have a vanilla LOGGING setup configured.
>
>
>
> On Thursday, April 23, 2020 at 7:02:11 AM UTC-4, Brian Tiemann wrote:
>>
>> Hi — Thanks for looking into this. Adam, I skipped over details like the 
>> Python version because I noted them in the readme on my gitlab demo. But I 
>> was using Python 3.7.2 for the record.
>>
>> I used "deadlock" because my digging into this had previously turned up 
>> https://bugs.python.org/issue6721 , suggesting this was to do with 
>> thread locking. But I'm not qualified to make that call for sure, so I 
>> don't want to throw red herrings into the mix. If "standstill" is a better 
>> term for this that's what I'll use.
>>
>> Thanks again.
>>
>>
>> On Wednesday, April 22, 2020 at 1:23:31 PM UTC-4, Brian Tiemann wrote:
>>>
>>> Hi all,
>>>
>>> I was directed here after getting corroboration on #django and elsewhere.
>>>
>>> I have what appears to be a bug in which a Django app can deadlock if 
>>> you hit it with a lot (>3) of AJAX requests within a short time (i.e. all 
>>> triggered in parallel from a single HTML page). I have a reproducible case 
>>> here:
>>>
>>> https://github.com/data-graham/wedge
>>>
>>> I'd really appreciate it if someone could take a look and let me know 
>>> whether I'm doing something wrong, or if there's something systemic going 
>>> on. Thanks much!
>>>
>>>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/9a5bc078-66b9-4e09-97c2-8340b8882c02%40googlegroups.com
>  
> 
> .
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/aa37f782-50bb-42a9-b4ed-45516e3751e5%40googlegroups.com.

Re: Deadlock bug in logging? Reproducible case

[Tom Forbes]

Just to confirm - are you running into this issue while using the Django 
development server in production, or does this occur with Gunicorn/uwsgi as 
well?

Tom

> On 23 Apr 2020, at 12:40, Brian Tiemann  wrote:
> 
> It also happens in 3.6.9, which is my prod environment.
> 
> FWIW, by way of background context, this is not a heavily used app, not 
> enough so to warrant logging to a custom buffered consumer or remote agent. 
> It's just a small office using it. But the main home view's UI has 5 AJAX 
> calls that cause the site to lock up on a regular basis, such that I have to 
> have a watchdog in place to bounce Apache whenever it happens.
> 
> It's not the logging itself that's the issue; none of the AJAX views log 
> anything. The app only actually logs maybe 100 lines per day. My 
> investigation however turned up that if I commented out LOGGING altogether, 
> it worked fine. Further trimming down of the codebase showed that I could 
> remove all the loggers and it would still happen as long as any handlers were 
> defined. So that's how I set up the demo. The logging itself isn't important, 
> in other words; it's that the whole app is wedging just because I happen to 
> have a vanilla LOGGING setup configured.
> 
> 
> 
> On Thursday, April 23, 2020 at 7:02:11 AM UTC-4, Brian Tiemann wrote:
> Hi — Thanks for looking into this. Adam, I skipped over details like the 
> Python version because I noted them in the readme on my gitlab demo. But I 
> was using Python 3.7.2 for the record.
> 
> I used "deadlock" because my digging into this had previously turned up 
> https://bugs.python.org/issue6721  , 
> suggesting this was to do with thread locking. But I'm not qualified to make 
> that call for sure, so I don't want to throw red herrings into the mix. If 
> "standstill" is a better term for this that's what I'll use.
> 
> Thanks again.
> 
> 
> On Wednesday, April 22, 2020 at 1:23:31 PM UTC-4, Brian Tiemann wrote:
> Hi all,
> 
> I was directed here after getting corroboration on #django and elsewhere.
> 
> I have what appears to be a bug in which a Django app can deadlock if you hit 
> it with a lot (>3) of AJAX requests within a short time (i.e. all triggered 
> in parallel from a single HTML page). I have a reproducible case here:
> 
> https://github.com/data-graham/wedge 
> 
> I'd really appreciate it if someone could take a look and let me know whether 
> I'm doing something wrong, or if there's something systemic going on. Thanks 
> much!
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/9a5bc078-66b9-4e09-97c2-8340b8882c02%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9B0CF9F5-DB99-4D82-BAFC-4D9E481BCFD7%40tomforb.es.

Re: Deadlock bug in logging? Reproducible case

[Brian Tiemann]

It also happens in 3.6.9, which is my prod environment.

FWIW, by way of background context, this is not a heavily used app, not 
enough so to warrant logging to a custom buffered consumer or remote agent. 
It's just a small office using it. But the main home view's UI has 5 AJAX 
calls that cause the site to lock up on a regular basis, such that I have 
to have a watchdog in place to bounce Apache whenever it happens.

It's not the logging itself that's the issue; none of the AJAX views log 
anything. The app only actually logs maybe 100 lines per day. My 
investigation however turned up that if I commented out LOGGING altogether, 
it worked fine. Further trimming down of the codebase showed that I could 
remove all the loggers and it would still happen as long as any handlers 
were defined. So that's how I set up the demo. The logging itself isn't 
important, in other words; it's that the whole app is wedging just because 
I happen to have a vanilla LOGGING setup configured.



On Thursday, April 23, 2020 at 7:02:11 AM UTC-4, Brian Tiemann wrote:
>
> Hi — Thanks for looking into this. Adam, I skipped over details like the 
> Python version because I noted them in the readme on my gitlab demo. But I 
> was using Python 3.7.2 for the record.
>
> I used "deadlock" because my digging into this had previously turned up 
> https://bugs.python.org/issue6721 , suggesting this was to do with thread 
> locking. But I'm not qualified to make that call for sure, so I don't want 
> to throw red herrings into the mix. If "standstill" is a better term for 
> this that's what I'll use.
>
> Thanks again.
>
>
> On Wednesday, April 22, 2020 at 1:23:31 PM UTC-4, Brian Tiemann wrote:
>>
>> Hi all,
>>
>> I was directed here after getting corroboration on #django and elsewhere.
>>
>> I have what appears to be a bug in which a Django app can deadlock if you 
>> hit it with a lot (>3) of AJAX requests within a short time (i.e. all 
>> triggered in parallel from a single HTML page). I have a reproducible case 
>> here:
>>
>> https://github.com/data-graham/wedge
>>
>> I'd really appreciate it if someone could take a look and let me know 
>> whether I'm doing something wrong, or if there's something systemic going 
>> on. Thanks much!
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9a5bc078-66b9-4e09-97c2-8340b8882c02%40googlegroups.com.

Re: Deadlock bug in logging? Reproducible case

[Brian Tiemann]

Hi — Thanks for looking into this. Adam, I skipped over details like the 
Python version because I noted them in the readme on my gitlab demo. But I 
was using Python 3.7.2 for the record.

I used "deadlock" because my digging into this had previously turned up 
https://bugs.python.org/issue6721 , suggesting this was to do with thread 
locking. But I'm not qualified to make that call for sure, so I don't want 
to throw red herrings into the mix. If "standstill" is a better term for 
this that's what I'll use.

Thanks again.


On Wednesday, April 22, 2020 at 1:23:31 PM UTC-4, Brian Tiemann wrote:
>
> Hi all,
>
> I was directed here after getting corroboration on #django and elsewhere.
>
> I have what appears to be a bug in which a Django app can deadlock if you 
> hit it with a lot (>3) of AJAX requests within a short time (i.e. all 
> triggered in parallel from a single HTML page). I have a reproducible case 
> here:
>
> https://github.com/data-graham/wedge
>
> I'd really appreciate it if someone could take a look and let me know 
> whether I'm doing something wrong, or if there's something systemic going 
> on. Thanks much!
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/db88ca65-b8f7-4f29-b156-538ee373b518%40googlegroups.com.

Re: Deadlock bug in logging? Reproducible case

[Adam Johnson]

What version of Python René?

On Thu, 23 Apr 2020 at 10:58, René Fleschenberg 
wrote:

> Hi,
>
> On 4/23/20 9:40 AM, Adam Johnson wrote:
> > Brian - You didn't describe the exact symptoms you're seeing. "Deadlock"
> > has a particular technical meaning around multiple processes requesting
> > locks mutually off each other, and normally ends in one process being
> > terminated. But I presume you're experiencing more of a "standstill" and
> > future requests no longer work?
>
> I was able to reproduce this. It leads to a standstill, further requests
> are not getting processed. I can reproduce it when logging to the
> console, so I don't think disk I/O is the problem. It might still be a
> problem in the Python stdlib, of course.
>
> --
> René
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/1b14cc08-4abf-9a97-8e02-a16931c0bf28%40fleschenberg.net
> .
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM3cH-FM9Rm8ZLHwXafATPT6G4z4w%2B_M4yu_hQOgFQ%2B%2Bag%40mail.gmail.com.

Re: Deadlock bug in logging? Reproducible case

[René Fleschenberg]

Hi,

On 4/23/20 9:40 AM, Adam Johnson wrote:
> Brian - You didn't describe the exact symptoms you're seeing. "Deadlock"
> has a particular technical meaning around multiple processes requesting
> locks mutually off each other, and normally ends in one process being
> terminated. But I presume you're experiencing more of a "standstill" and
> future requests no longer work?

I was able to reproduce this. It leads to a standstill, further requests
are not getting processed. I can reproduce it when logging to the
console, so I don't think disk I/O is the problem. It might still be a
problem in the Python stdlib, of course.

-- 
René

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1b14cc08-4abf-9a97-8e02-a16931c0bf28%40fleschenberg.net.

Re: Hashing Session Keys in backends

[mark]

Hey Adam, thanks for the feedback, I'll make sure to credit Chris' original 
work in a new PR, I think I'm getting close to having one ready.

Is there a way to avoid breaking third party backends, but raising 
> deprecation warnings?
>
 
I could create a new SessionBase child class (something like 
*HashingSessionBase*) and make all the default backends inherit from this 
class, instead of from *SessionBase* directly. Any sessions backend that 
does not inherit from the new *HashingSessionBase* could then be nominated 
for deprecation. This will also allow making no/minimal changes to the 
*SessionBase* class to optimise backward compatibility.


 

On Wednesday, April 15, 2020 at 1:41:00 AM UTC+2, Adam Johnson wrote:
>
> Hi Mark
>
> Thanks for looking into this tricky security issue.
>
> I'm suggesting to use the names frontend_key and backend_key for these two 
>> concepts.
>>
>
> They seem reasonable to me, as long as there's an explanatory comment. 
> Perhaps it even needs documenting for third party backends.
>
> My second suggestion is to refactor the SessionBase class to make sure the 
>> session-key-hashing happens in one place and isn't spread across all 
>> different backend implementations as is the case now because the subclasses 
>> have to implemented public methods that receive the frontend_key.
>>
>
> Yes that seems reasonable, although I haven't looked closely. Is there a 
> way to avoid breaking third party backends, but raising deprecation 
> warnings? Perhaps using func_supports_parameter 
> 
>  
> or similar that we've used in past deprecations?
>
> I see the PR is quite old and not owned by you. If you want to open a new 
> PR, rebase the existing code, and refactor it as you see fit, you can use 
> Co-Authored-By 
>  to 
> acknowledge Chris' original work.
>
> Thanks,
>
> Adam
>
> On Fri, 10 Apr 2020 at 10:41, mark > 
> wrote:
>
>> After renewed interest because of potential database timing attacks (
>> T31412 ) I'm looking into 
>> an existing PR (PR8736  for 
>> T21076 ) for adding the 
>> possibility of storing hashes of session keys.
>>
>> I'm looking to get some feedback on two things;
>>
>> After going through the existing commits of Chris Griffin, I agree with 
>> Aymeric Augustin (who did an initial review of the pull request) that there 
>> should be a clearer distinction between the incoming session key (Aymeric 
>> talks about a "clear text session key") and the key that gets stored in the 
>> sessions backend (Aymeric talks about a "hashed if needed session key").
>> I'm suggesting 
>>  to 
>> use the names *frontend_key* and *backend_key* for these two concepts.
>>
>> My second suggestion 
>>  is 
>> to refactor the *SessionBase* class to make sure the 
>> session-key-hashing happens in one place and isn't spread across all 
>> different backend implementations as is the case now because the subclasses 
>> have to implemented public methods that receive the frontend_key. I'm 
>> suggesting to basically have subclasses implement private methods that 
>> receive a backend_key, which will be invoked by the public methods in the 
>> BaseClass. Obviously this will have consequences for any existing custom 
>> backends out there, though I think those will be affected either way. 
>>
>> I welcome any thoughts on both the naming convention and the refactoring.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-d...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/41c26919-f15f-4151-aa82-1281e24656da%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Adam
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8faf8d61-d681-40e8-839e-72b2a3a8915a%40googlegroups.com.

Proposal: FileSystemFinder and AppDirectoriesFinder shall serve unminimized assets in DEBUG mode

[Jacob Rief]

When specifying paths to assets like JavaScript files, but also CSS, the 
Django documentation states: 

> Any links to the file in the codebase should point to the compressed 
> version. 
> 


This statement however does not properly reflect Django's internal handling 
of such assets, for instance in
https://github.com/django/django/blob/master/django/contrib/admin/helpers.py#L83-L84
https://github.com/django/django/blob/master/django/contrib/admin/options.py#L639-L648
https://github.com/django/django/blob/master/django/contrib/admin/options.py#L2026-L2028
https://github.com/django/django/blob/master/django/contrib/admin/widgets.py#L446-L460
we see, that Django only serves the minimized version of the file, when not 
in DEBUG mode.

This inconsistent handling of assets can cause other problems for third 
party apps, if they follow the documentation as show in the first link.
If in their Media definition they refer to the minimized version of a 
Django asset, say 'admin/js/vendor/jquery/jquery.min.js', then 
the automatic sorting does not work anymore in DEBUG mode. If on the other 
side they refer to 'admin/js/vendor/jquery/jquery.js',
then automatic sorting fails in production. 

There are two solutions to this problem.

*Either*

The documentation explicitly states that when a Media definition refers to 
internal Django assets, one *must* distinguish between the
minimized and unminimized version, just as Django's internal Media 
definitions do (see code examples above).
This however can easily be forgotten and errors therefore become visible 
only after deployment.

*Or*

We rewrite the FileSystemFinder and AppDirectoriesFinder so that in DEBUG 
mode, Django looks if an unminimized version
of the same asset exists, and if so it then serves that. Doing that 
automatically is easy, because the general convention is, that the .min
infix is always placed immediately before the .js- or .css file extension.

In many of my projects I therefore use these two alternative finders, which 
do exactly that:

import os
from django.conf import settings
from django.contrib.staticfiles.finders import (FileSystemFinder as 
FileSystemFinderBase, AppDirectoriesFinder as AppDirectoriesFinderBase)


class FileSystemFinder(FileSystemFinderBase):
"""
In debug mode, serve /static/any/asset.min.ext as /static/any/asset.ext
"""
locations = []
serve_unminimized = getattr(settings, 'DEBUG', False)

def find_location(self, root, path, prefix=None):
if self.serve_unminimized:
# search for the unminimized version, and if it exists, return it
base, ext = os.path.splitext(path)
base, minext = os.path.splitext(base)
if minext == '.min':
unminimized_path = super().find_location(root, base + ext, 
prefix)
if unminimized_path:
return unminimized_path
# otherwise proceed with the given one
path = super().find_location(root, path, prefix)
return path


class AppDirectoriesFinder(AppDirectoriesFinderBase):
serve_unminimized = getattr(settings, 'DEBUG', False)

def find_in_app(self, app, path):
matched_path = super().find_in_app(app, path)
if matched_path and self.serve_unminimized:
base, ext = os.path.splitext(matched_path)
base, minext = os.path.splitext(base)
if minext == '.min':
storage = self.storages.get(app, None)
path = base + ext
if storage.exists(path):
path = storage.path(path)
if path:
return path
return matched_path


In my opinion this approach makes Django's internal and 3rd party package 
code more readable, because it removes the clutter of
case distinction between DEBUG and production mode for each referred 
JavaScript and CSS asset. It also might be less error prone.


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/dd904eeb-a877-48d3-81fa-e72fe71a6735%40googlegroups.com.

Re: Deadlock bug in logging? Reproducible case

[Adam Johnson]

Brian - You didn't describe the exact symptoms you're seeing. "Deadlock"
has a particular technical meaning around multiple processes requesting
locks mutually off each other, and normally ends in one process being
terminated. But I presume you're experiencing more of a "standstill" and
future requests no longer work?

Additionally you don't describe your Python version. Since logging is part
of the Python standard library (and you're not using any extensions) any
logging issues you're seeing are part of Python, and may be fixed on later
versions. Minor versions contain lots of such fixes and Django only
officially supports the latest in each series.

Steven has managed to test with 1000's of requests and I think that's an
indicator it could be a Python bug.

Also Steven is right, writing to disk is expensive and something to avoid
in production (at scale). In production I've previously used two high
quality setups:

   1. Logging to stdout/stderr, running the server under systemd, and
   letting it capture stdout logs into journald
   2. Logging directly into a running fluentd process

Both logging agents - journald and fluentd - provide in-memory buffering
before disk, discarding of extreme bursts, etc. Journald is easier to get
started with since it's included in most linuxes, fluentd is more flexible
and easy to configure. There are many others out there too!

And if you use a PaaS like Heroku, they normally capture your stdout/stderr
into such a logging agent already.

Hope that helps,

Adam

On Wed, 22 Apr 2020 at 19:47, Steven Mapes  wrote:

> This is more an issue at the file system level and the hardware not being
> able to keep up rather than anything Django is doing.  It's the same
> fundamental principal for why you may turn off webserver logging to
> increase performance, writing to disk is expensive and so when dealing with
> high throughout you probably want to question why.
>
> I've just run that on my laptop and hit your 1000 requests fine. I
> actually them bumped it up to run 6 requests per iteration and still was
> fine writing to disk. I started to have network resource issues when I
> bumped it up further but that was more limitations within chrome
>
> In the real world if you were writing logs like this for information or
> debugging you would be best served to defer them by writing to a message
> queue of sort some, probably FIFO based as its logging, and then have as
> many subscribers as you want dealing with reading and processing those
> queued messages. This could be as few as one so that you never hit a
> deadlock or it could be a few, again it depends on the performance of the
> underlying hardware, what you are writing and where (database, disk,
> memory).
>
>
> On Wednesday, 22 April 2020 18:23:31 UTC+1, Brian Tiemann wrote:
>>
>> Hi all,
>>
>> I was directed here after getting corroboration on #django and elsewhere.
>>
>> I have what appears to be a bug in which a Django app can deadlock if you
>> hit it with a lot (>3) of AJAX requests within a short time (i.e. all
>> triggered in parallel from a single HTML page). I have a reproducible case
>> here:
>>
>> https://github.com/data-graham/wedge
>>
>> I'd really appreciate it if someone could take a look and let me know
>> whether I'm doing something wrong, or if there's something systemic going
>> on. Thanks much!
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/1445d05d-eefc-43b1-8913-79d6e55de216%40googlegroups.com
> 
> .
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM3dM2B%2BhKTVEgUA458G0FO%2BWhU1%3Dy_VDY5P7e2OTis1mw%40mail.gmail.com.