[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Terry J. Reedy added the comment: Phillipe: I was the first to comment, but had no futher involvement with this issue until now. Multiple people, including coredevs, considered the old behavior to be buggy and multiple people, including coredevs, contributed to the fix. You are unlikely

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Philippe Ombredanne added the comment: Actually this is problematic on multiples counts: 1. the behaviour changes and this is a regression 2. even if that new buggy behaviour was the one to use, it should not give preference to knownfiles ovr init-provided files, but at least take the

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Philippe Ombredanne added the comment: The changes introduced by this ticket in https://github.com/python/cpython/commit/9fc720e5e4f772598013ea48a3f0d22b2b6b04fa#r45794801 are problematic. I discovered this from having tests failing when testing on Python 3.7 and up The bug is that calling

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Change by Steve Dower : -- assignee: -> steve.dower resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Steve Dower added the comment: New changeset 2a99fd911ebeecedbb250a05667cd46eca4735b9 by Steve Dower in branch '3.7': bpo-4963: Fix for initialization and non-deterministic behavior issues in mimetypes (GH-14376)

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David K. Hess added the comment: Thank you Steve! Nice to see this one make it across the finish line. -- ___ Python tracker ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Steve Dower added the comment: New changeset 25fbe33b92cd938e809839feaa3fda97e6ad0980 by Steve Dower in branch '3.8': bpo-4963: Fix for initialization and non-deterministic behavior issues in mimetypes (GH-14375)

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Change by Steve Dower : -- pull_requests: +14190 stage: backport needed -> patch review pull_request: https://github.com/python/cpython/pull/14375 ___ Python tracker ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Change by Steve Dower : -- pull_requests: +14191 pull_request: https://github.com/python/cpython/pull/14376 ___ Python tracker ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Steve Dower added the comment: Sorry for the delays! I've merged into master. The backports (3.7, 3.8) are going to need some help, so I'll take a look at them soon unless someone else gets there first. -- stage: needs patch -> backport needed versions: +Python 3.8, Python 3.9

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Steve Dower added the comment: New changeset 9fc720e5e4f772598013ea48a3f0d22b2b6b04fa by Steve Dower (David K. Hess) in branch 'master': bpo-4963: Fix for initialization and non-deterministic behavior issues in mimetypes (GH-3062)

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Change by Abhilash Raj : -- nosy: +maxking ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Christian Heimes added the comment: The PR has a merge conflict. -- nosy: +christian.heimes ___ Python tracker ___ ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Change by Sascha Silbe : -- nosy: +sascha_silbe ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David K. Hess added the comment: Are there any committers watching this issue that are able to review the PR? https://github.com/python/cpython/pull/3062 It's close to 6 months old now with no action on it. I'm willing to help but doing so and then having the PR gather

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Change by Bert JW Regeer : -- nosy: +X-Istence ___ Python tracker ___ ___ Python-bugs-list

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David K. Hess added the comment: FYI, PR opened: https://github.com/python/cpython/pull/3062 -- ___ Python tracker ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Changes by David K. Hess : -- pull_requests: +3096 ___ Python tracker ___ ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David K. Hess added the comment: Pushed more commits so here's a branch compare: https://github.com/python/cpython/compare/master...davidkhess:fix-issue-4963 -- ___ Python tracker

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David K. Hess added the comment: Ok, I followed @r.david.murray's advice and decided to take a shot at this. First, I noticed that I couldn't reproduce the non-deterministic behavior that I reported above on the latest code (i.e. pre-3.7). After doing some research it appears this was the

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Martin Panter added the comment: I understand hash randomization was added after this bug was opened. Here is a demonstration with “video/mp4”, which only has the extension “.mp4” built in. But my /etc/mime.types file lists “mp4 mp4v mpg4”, so after the second initialization the behaviour

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Martin Panter added the comment: I suggest to discuss the non-determinism problem in Issue 1043134 (about determining a canonical extension for each content type). I understood this bug (Issue 4963) is about the behaviour of repeated initialization of the same instance of mimetypes. BTW an

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Changes by R. David Murray : -- stage: test needed -> needs patch versions: +Python 3.6, Python 3.7 -Python 2.7, Python 3.2, Python 3.3, Python 3.4 ___ Python tracker

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

R. David Murray added the comment: @dhess: do you want to work on the OrderedDict + correctly ordered hardcoded lists solution? -- ___ Python tracker ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David Hess added the comment: And the underlying problem causing this: $ python -c 'import mimetypes;print(mimetypes.guess_all_extensions("image/jpeg"))' ['.jpeg', '.jpg', '.jpe'] $ python -c 'import mimetypes;print(mimetypes.guess_all_extensions("image/jpeg"))' ['.jpg', '.jpe', '.jpeg'] $

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

David Hess added the comment: Concur with @sivert – the result of guess_extension() is non-deterministic between mimetypes module initialization. $ python Python 3.4.3 (default, Nov 17 2016, 01:08:31) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information.

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Changes by Marcin Szewczyk : -- nosy: +wodny ___ Python tracker ___ ___ Python-bugs-list

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Julian Sivertsen added the comment: I bumped into a similar issue with mimetypes.guess_extension on Arch Linux 64-bit in February. The behavior is still present in python 3.4.3. $ python test.py .htm $ python test.py .html $ cat test.py from mimetypes import guess_extension

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

R. David Murray added the comment: OK, it is great having a test that makes this at least mostly reproducible :) Having reloaded my brain on this thing, I'm thinking that the best solution may be indeed to switch to ordered dicts. If we then reorder the hardcoded lists to be in preferred

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Toshio Kuratomi added the comment: Took a look at this and was able to reproduce it on Fedora Linux 20 and current cpython head. It is somewhat random though. I'm able to get reasonably consistent failures using image/jpeg and iterating the test case about 20 times. Additionally, it looks

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Wichert Akkerman added the comment: I can reproduce this on Both OSX 10.9 and Ubuntu 12.04: import mimetypes mimetypes.guess_extension('image/jpeg') '.jpe' mimetypes.init() mimetypes.guess_extension('image/jpeg') '.jpeg' The same thing happens for Python 3.4: Python 3.4.0rc3 (default, Mar

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

R. David Murray added the comment: I'd forgotten about this issue. I wonder if the dictionary randomization makes the problem worse. -- components: +email nosy: +barry versions: +Python 3.2, Python 3.3, Python 3.4 ___ Python tracker

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

S Arrowsmith si...@chiark.greenend.org.uk added the comment: That solution looks sound to me, in particular documenting the semantics of repeated init() calls! As for the underlying problem, it seems to me that an alternative to copying the existing structures rather than rebuilding them

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

R. David Murray rdmur...@bitdance.com added the comment: It must be that the different key order only happens on the one platform because of the quirky nature of dictionary construction. That is, there is *something* on that platform that is changing where things get hashed when the

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

S Arrowsmith si...@chiark.greenend.org.uk added the comment: I've dug into it -- again -- and my original analysis still holds. Getting consistent guess_extension() results across an explicit init() call depends on dict.items() returning keys in the same order on two different dictionaries

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

S Arrowsmith si...@chiark.greenend.org.uk added the comment: Sorry, still there: Python 2.7rc2 (r27rc2:82137, Jun 26 2010, 11:27:59) [GCC 4.3.2] on linux2 Type help, copyright, credits or license for more information. import mimetypes mimetypes.guess_extension('image/jpeg') '.jpe'

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Shashwat Anand anand.shash...@gmail.com added the comment: Can't reproduce. 16:36:36 l0nwlf-MBP:~$ python2.7 Python 2.7rc2+ (trunk:82148M, Jun 22 2010, 10:32:46) [GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin Type help, copyright, credits or license for more information. import

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

R. David Murray rdmur...@bitdance.com added the comment: S Arrowsmith: can put a print statement into mimetypes.init, find out what files are loading, and paste the image/jpeg lines from each of those files here? That might provide a clue. -- priority: normal - low stage: - unit

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

S Arrowsmith si...@chiark.greenend.org.uk added the comment: import mimetypes mimetypes.guess_extension('image/jpeg') /etc/mime.types '.jpe' mimetypes.init() /etc/mime.types mimetypes.guess_extension('image/jpeg') '.jpeg' $ grep jpeg /etc/mime.types image/jpeg

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Changes by Terry J. Reedy tjre...@udel.edu: -- versions: +Python 2.7 -Python 2.4, Python 2.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4963 ___

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

Antoine Pitrou pit...@free.fr added the comment: Can't reproduce under Mandriva Linux: import mimetypes print(mimetypes.guess_extension('image/jpeg')) .jpe mimetypes.init() print(mimetypes.guess_extension('image/jpeg')) .jpe The fact that it returns .jpe rather than .jpg, however, could be

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

New submission from S Arrowsmith si...@chiark.greenend.org.uk: Asking mimetypes to reload mime.types can cause guess_extension() to return a different result if multiple extensions are mapped to that mime type: import mimetypes mimetypes.guess_extension('image/jpeg') '.jpe' mimetypes.init()

[issue4963] mimetypes.guess_extension result changes after mimetypes.init()

S Arrowsmith si...@chiark.greenend.org.uk added the comment: Ah, yes, forgot to mention this is on Debian 4.0. I doubt you're going to run into it on a Windows system unless you explicitly give init() a mime.types file, looking at the knownfiles list used by default.