[issue23120] installation order of 32bit and 64bit python seems to matter
New submission from Peter Santoro: It appears that installation order matters when installing both 32bit and 64bit versions of Python. If you install the 32bit version first, the 64bit version will uninstall the 32bit version. Here are the steps I used: 1. Starting point (Windows 7 64bit with latest MS patches): Python 3.3.5 32bit (c:\bin32\python33, required for support of older systems that will be upgraded to 3.4.3 when it's available) Python 3.4.1 32bit (c:\bin32\python34) Python 3.4.1 64bit (c:\bin\python34, included in PATH) 2. Uninstalled Python 3.4.1 32bit - ran as expected Python 3.4.1 64bit - ran as expected 3. Attempted Install (32bit install before 64bit install) Python 3.4.2 32bit (c:\bin32\python34) - ran as expected Python 3.4.2 64bit (c:\bin\python34) - install dialog had red warning message at top stating that previous version will be uninstalled; if you continue, the 64bit install deletes the Python 3.4.2 32bit install 4. Work around installation order (64bit install before 32bit install): Python 3.4.2 64bit (c:\bin\python34) - ran as expected Python 3.4.2 32bit (c:\bin32\python34) - ran as expected Notes: I've reported other strange issues with multiple versions of python installed, but they were deemed different than the original reported bug and were not further discussed. It's quite possible that the combination of python versions installed/uninstalled over time on this system has caused some bad/unexpected/untested state (e.g. Windows registry entries). -- components: Installation messages: 233131 nosy: pe...@psantoro.net priority: normal severity: normal status: open title: installation order of 32bit and 64bit python seems to matter type: behavior versions: Python 3.4 ___ Python tracker <http://bugs.python.org/issue23120> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21427] installer not working
Peter Santoro added the comment: I believe I may have hit a related issue yesterday. I'm using Python 3.3.5 (32 and 64 bit) and 3.4.1 (32 and 64 bit) releases all on the same Windows 7SP1/64bit PC (patched with latest MS updates). The Tkinter applications that I wrote and have been using with 3.x for a while now stopped working sometime after installing 3.4.1 - I only noticed this behavior yesterday. All the Tkinter apps run fine as .py files (but with an associated console window); however, they no longer run as .pyw files. Instead, they silently fail. I checked my file associations/types and I believe they are correct. The py.exe and pyw.exe files are in the Windows directory. None of my colleagues have this issue, but they only have one Python installation (3.4.1 64bit). My users also only have one Python version installed (3.3.5 or 3.4.1) and they are not experiencing this issue. I did search for a empty file on the path with the same name as the python script that I was trying to execute, as this will lead to silent failures - but I found none. I also tried using the Python Windows installer repair facility on the Python 3.4.1 64bit install, but that didn't help. I then uninstalled and reinstalled my Python 3.4.1 64bit release, but that didn't help either. I did notice that the py.exe and pyw.exe files in my Windows directory were not identical to a colleague's Python 3.4.1 64bit PC. I then ran pyw.exe via windbg (e.g. windbg pyw.exe -3.4 irtool.pyw, windbg pyw.exe -3.4-32 irtool.pyw) to see if there were obvious errors. I discovered that my pyw.exe silently fails whenever it is instructed to use the 3.4.1 release. However, when I specified Python 3.3.5 (pyw.exe -3.3 or -3.3-32) my Tkinter applications ran fine and as expected, without an associated console window. I'm reasonably certain that I did not intentionally modify the py.exe and pyw.exe files in my Windows directory. Also, all my scripts start with the following: #!/usr/bin/python3 Unfortunately, I'm not able to uninstall 3.3.5, until we upgrade all users to 3.4.x (probably shortly after 3.4.2 is released). -- nosy: +pe...@psantoro.net ___ Python tracker <http://bugs.python.org/issue21427> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20907] behavioral differences between shutil.unpack_archive and ZipFile.extractall
Peter Santoro added the comment: It seems clear to me that the logic in shutil._unpack_zipfile that silently skips paths that start with '/' (indicates absolute path) or that contain references to the parent directory ('..') was added to prevent malicious zip files from making potential malicious/unwanted modifications to the filesystem (perhaps at a time when zipfile did not itself contain such logic). This conservative approach works, but it can have unexpected results. For example, if all entries in a zip file contain these invalid characters, then shutil._unpack_zipfile appears to do nothing (i.e. the zip file is not unpacked). This is good (except for the silent part), if the zip file is truly malicious. However, I recently had to deal with thousands of zip files created by well known software vendors where hundreds of the zip files were created incorrectly and contained these invalid characters. These files were not malicious, but they were created improperly. Note that shutil._unpack_zipfile silently fai led to unzip these files, but by using ZipFile.extractall I could unzip them. It appears that most unzipping software today either either ignores (sometimes silently) potentially malicious zip entries (e.g. Windows 7 Explorer displays an invalid zip file error) or it attempts to filter out/replace known bad characters so that the zip entries can be extracted (e.g. WinZip, gnu unzip). I created this issue because the Python library uses both approaches, which may need rethinking. The newer logic in ZipFile._extract_member, which is used by ZipFile.extractall, takes a different approach. Instead of silently ignoring potentially malicious zip entries, it attempts to filter out or replace known invalid characters before extracting the zip entries. >From the Python zipfile docs: --- If a member filename is an absolute path, a drive/UNC sharepoint and leading (back)slashes will be stripped, e.g.: ///foo/bar becomes foo/bar on Unix, and C:\foo\bar becomes foo\bar on Windows. And all ".." components in a member filename will be removed, e.g.: ../../foo../../ba..r becomes foo../ba..r. On Windows illegal characters (:, <, >, |, ", ?, and *) replaced by underscore (_). --- As ZipFile._extract_member filters out/replaces more invalid characters than shutil._unpack_zipfile handles, one could argue that the (apparent older) approach used by shutil._unpack_zipfile is less safe. The approaches used by shutil._unpack_zipfile and ZipFile.extractall to deal with potentially malicious zip file entries are different. This issue could be closed if not deemed important by the Python core developers or it could be handled by documentation and/or coding changes. -- ___ Python tracker <http://bugs.python.org/issue20907> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20907] behavioral differences between shutil.unpack_archive and ZipFile.extractall
Peter Santoro added the comment: I've attached a zip file which contains a test script and test zip files for the previously submitted Python 3.3.5 patch. See the included README.txt for more information. To view the contents of the included bad.zip file, use the following command: > unzip -l bad.zip -- Added file: http://bugs.python.org/file34408/test_unpack.zip ___ Python tracker <http://bugs.python.org/issue20907> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20907] behavioral differences between shutil.unpack_archive and ZipFile.extractall
New submission from Peter Santoro: Since Python 3.3.1, ZipFile.extractall was enhanced to better handle absolute paths and illegal characters. The associated logic within shutil._unpack_zipfile essentially skips zip members with these issues. If a zip file contains all absolute paths, ZipFile.extractall works as expected (i.e. the zip file is unpacked), but shutil._unpack_zipfile (normally called indirectly via shutil.unpack_archive) appears to do nothing (i.e. it silently fails to unpack the zip file). The attached patch attempts to unify the behavior of extracting zip files between shutil.unpack_archive with ZipFile.extractall. -- components: Library (Lib) files: shutil.diff keywords: patch messages: 213374 nosy: pe...@psantoro.net priority: normal severity: normal status: open title: behavioral differences between shutil.unpack_archive and ZipFile.extractall type: behavior versions: Python 3.3, Python 3.4 Added file: http://bugs.python.org/file34393/shutil.diff ___ Python tracker <http://bugs.python.org/issue20907> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18577] lru_cache enhancement: lru_timestamp helper function
Peter Santoro added the comment: As requested, I published this for review on http://code.activestate.com/recipes/578817-lru_timestamp-cache-entry-aging-for-functoolslru_c/ -- ___ Python tracker <http://bugs.python.org/issue18577> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20102] shutil._make_zipfile possible resource leak
New submission from Peter Santoro: Now that zipfile.ZipFile supports the context manager protocol, shouldn't shutil._make_zipfile make use of it to avoid the possibility of the archive file not being closed properly if an exception occurs? It should be noted that shutil._unpack_zipfile does use try/finally to ensure that files are closed. -- components: Library (Lib) files: shutil.diff keywords: patch messages: 207132 nosy: pe...@psantoro.net priority: normal severity: normal status: open title: shutil._make_zipfile possible resource leak type: behavior versions: Python 3.3, Python 3.4, Python 3.5 Added file: http://bugs.python.org/file33291/shutil.diff ___ Python tracker <http://bugs.python.org/issue20102> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18577] lru_cache enhancement: lru_timestamp helper function
Peter Santoro added the comment: I updated my proposed lru_timestamp function with the following change: 1) raise TypeError instead of ValueError -- Added file: http://bugs.python.org/file31079/lru.py ___ Python tracker <http://bugs.python.org/issue18577> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18577] lru_cache enhancement: lru_timestamp helper function
Peter Santoro added the comment: I updated my proposed lru_timestamp function with the following changes: 1) restricted refresh_interval to int type 2) updated doc string Updated doc string follows: def lru_timestamp(refresh_interval=60): """ Return a timestamp string for @lru_cache decorated functions. The returned timestamp is used as the value of an extra parameter to @lru_cache decorated functions, allowing for more control over how often cache entries are refreshed. The lru_timestamp function should be called with the same refresh_interval value for a given @lru_cache decorated function. The returned timestamp is for the benefit of the @lru_cache decorator and is normally not used by the decorated function. Positional arguments: refresh_interval -- in minutes (default 60), values less than 1 are coerced to 1, values more than 1440 are coerced to 1440 """ -- Added file: http://bugs.python.org/file31064/lru.py ___ Python tracker <http://bugs.python.org/issue18577> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18577] lru_cache enhancement: lru_timestamp helper function
New submission from Peter Santoro: The attached proposed lru_timestamp function provides developers with more control over how often lru_cache entries are refreshed. Doc string follows: def lru_timestamp(refresh_interval=60): """ Return a timestamp string for @lru_cache decorated functions. The returned timestamp is used as the value of an extra parameter to @lru_cache decorated functions, allowing for more control over how often cache entries are refreshed. The lru_timestamp function should be called with the same refresh_interval value for a given lru_cache decorated function. Positional arguments: refresh_interval -- 1-1440 minutes (default 60) as int or float """ Rationale: Some functions have input parameters that rarely change, but yet return different results over time. It would be nice to have a ready-made solution to force lru_cache entries to be refreshed at specified time intervals. An common example is using a stable userid to read user information from a database. By itself, the lru_cache decorator can be used to cache the user information and prevent unnecessary i/o. However, if a given user's information is updated in the database, but the associated lru_cache entry has not yet been discarded, the application will be using stale data. The lru_timestamp function is a simple, ready-made helper function that gives the developer more control over the age of lru_cache entries in such situations. Sample usage: @functools.lru_cache() def user_info(userid, timestamp): # expensive database i/o, but value changes over time # the timestamp parameter is normally not used, it is # for the benefit of the @lru_cache decorator pass # read user info from database, if not in cache or # older than 120 minutes info = user_info('johndoe', functools.lru_timestamp(120)) -- components: Library (Lib) files: lru.py messages: 193820 nosy: pe...@psantoro.net priority: normal severity: normal status: open title: lru_cache enhancement: lru_timestamp helper function type: enhancement versions: Python 3.3 Added file: http://bugs.python.org/file31063/lru.py ___ Python tracker <http://bugs.python.org/issue18577> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18224] pyvenv pydoc.py script causing AttributeErrors on Windows
Peter Santoro added the comment: As requested, I've attached a small test script called shadow.py. Steps to reproduce: 1) pyvenv.py bugtest 2) copy the attached shadow.py script to bugtest and bugtest\scripts 3) cd bugtest 4) run shadow.py (first entry in sys.path is refers to bugtest directory per Python docs; finds the systems's pydoc module) 5) run bugtest\shadow.py (first entry in sys.path refers to bugtest\scripts directory per Python docs; finds the bugtest\scripts pydoc module instead of the system's pydoc module) According to the Python documentation (http://docs.python.org/3/library/sys.html#sys.path): "As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter. If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH." Maybe I'm missing something here, but isn't this problem caused by the fact that Python initializes sys.path[0] to contain the directory of the executing script and that having pydoc.py in that same directory (i.e. the venv's scripts directory) shadows the system's pydoc.py module? On Linux, I didn't have this problem, because the pydoc script doesn't have the .py extension. However, if you rename the pydoc script on Linux to pydoc.py, the same problem occurs. I don't think a pydoc.py (or any other .py file which shadows a system module) can exist in the venv scripts (or bin) directory without shadowing/breaking the system provided module. Maybe a pydoc.exe or pydoc.bat file is needed on Windows? Another option would be to rename the pydoc.py file to something like pydocs.py, but that would be incompatible with other platforms and the existing documentation. -- Added file: http://bugs.python.org/file30620/shadow.py ___ Python tracker <http://bugs.python.org/issue18224> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18224] pyvenv pydoc.py script causing AttributeErrors on Windows
New submission from Peter Santoro: I've recently hit an issue with pyvenv in Python 3.3.2 that is causing AttributeErrors in other packages on Windows (see https://groups.google.com/forum/?fromgroups#!topic/pylons-discuss/FpOSMDpdvy4). Here's what I believe is going on: On Windows, the pyvenv pydoc script has a .py extension - so import finds it instead of the system's pydoc module. On Linux, the pyvenv pydoc script doesn't have an extension - so import finds the system's pydoc module. I believe the Windows pyvenv pydoc.py script should be renamed to something like pydocs.py to prevent AttributeErrors. -- components: Windows messages: 191210 nosy: pe...@psantoro.net priority: normal severity: normal status: open title: pyvenv pydoc.py script causing AttributeErrors on Windows type: behavior versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue18224> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com