Re: [Python-Dev] [DLFILTER] Exporting Python functions on AIX
On 27 July 2018 at 20:23, Rob Boehne wrote: > Why would a VIM build refer to the export file for python? Because vim includes an optional embedded Python interpreter. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [DLFILTER] Exporting Python functions on AIX
Why would a VIM build refer to the export file for python? From: Python-Dev on behalf of "WILSON, MICHAEL" Date: Friday, July 27, 2018 at 10:27 AM To: "python-dev@python.org" Cc: "WILSON, MICHAEL" Subject: [DLFILTER] [Python-Dev] Exporting Python functions on AIX All, My excuse if this is not the appropriate list for a question essentially concerning the AIX port of Python. The current port of Python for AIX includes composing an export file (/lib/python2.7/config/python.exp) in which there are a number of functions starting “Py_” or “_Py_”. The Vim package for AIX is built referencing the python.exp file and unfortunately, when functions are removed from libpython, even those which are not called, the vim command detects missing symbols. The most recent case (May 2017), functions _Py_hgidentity, _Py_hgversion and _Py_svnversion were replaced/removed, see “bpo-27593: Get SCM build info from git instead of hg (#1327)”. Is it correct to assume that the “_Py_” functions are internal (Python name space) that should/must not be called by or made visible to application code ? Could you indicate a URL to the authoritative API documentation ? Thanks for your replies. Mike Wilson ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Exporting Python functions on AIX
Yes, all symbols starting with _Py are private and must not be used outside CPython internals. Victor Le vendredi 27 juillet 2018, WILSON, MICHAEL a écrit : > All, > > My excuse if this is not the appropriate list for a question essentially concerning the AIX port of Python. > > The current port of Python for AIX includes composing an export file (/lib/python2.7/config/python.exp) in which there are a number of functions starting “Py_” or “_Py_”. > > The Vim package for AIX is built referencing the python.exp file and unfortunately, when functions are removed from libpython, even those which are not called, the vim command detects missing symbols. > > The most recent case (May 2017), functions _Py_hgidentity, _Py_hgversion and _Py_svnversion were replaced/removed, see “bpo-27593: Get SCM build info from git instead of hg (#1327)”. > > Is it correct to assume that the “_Py_” functions are internal (Python name space) that should/must not be called by or made visible to application code ? > > Could you indicate a URL to the authoritative API documentation ? > > Thanks for your replies. > > Mike Wilson > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2018-07-20 - 2018-07-27) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open6740 ( +6) closed 39260 (+73) total 46000 (+79) Open issues with patches: 2681 Issues opened (50) == #34126: Profiling certain invalid calls crashes Python https://bugs.python.org/issue34126 reopened by vstinner #34157: NamedTemporaryFile can leave temporary files behind https://bugs.python.org/issue34157 reopened by r.david.murray #34171: Lib/trace.cover not removed by the clean target https://bugs.python.org/issue34171 opened by doko #34172: multiprocessing.Pool and ThreadPool leak resources after being https://bugs.python.org/issue34172 opened by tzickel #34173: [3.7] deadlock in /usr/lib/python3.7/concurrent/futures/thread https://bugs.python.org/issue34173 opened by corey.bryant #34176: Asyncio StreamReader fails to close Transport https://bugs.python.org/issue34176 opened by JDLH #34178: test_tcl fails on the 3.7 branch https://bugs.python.org/issue34178 opened by doko #34182: Lib/test/test_pydoc.py failed when ran as a script https://bugs.python.org/issue34182 opened by serhiy.storchaka #34185: Lib/test/test_bdb.py failed when ran as a script https://bugs.python.org/issue34185 opened by serhiy.storchaka #34187: Issues with lazy fd support in _WindowsConsoleIO fileno() and https://bugs.python.org/issue34187 opened by eryksun #34188: Allow dict choices to "transform" values in argpagse https://bugs.python.org/issue34188 opened by porton #34191: argparse: Missing subparser error message should be more clear https://bugs.python.org/issue34191 opened by porton #34192: FunctionType.__new__ can generate functions that immediately c https://bugs.python.org/issue34192 opened by bup #34193: Fix pluralization in TypeError messages in getargs.c https://bugs.python.org/issue34193 opened by xtreak #34194: test_ssl, AIX, and defaults for _ssl connections https://bugs.python.org/issue34194 opened by Michael.Felt #34198: Additional encoding options to tkinter.filedialog https://bugs.python.org/issue34198 opened by narito #34199: Add support for delete logger in log module. https://bugs.python.org/issue34199 opened by chetankolhe #34200: importlib: python -m test test_pkg -m test_7 fails randomly https://bugs.python.org/issue34200 opened by vstinner #34203: documentation: recommend Python 3 over 2 in faq https://bugs.python.org/issue34203 opened by abcdef #34204: Bump the default pickle protocol in shelve https://bugs.python.org/issue34204 opened by serhiy.storchaka #34205: Ansible: _PyImport_LoadDynamicModuleWithSpec() crash on an inv https://bugs.python.org/issue34205 opened by mdk #34206: Move and clarify Py_Main documentation https://bugs.python.org/issue34206 opened by ncoghlan #34207: test_cmd_line test_utf8_mode test_warnings fail in AMD64 FreeB https://bugs.python.org/issue34207 opened by pablogsal #34211: Cygwin build broken due to use of &PyType_Type in static decla https://bugs.python.org/issue34211 opened by erik.bray #34212: Cygwin link failure with builtin modules since issue30860 https://bugs.python.org/issue34212 opened by erik.bray #34213: Frozen dataclass __init__ fails for "object" property" https://bugs.python.org/issue34213 opened by Omenien #34214: Pylint recusion stack overflow abort https://bugs.python.org/issue34214 opened by nickdrozd #34215: streams.py:readuntil IncompleteReadError is message is incorre https://bugs.python.org/issue34215 opened by mrbell...@gmail.com #34216: python platform no child error https://bugs.python.org/issue34216 opened by sskamble619 #34219: distutils: build_ext -D wrongly assume defining a symbol with https://bugs.python.org/issue34219 opened by monson #34222: Email message serialization enters an infinite loop when foldi https://bugs.python.org/issue34222 opened by altvod #34223: PYTHONDUMPREFS=1 ./python -c pass does crash https://bugs.python.org/issue34223 opened by vstinner #34224: python 3.7 inside venv tries to write back to read-only instal https://bugs.python.org/issue34224 opened by ajung #34226: cgi.parse_multipart() requires undocumented CONTENT-LENGTH in https://bugs.python.org/issue34226 opened by yan12125 #34231: PYTHONBREAKPOINT is not documented with python --help https://bugs.python.org/issue34231 opened by matrixise #34232: Python3.7.0 exe installers (32 and 64 bit) failing on Windows7 https://bugs.python.org/issue34232 opened by wolma #34234: Use _PyAnyInt_Check() and _PyAnyInt_CheckExact() in 2.7 https://bugs.python.org/issue34234 opened by serhiy.storchaka #34235: PyArg_ParseTupleAndKeywords: support required keyword argument https://bugs.python.org/issue34235 opened by serhiy.storchaka #34236: Test6012 in test_capi is not run as part of make test https://bugs.python.org/issue34236 opened by xtreak #34238: Wh
[Python-Dev] Exporting Python functions on AIX
All, My excuse if this is not the appropriate list for a question essentially concerning the AIX port of Python. The current port of Python for AIX includes composing an export file (/lib/python2.7/config/python.exp) in which there are a number of functions starting "Py_" or "_Py_". The Vim package for AIX is built referencing the python.exp file and unfortunately, when functions are removed from libpython, even those which are not called, the vim command detects missing symbols. The most recent case (May 2017), functions _Py_hgidentity, _Py_hgversion and _Py_svnversion were replaced/removed, see "bpo-27593: Get SCM build info from git instead of hg (#1327)". Is it correct to assume that the "_Py_" functions are internal (Python name space) that should/must not be called by or made visible to application code ? Could you indicate a URL to the authoritative API documentation ? Thanks for your replies. Mike Wilson ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tests failing on Windows with TESTFN
On Fri, Jul 27, 2018 at 4:48 PM Chris Jerdonek wrote:
>
> On Fri, Jul 27, 2018 at 6:41 AM, Giampaolo Rodola' wrote:
> >
> > Being TESTFN a global name it appears not suited for parallel testing.
>
> It was designed for parallel testing though:
>
> # Disambiguate TESTFN for parallel testing, while letting it remain a valid
> # module name.
> TESTFN = "{}_{}_tmp".format(TESTFN, os.getpid())
>
> https://github.com/python/cpython/blob/aee632dfbb0abbc0d2bcc988c43a736afd568c55/Lib/test/support/__init__.py#L807-L809
Oh, nice, I didn't notice that, sorry.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tests failing on Windows with TESTFN
On Fri, Jul 27, 2018 at 6:41 AM, Giampaolo Rodola' wrote:
>
> Being TESTFN a global name it appears not suited for parallel testing.
It was designed for parallel testing though:
# Disambiguate TESTFN for parallel testing, while letting it remain a valid
# module name.
TESTFN = "{}_{}_tmp".format(TESTFN, os.getpid())
https://github.com/python/cpython/blob/aee632dfbb0abbc0d2bcc988c43a736afd568c55/Lib/test/support/__init__.py#L807-L809
> Windows makes this more noticeable than POSIX as it's more strict, but
> accessing the same file from 2 unit tests is technically a problem
> regardless of the platform. I don't think that means we should ditch
> TESTFN in favor of tempfile.mktemp() though. Instead the file cleanup
> functions (support.unlink() and support.rmtree()) may be more clever
> and (important) they should always be used in setUp / tearDown. For
> instance, the traceback you pasted refers to a test class which
> doesn't do this.
The "test_file" test method referenced in the traceback calls
os.remove(TESTFN) in finally blocks preceding its calls to
open(TESTFN, "wb"), and inspecting the method shows that it must have
been able to open TESTFN earlier in the method (the same test method
uses TESTFN multiple times):
https://github.com/python/cpython/blob/aee632dfbb0abbc0d2bcc988c43a736afd568c55/Lib/test/test_urllib2.py#L811-L830
So I think one should investigate what can be causing the error / how
it can be happening.
TESTFN uses the pid of the process, so it doesn't seem like another
test case could be interfering and opening the same TESTFN while the
"test_file" test method is running. On Stack Overflow, there are some
comments suggesting that in some cases os.remove() doesn't always
complete right away (e.g. because of anti-malware software briefly
holding a second reference).
--Chris
>
> In psutil I've had occasional Windows failures like this for years
> then I got tired of it and came up with this:
> https://github.com/giampaolo/psutil/blob/1b09b5fff78f705dfb42458726ff9789c26f6f21/psutil/tests/__init__.py#L686
> ...which basically aggressively retries os.unlink or shutil.rmtree for
> 1 sec in case of (any) error, and I haven't had this problem since
> then.
>
> I suppose test.support's unlink() and rmtree() can do something
> similar, maybe just by using a better exception handling, and all unit
> tests should use them in setUp / tearDown. I think this will diminish
> the occasional failures on Windows, although not completely.
>
> --
> Giampaolo - http://grodola.blogspot.com
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tests failing on Windows with TESTFN
On Thu, Jul 26, 2018 at 12:16 AM Tim Golden wrote: > > I'm just easing back into core development work by trying to get a > stable testing environment for Python development on Windows. > > One problem is that certain tests use support.TESTFN (a local directory > constructed from the pid) for output files etc. However this can cause > issues on Windows when recreating the folders / files for multiple > tests, especially when running in parallel. > > Here's an example on my laptop deliberately running 3 tests with -j0 > which I know will generate an error about one time in three: > > C:\work-in-progress\cpython>python -mtest -j0 test_urllib2 test_bz2 > test_importlib > > Running Debug|Win32 interpreter... > Run tests in parallel using 6 child processes > 0:00:23 [1/3/1] test_urllib2 failed > test test_urllib2 failed -- Traceback (most recent call last): >File "C:\work-in-progress\cpython\lib\test\test_urllib2.py", line > 821, in test_file > f = open(TESTFN, "wb") > PermissionError: [Errno 13] Permission denied: '@test_15564_tmp' > > Although these errors are both intermittent and fairly easily spotted, > the effect is that I rarely get a clean test run when I'm applying a patch. > > I started to address this years ago but things stalled. I'm happy to > pick this up again and have another go, but I wanted to ask first > whether there was any objection to my converting tests to using tempfile > functions which should avoid the problem? > > TJG >From my experience open() raising PermissionDenied on Windows often means "file is already in use" as I think is this case. The same issue exists for directories: https://bugs.python.org/issue33240. Being TESTFN a global name it appears not suited for parallel testing. Windows makes this more noticeable than POSIX as it's more strict, but accessing the same file from 2 unit tests is technically a problem regardless of the platform. I don't think that means we should ditch TESTFN in favor of tempfile.mktemp() though. Instead the file cleanup functions (support.unlink() and support.rmtree()) may be more clever and (important) they should always be used in setUp / tearDown. For instance, the traceback you pasted refers to a test class which doesn't do this. In psutil I've had occasional Windows failures like this for years then I got tired of it and came up with this: https://github.com/giampaolo/psutil/blob/1b09b5fff78f705dfb42458726ff9789c26f6f21/psutil/tests/__init__.py#L686 ...which basically aggressively retries os.unlink or shutil.rmtree for 1 sec in case of (any) error, and I haven't had this problem since then. I suppose test.support's unlink() and rmtree() can do something similar, maybe just by using a better exception handling, and all unit tests should use them in setUp / tearDown. I think this will diminish the occasional failures on Windows, although not completely. -- Giampaolo - http://grodola.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 576/579/580 benchmark: mistune
Hello all, since my latest benchmark for PEP 580 [1] involved SageMath, which is quite a big project, I instead propose a much simpler benchmark involving mistune. mistune [2] is a Markdown parser implemented in the Python language. It optionally allows Cython compilation. It doesn't use any kind of optimization beyond that, but I created a branch [3] to use extension types instead of Python classes. Cython can either use built-in functions/methods or a custom class (which is not optimized but which would be optimized with PEP 580). I benchmarked mistune with custom classes [3] (binding=True, the default) and with built-in functions/methods [4] (binding=False). This is the median time of 5 runs: Binding=True: 9.063s Binding=False: 8.658s So this shows again that PEP 580 improves performance in actual real-world use cases. Jeroen. [1] https://mail.python.org/pipermail/python-dev/2018-July/154740.html [2] https://github.com/lepture/mistune [3] https://github.com/jdemeyer/mistune/tree/cython_pxd [4] https://github.com/jdemeyer/mistune/tree/cython_pxd_nobinding ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
