Change by Antti Haapala :
--
title: functools.cached_property locking is plain wrong. ->
functools.cached_property incorrectly locks the entire descriptor on class
instead of per-instance locking
___
Python tracker
<https://bugs.pyth
Antti Haapala added the comment:
I've been giving thought to implementing the locking on the instance or per
instance instead, and there are bad and worse ideas like inserting per
(instance, descriptor) into the instance `__dict__`, guarded by the
per-descriptor lock; using a per-descriptor
Antti Haapala added the comment:
Django was going to replace their cached_property by the standard library one
https://code.djangoproject.com/ticket/30949
--
___
Python tracker
<https://bugs.python.org/issue43
New submission from Antti Haapala :
The locking on functools.cached_property
(https://github.com/python/cpython/blob/87f649a409da9d99682e78a55a83fc43225a8729/Lib/functools.py#L934)
as it was written is completely undesirable for I/O bound values, parallel
processing. Instead of protecting
New submission from Antti Haapala :
The current documentation of `thread.local` is
Thread-Local Data
Thread-local data is data whose values are thread specific. To manage
thread-local data, just create an instance of local (or a subclass) and store
attributes on it:
mydata
Antti Haapala added the comment:
... to further clarify, it is disappointing that either BytesIO or
TemporaryFile would work alone, but the one that merges these two doesn't.
--
___
Python tracker
<https://bugs.python.org/issue26
Antti Haapala added the comment:
Another test case:
import tempfile
import io
import json
with tempfile.SpooledTemporaryFile(max_size=2**20) as f:
tf = io.TextIOWrapper(f, encoding='utf-8')
json.dump({}, fp=tf)
I was writing json to a file-like object that I need to read
Antti Haapala added the comment:
The messages above seem to be a (quite likely a machine) translation of André's
comment with a spam link to a paint ad site, so no need to bother to translate
it.
Also, I invited Hiếu to the nosy list in case this patch needs some info that
requires a native
Antti Haapala added the comment:
Unsigned long long needs to be at least 64 bits wide, so it is probably all
32-bit platforms and 64-bit window at least. These functions are not used only
in a few places within the CPython code and when they are they're guarded with
`PyLong_Check`s
Antti Haapala added the comment:
Victor, as a friendly reminder, (unsigned long)-1 is not necessarily the same
number as (unsigned long long)-1. The documentation means the latter.
--
___
Python tracker
<https://bugs.python.org/issue37
New submission from Antti Haapala :
Hi, while checking the longobject implementation for a Stack Overflow answer, I
noticed that the functions `_PyLong_AsUnsignedLongLongMask` and
`PyLong_AsUnsignedLongLongMask` erroneously return `(unsigned long)-1` on error
when bad internal call is thrown
Antti Haapala added the comment:
This should be added to `asctime` too.
The space-padded behaviour complies with the C standard which was the intent -
after all, before it was using C `asctime` directly, and says that unlike C
asctime, it doesn't have the newline character, meaning
Antti Haapala added the comment:
C11 specifies the format used by asctime as being exactly
"%.3s %.3s%3d %.2d:%.2d:%.2d %d\n",
which matches the *new* output with space padding, less the newline.
As always, Microsoft got it wrong:
https://docs.microsoft.com/en-us/cpp/c-runti
Antti Haapala added the comment:
This week we were bit by this in production. I foolishly thought that the docs
would give me correct default values... It is worse that it didn't actually
occur until we went over the limit.
--
nosy: +ztane
Antti Haapala <an...@haapala.name> added the comment:
Well, there's only one problem with casting to void *: while converting the
function pointer to another *is* standard-compliant, and GCC is being just
hypersensitive here, casting a function pointer to void * isn't, though it is a
Antti Haapala <an...@haapala.name> added the comment:
Took 2 seconds.
% sudo python3 -mpip --version
hello world
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name
Antti Haapala <an...@haapala.name> added the comment:
Yea, I looked into `ceval.c` and the function is *called incorrectly*, so there
is undefined behaviour there - it has been wrong all along, in 3.5 all the way
down to 2-something
if (flags & (METH_NOARG
Antti Haapala <an...@haapala.name> added the comment:
I don't have GCC 8 so I cannot verify this bug, but *function pointer casts*
are fine - any function pointer can be cast to any other function pointer - it
is only that they must *not* be called unless cast back again to be comp
New submission from Antti Haapala <an...@haapala.name>:
I think this is a really stupid security bug. Running a module with `-mmodule`
seems to add '' as a path in sys.path, and in front. This is doubly wrong,
because '' will stand for whatever the current working directory might
Antti Haapala <an...@haapala.name> added the comment:
I've been hit by this too, in similar contexts, and several times. It is really
annoying that it is easier to coerce an UUID or UUID-string to a string than to
coerce to a UUID. Usually when the copy semantics are clear and the
Antti Haapala <an...@haapala.name> added the comment:
"Antti, is there a place in the ctypes documentation that explicitly says
ctypes is not meant to be used cross-platform? If not, shouldn't that be
mentioned?"
I don't know about that, but the thing is nowhere does it say
Antti Haapala added the comment:
To Charles first: "Gives back a sizeof of 8 on Windows and 10 on Linux. The
inconsistency makes it difficult to have code work cross-platform."
The bitfields in particular and ctypes in general have *never* been meant to be
cross-platform - instead
Changes by Antti Haapala <an...@haapala.name>:
--
pull_requests: +2820
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue30969>
___
_
New submission from Antti Haapala:
The doc reference/expressions.srt says that
> For user-defined classes which do not define __contains__() but do
> define __iter__(), x in y is True if some value z with x == z is
> produced while iterating over y. If an exception is rais
Antti Haapala added the comment:
Every now and then there are new questions and answers regarding the use of
`linecache` module on Stack Overflow for doing random access to text files,
even though the documentation states that it is meant for Python source code
files.
One problem
Antti Haapala added the comment:
shouldn't this get in sooner, as the 3.5.2 documentation says that it behaves
exactly like `os.walk`, with some additions, none of which says "bytes paths
are not supported". This looks like a bug to me.
--
no
Antti Haapala added the comment:
While at it, another POSIX semantic that execvp doesn't support is the
behaviour when `PATH` is not set, e.g. on Linux, the search path is set to '.',
followed by confstr(_CS_PATH). It is debatable whether this is desired (having
current directory first
Antti Haapala added the comment:
Someone on Stack Overflow just had a problem where their shell script would
work in shell but get `OSError: [Errno 8] Exec format error` when calling it
with `subprocess.call`. I'd say rather fix this (on POSIX platforms).
Why does Python do path resolving
Antti Haapala added the comment:
Ah there was something that I overlooked before - the VN1 and VN2 both have
combining accents too. If I read correctly, the main letter should precede the
combining character, just as in Unicode; VN3 seems to lack combining characters
altogether.
Thus
Antti Haapala added the comment:
I found the full document on SlideShare:
http://www.slideshare.net/sacobat/tcvn-5712-1993-cng-ngh-thng-tin-b-m-chun-8bit-k-t-vit-dng-trong-trao-i-thng-tin
As far as I can understand, they're "subsets" of each other only in the sense
that VN1 has
Antti Haapala added the comment:
One question is why doesn't it just try to `open`, but wants to stat first,
when the python principle has always been EAFP.
--
nosy: +ztane
___
Python tracker <rep...@bugs.python.org>
<http://bugs.p
Antti Haapala added the comment:
To be more precise, C90, C99, C11 all say that ~"For d, i, o, u, x and X
conversions, if a precision is specified, the 0 flag will be ignored."
--
nosy: +ztane
___
Python tracker <rep...@bugs.pyth
Antti Haapala added the comment:
Serhiy, you actually did make a mistake above; `'%s' % x` cannot be rewritten
as `f'{x!s}'`, only `'%s' % (x,)` can be optimized...
(just try with `x = 1, 2`)
--
nosy: +ztane
___
Python tracker <
Antti Haapala added the comment:
*wow* isn't this in Python 3 yet (I've been using Zope translationstrings
lately so I didn't know that pgettext is still not supported).
--
nosy: +ztane
___
Python tracker <rep...@bugs.python.org>
Antti Haapala added the comment:
Totally inlined when built with GCC, release mode, there is no trace of the
maybe_small_long symbol, I couldn't even really decipher where it was being
done in the disassembly of longobject.c. Was this the Windows release build?
--
nosy: +ztane
Antti Haapala added the comment:
Well, as far as I understand, this is a bug in the backport only. The Python
2.6+ backport does not behave identically to Python 3 code in Python 2, unless
`__hash__ = None` is added. unittest.mock in Python 3 is not affected in any
way
Antti Haapala added the comment:
So you mean that unittest.mock should explicitly set `__hash__ = None`; but
this is already what the unittest.mock does, in Python 3; there is no need to
set `__hash__ = None`; it is the backport that needs to do this in order to
stay compatible
Antti Haapala added the comment:
*I mean the problem exists in `urlparse`, not only in `urlunparse`
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Antti Haapala added the comment:
The problem is in `urlunparse`. If you check the RFC 2396, it has the following
regular expression:
^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?
where group 3 is the //netloc, and 4 is netloc substring, of an url
http://netloc/foobar
Antti Haapala added the comment:
I am very negative to this idea. Correct code in Python **2** would either set
`__hash__ = None` or redefine `__hash__` in *any* class that defines `__eq__`.
That it just wasn't used like that is no excuse.
This warning is even more important if even Ned
Antti Haapala added the comment:
The title of the issue is still wrong. As I noted before the problem is not
with subprocess leaking 4K memory *always*. The issue comes from the fact that
subprocess seems to leak 4K memory per individual thread. The test code to use
is thus
def test
Antti Haapala added the comment:
I wouldn't actually consider the builtin lookup speed almost at all, in all
non-trivial applications the ratio of builtins to any other names will
diminish; and if there is a tight loop, it is possible to always speed it up by
`min = min` if you do not need
Antti Haapala added the comment:
Ahhah, the title should say: subprocess module leaks 4kiB memory **per thread**.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Antti Haapala added the comment:
I just noticed that you're actually testing a builtin instead of something in
just module globals. How is the performance with module globals?
--
nosy: +ztane
___
Python tracker <rep...@bugs.python.org>
Antti Haapala added the comment:
3.5.1+ ubuntu; I run the Popen case in while True, and watch `top` - not a
single digit changes in the memory usage (the last digit being the kilobytes).
That the memory footprint increases *once* by 4KiB is nothing; please run this
in a loop.
--
nosy
Antti Haapala added the comment:
There is no need to ifdef anything, the memcpy is the only correct way to do
it. As memcpy is also a reserved identifier in C, the compiler can and will
optimize this into a 64-bit access on those platforms where it can be safely
done so (x86 for example), e.g
Antti Haapala added the comment:
Though it is clean to do like this: let _PyUnicode_JoinArray have `NULL` mean
empty string, as it is more logical anyway; then PyUnicode_Join itself just
needs to:
if (separator == NULL) {
separator = PyUnicode_FromOrdinal(' ');
/* check
Antti Haapala added the comment:
I tried to post a comment on rietveld, but 500 infernal error...
PyUnicode_New(0, 0) will return unicode_empty. If unicode_empty is NULL, then
it will also initialize it. It would be cleanest if unicode_empty was
statically created.
NULL cannot be used
Antti Haapala added the comment:
So does this (new opcode) count as a new feature? It would be great to give f''
strings a flying start, saying that not only they're cool, they're also faster
than anything that you've used before.
Here some more mini-benchmarks with serhiy's patch2 applied
Antti Haapala added the comment:
I've got one idea about how to implement this, but it would require adding a
new flag field to PyExc_AttributeError type.
This flag, if set, would tell that the AttributeError in question was raised in
C descriptor code or under similar circumstances
Antti Haapala added the comment:
Yeah, it definitely is a bug in CPython. open(mode='a') should always append to
the end of the given file.
If you're writing an append-only text log to some file-like object, that's the
mode you use, not some version/platform/filesystem specific voodoo
Antti Haapala added the comment:
Presumably the case was that a *named* log file is opened with 'a' mode, and
one could pass '/dev/stdout' just like any other name of a file, and it did
work, but not in Python 3.5.
--
nosy: +ztane
___
Python
Antti Haapala added the comment:
Unfortunately it seems that it is not that straightforward. The descriptor
object doesn't know the name of the property. The error is raised in
`property_descr_set`. However the error itself can be propagated from setting
another property
New submission from Antti Haapala:
Today we had an internal server error in production. I went to see the sentry
logs for the error, and was dismayed: the error was `AttributeError: can't set
attribute`, and the faulting line was `setattr(obj, attr, value)` that happens
in a for-loop
Antti Haapala added the comment:
Excel's behaviour has always been locale-dependent. If the user's locale uses ,
as the decimal mark , then ; has been used as the column separator in "C"SV.
However, even if you use autodetection with sniff, it is impossible to detect
with 100 % acc
Antti Haapala added the comment:
Anyhow, in this case it is easy to simulate the Python 2 randint behaviour (add
checks for hi >= lo if needed):
>>> random.seed(5, version=1)
>>> randint_compat = lambda lo, hi: lo + int(random.random() * (hi + 1 -
lo))
Antti Haapala added the comment:
but yes, now that I read the documentation, 3.5 docs it say very explicitly
that:
Two aspects are guaranteed not to change:
- If a new seeding method is added, then a backward compatible seeder will
be offered.
- The generator’s random() method
Antti Haapala added the comment:
Sorry + and - are backwards there (I did the delta in wrong direction); + is
before, and - after Raymond's commit. The `if` was retained there for
backward-compatibility.
--
___
Python tracker <
Antti Haapala added the comment:
It is this change in 3.2:
randrange is more sophisticated about producing equally distributed
values. Formerly it used a style like ``int(random()*n)`` which '
could produce slightly uneven distributions.
-return self._randbelow
Antti Haapala added the comment:
About "I hesitate between the C types "int" and "Py_ssize_t" for nargs. I read
once that using "int" can cause performance issues on a loop using "i++" and
"data[i]" because the compiler has to handle i
Antti Haapala added the comment:
Ah, indeed, I somehow missed that. Though, there is no good reason for it being
unsigned either; as the actual type in SSL API's is of type int. Another
argument of type int is cast to unsigned just for the comparison on line 4419,
and unsigned int counters i
Antti Haapala added the comment:
I don't think adding -Wno-type-limits is a good idea.
The good question is how that can be happening, e.g. how PY_SSIZE_T_MAX divided
by sizeof anything can be *more* than max(size_t)? E.g now that I stare at the
code, *that* warning should be impossible
Antti Haapala added the comment:
@Stefan after reading the bad standard I agree that it follows the standard, as
unfortunate as it is.
However, that part is then also wrong in Raymond's documentation patch. It
should be something like: the exponent is adjusted to a multiple of 3 if *any
Antti Haapala added the comment:
Ok, after reading the "spec" it seems that the engineering exponent is indeed
printed for positive exponents *if* the precision of the number is less than
the digits of the exponent, which I didn't realize that I should be testing.
However the
Antti Haapala added the comment:
Indeed engineering notation is now utterly broken, the engineering notation is
not printed for pretty much _any *engineering* numbers at all_ in 3.6.
Engineering numbers mean numbers that could be met in an *engineering* context,
not cosmological
Antti Haapala added the comment:
Raymond: your doc patch is not quite right. Decimal('123e1') is converted to
Decimal('1.23e3') internally already; so that str(d) will print 1.23e3,
scientific notation of that number is '1.23e3' and engineering notation is
'1.23e3', thus not a good example
Antti Haapala added the comment:
And as it is documented, it would be a change against documentation.
However as a stop-gap it is rather trivial to make your own copy function to
fix this. copy2 returns the actual destination, so you could do
def copy_with_ownership(src, dest
New submission from Antti Haapala:
https://docs.python.org/3/library/decimal.html#decimal.Context.to_eng_string
The docstring for `Context.to_eng_string` says "Converts a number to a string,
using scientific notation.", which is, less extra comma, exactly the
Antti Haapala added the comment:
No, I was just trying to explain why your change could be considered beneficial.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Antti Haapala added the comment:
Normally I wouldn't recommend changing working code. However those asserts
would be OK; if either of them is NULL, then the previous if would have had
undefined behaviour already. Thus the `XDECREF` wrongly signals that it'd be OK
if they were NULLs until
Antti Haapala added the comment:
I don't believe Python would really ever work on a platform with
non-8-bit-bytes, I believe there are way too much assumptions *everywhere*. You
can program in C on such a platform, yes, but not that sure about Python.
And on 8-bit-byte platfomrs
Antti Haapala added the comment:
I would very much like to see this in 3.6. Who could review it?
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Antti Haapala added the comment:
It must be noted that `getlines` itself is not documented, and thus there is no
backwards-compatibility to preserve really. `getline` returns '' for *any*
erroneous line, so it wouldn't affect it.
--
___
Python
Antti Haapala added the comment:
Or perhaps getlines should return [''] for empty regular files?
--
nosy: +ztane
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Antti Haapala added the comment:
About shifts, according to C standard, right shifts >> of negative numbers are
implementation-defined:
"[in E1 >> E2], If E1 has a signed type and a negative value, the
resulting value is implementation-defined."
In K this mea
Antti Haapala added the comment:
I am pretty sure **it never calls zlib.compress**. I get memory error from that
argument alone, on Linux with overcommit memory enabled, 16G ram, swap.
--
nosy: +ztane
___
Python tracker <rep...@bugs.python.
Antti Haapala added the comment:
I was thinking that perhaps an exception is always raised somewhere before? I
tried skipping site, but it still works, so I am not too sure.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.p
Antti Haapala added the comment:
more easily reproducible by
import threading
def foo():
raise
threading.Thread(target=foo).start()
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Antti Haapala added the comment:
OTOH, if you put sys.exc_info() in place of raise there, it correctly (None,
None, None) there, because it does (sysmodule.c:sys_exc_info)
tstate->exc_type != NULL ? tstate->exc_type : Py_None,
Easiest fix would be to make do_raise test for bot
Antti Haapala added the comment:
Reproducible on Python 3.6a4ish on Ubuntu. I believe this needs forking
multiprocessing.
do_raise is called with 2 NULLs as arguments, it should raise
PyErr_SetString(PyExc_RuntimeError,
"No active exception to reraise&quo
Antti Haapala added the comment:
Note that this has nothing to do with `hex()` function. The part that is
problem here is 10**80, which takes ages to compute. You can
interrupt it with Ctrl-C.
--
nosy: +ztane
___
Python tracker <
Antti Haapala added the comment:
Ah indeed, this is a bytearray and it is indeed possible to theoretically
allocate PY_SSIZE_T_MAX bytes, if on an architecture that does segmented memory.
As for
if (addition > PY_SSIZE_T_MAX - len - 1) {
it is very clear to *us* but it is not quite s
Antti Haapala added the comment:
Gnulib portability library has
https://www.gnu.org/software/gnulib/manual/html_node/Integer-Range-Overflow.html
and
https://www.gnu.org/softwarhe/gnulib/manual/html_node/Integer-Type-Overflow.html
and even macros for producing well-defined integer wraparound
Antti Haapala added the comment:
Alexander: that is true, because they are *separate* conversion flags.
However even the POSIX standard strptime has some leniency: '%m` and `%d`
accept the numbers *without* leading zeroes. This actually also means that one
cannot use `%Y%m%d` to detect
Antti Haapala added the comment:
"Be conservative in what you do, be liberal in what you accept from others"
they say. Also Z as a timezone designator is also widely used in iso 8601
timestamps. I believe the effort should be made to *parse* *any/all* of the ISO
8601 supported time-
Antti Haapala added the comment:
Serhiy suggested this in Rietveld:
> For additional optimization we can pack all constant strings, parsed formats
> and
> flags in one constant object and use the single LOAD_CONST. But this requires
> much larger changes (perhaps including changing
Antti Haapala added the comment:
I believe tests is that they should *especially* be in place for any previously
found "careless omissions". If it has been done before, who is to say that it
wouldn't happen again?
--
nosy: +ztane
___
Pyth
Antti Haapala added the comment:
if (len == PY_SSIZE_T_MAX) is necessary for the case that the iterable is
already PY_SSIZE_T_MAX items. However it could be moved inside the *other* if
because if (len == PY_SSIZE_T_MAX) should also fail the overflow check.
However, I believe it is theoretical
Antti Haapala added the comment:
Thanks Serhiy, I was writing my comment for a long time, and only now noticed
that you'd already posted the patch.
Indeed, it seems that not only is this the fastest method, it might also be the
fastest string concatenation method in the history of Python. I
Antti Haapala added the comment:
It seems Eric has done some special casing for strings already in FORMAT_VALUE.
Here are the results from my computer after applying Demur's patch for
concatenating *strings*.
python3.6 -m timeit -s "x = 'a'" -- '"X is %s" % x'
1000
Antti Haapala added the comment:
Yet the test cases just prove what is so expensive there: name lookups (global
name `str`; looking up `join` on a string instance); building a tuple (for
function arguments) is expensive as well. Of course `__format__` will be costly
as well
Antti Haapala added the comment:
Ah so it seems. Somehow I thought __format__ was slotted, but that is not the
case and it needs to be looked up, and what is worse, of course a tuple needs
to be built as well.
Oh well, at least it should be quite minimal to make it be faster than `f(x
Antti Haapala added the comment:
And the expected performance for optimal `f'X is {x}'` code would be *faster*
than `"'X is %s' % (x,)"` which still needs to interpret the string at runtime,
and build a proper tuple object on stack.
--
Antti Haapala added the comment:
One common case where signed integer overflow has been assumed has been the
wraparound/overflow checks like in http://bugs.python.org/issue27473
I propose that such commonly erroneous tasks such as overflow checks be
implemented as common macros in CPython
Antti Haapala added the comment:
I am not an expert on writing new opcodes to CPython (having never done it,
don't know where to change the disassembler and such, how to make compiler
generate them properly and such), but I'd be glad to help with testing, timing
and writing the possible
Antti Haapala added the comment:
The previous code was perfectly fine with `-fwrapv` since it makes signed
overflow behaviour defined. And afaik BDFLs stance is that signed integer
overflow should be defined to wrap anyhow.
In my opinion the `-fwrapv` itself makes one proliferate all
Antti Haapala added the comment:
it is handy to be able to use `\w` and `\d` in non-raw-string *regular
expressions*, without too much backslashitis. Seems to be in use in Python
standard library as well, for example in csv.py
--
nosy: +ztane
Antti Haapala added the comment:
I don't think setting environment variables is a solution, as it is not always
clear which script occurs early in the boot process, or even that which program
has components written in Python. However I'd want to be notified of failure as
well, perhaps
Antti Haapala added the comment:
Casting the pointer is OK, explicitly typing the pointer is even better; then
the arithmetic is even clearer and it is easy to add offsets in words!
However what I'd prefer here as the size of the type is important, to use
`uint16_t` or typedef instead of just
Antti Haapala added the comment:
This behaviour exists exactly because the return value also contains the
`.hostname`, which for the IPv6 addresses is *without* brackets:
>>> urlparse('http://[::1]:80/').hostname
'::1'
There is no way to get a proper parsing result from such
1 - 100 of 139 matches
Mail list logo
