[issue7946] Convoy effect with I/O bound threads and New GIL
Omer Katz added the comment: I think that we should focus our efforts on removing the GIL, now that we have a feasible solution for doing so without breaking anything (hopefully) however the removal of the GIL is still far from being complete and will need to be rebased upon the latest Python version to be merged. This issue would probably hurt Celery since some users use it with a thread pool and it uses a few threads itself but it seems like fixing it is too much effort so if we were to invest a lot of effort, I'd focus on removing the problem entirely. Instead, I suggest we document this with a warning in the relevant place so that people would know to avoid or workaround the problem entirely. On Mon, Mar 21, 2022, 20:32 Guido van Rossum wrote: > > Change by Guido van Rossum : > > > -- > nosy: +gvanrossum > > ___ > Python tracker > <https://bugs.python.org/issue7946> > ___ > -- ___ Python tracker <https://bugs.python.org/issue7946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43248] platform.libc_ver() returns no information on Alpine Linux
New submission from Omer Katz :
When calling platform.libc_ver() on Alpine I get the following tuple: ('', '')
Instead, I expect to get ('muslc', '') or an exception if the
libc type & version cannot be detected.
If one of them is not detected, the returned value shouldn't be an empty string
but None.
The current API is not what most people would expect it to be.
This is verified on Python 3.8 since that's the latest one available on Alpine.
This code reliably detects musl libc and its version:
https://github.com/lovell/detect-libc/blob/master/lib/detect-libc.js
I'd be happy to contribute a PR if needed.
--
messages: 387211
nosy: Omer.Katz
priority: normal
severity: normal
status: open
title: platform.libc_ver() returns no information on Alpine Linux
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue43248>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Omer Katz added the comment: FYI I can verify that the original benchmark is still valid on Python 3.7.3. I'm running the client on an 8 core CPU. The result is 30.702 seconds (341534.322 bytes/sec). I'll need somebody to decide how we're going to fix this problem. I can do the legwork. -- ___ Python tracker <https://bugs.python.org/issue7946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Omer Katz added the comment: Celery 5 is going async and in order to isolate the main event loop from task execution, the tasks are going to be executed in a different thread with it's own event loop. This thread may or may not be CPU bound. The main thread is I/O bound. This patch should help a lot. I like Nir's approach a lot (although I haven't looked into the patch itself yet). It's pretty novel. David's patch is also very interesting. I'm willing to help. -- nosy: +Omer.Katz ___ Python tracker <https://bugs.python.org/issue7946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21145] Add the @cached_property decorator
Omer Katz added the comment: I believe so. Let's do it :) בתאריך יום ב׳, 14 במאי 2018 ב-19:30 מאת Carl Meyer < rep...@bugs.python.org>: > > Carl Meyer added the comment: > > > I don't think it makes sense to try to make cached_property itself work > implicitly with both normal attributes and slot entries - instead, > cached_property can handle the common case as simply and efficiently as > possible, and the cached_slot case can be either handled separately or else > not at all. > > So it sounds like the current approach here is good to move forward? If I > update the patch during the PyCon sprints, we could merge it? > > -- > > ___ > Python tracker > <https://bugs.python.org/issue21145> > ___ > -- ___ Python tracker <https://bugs.python.org/issue21145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21145] Add the @cached_property decorator
Omer Katz added the comment: In that case, most of the users won't use the standard library @cached_property anyway since they require it. On Mon, Feb 15, 2016, 19:51 Antoine Pitrou wrote: > > Antoine Pitrou added the comment: > > The TTL idea is completely outlandish in a general-purpose library. Let's > keep things simple and not try to build a kitchen-sink decorator. > > -- > > ___ > Python tracker > <http://bugs.python.org/issue21145> > ___ > -- ___ Python tracker <http://bugs.python.org/issue21145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21145] Add the @cached_property decorator
Omer Katz added the comment: Most implementations these days support TTL because they require it. The whole point is to remove the need to reimplement such basic functionality over and over. בתאריך יום ב׳, 15 בפבר׳ 2016 ב-18:33 מאת STINNER Victor < rep...@bugs.python.org>: > > STINNER Victor added the comment: > > I like the idea of an helper to build a property on-demand, but I dislike > the TTL idea, it seems too specific. > > If you need TTL, implement your own decorator, or use a regular property > and implement your own logic there. > > -- > > ___ > Python tracker > <http://bugs.python.org/issue21145> > ___ > -- ___ Python tracker <http://bugs.python.org/issue21145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21145] Add the @cached_property decorator
Omer Katz added the comment: Can we make this happen for 3.6? -- versions: +Python 3.6 -Python 3.5 ___ Python tracker <http://bugs.python.org/issue21145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26263] Serialize array.array to JSON by default
New submission from Omer Katz: Is there a reason why the JSON module doesn't serialize array.array() instances by default? Currently you need to convert them to tuples but I'm confident that the C API for those types is enough to iterate over the array and serialize it to a JSON list. We should support serializing arrays by default in my opinion. -- components: Interpreter Core messages: 259341 nosy: Omer.Katz priority: normal severity: normal status: open title: Serialize array.array to JSON by default type: enhancement versions: Python 3.6 ___ Python tracker <http://bugs.python.org/issue26263> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16392] import crashes on circular imports in ext modules
Omer Katz added the comment: Is this issue resolved in any way? Has there been a decision made on how to resolve it? What's the status here? -- nosy: +Omer.Katz ___ Python tracker <http://bugs.python.org/issue16392> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22985] Segfault on time.sleep
Omer Katz added the comment:
Capturing the signal provides the following traceback:
File "mt_example.py", line 10, in
p1 = ThreadPool()
File
"/home/omer/.virtualenvs/billiard/lib/python2.7/site-packages/billiard/pool.py",
line 1016, in __init__
self._result_handler.start()
File
"/home/omer/.virtualenvs/billiard/lib/python2.7/site-packages/billiard/pool.py",
line 504, in start
super(PoolThread, self).start(*args, **kwargs)
File
"/home/omer/.virtualenvs/billiard/lib/python2.7/site-packages/billiard/py2/dummy/__init__.py",
line 69, in start
threading.Thread.start(self)
File "/home/omer/.pyenv/versions/2.7.8/lib/python2.7/threading.py", line 750,
in start
self.__started.wait()
File "/home/omer/.pyenv/versions/2.7.8/lib/python2.7/threading.py", line 620,
in wait
self.__cond.wait(timeout)
File "/home/omer/.pyenv/versions/2.7.8/lib/python2.7/threading.py", line 339,
in wait
waiter.acquire()
File "mt_example.py", line 6, in handler
print(''.join(traceback.format_stack(frame)))
File "mt_example.py", line 10, in
p1 = ThreadPool()
File
"/home/omer/.virtualenvs/billiard/lib/python2.7/site-packages/billiard/pool.py",
line 1016, in __init__
self._result_handler.start()
File
"/home/omer/.virtualenvs/billiard/lib/python2.7/site-packages/billiard/pool.py",
line 504, in start
super(PoolThread, self).start(*args, **kwargs)
File
"/home/omer/.virtualenvs/billiard/lib/python2.7/site-packages/billiard/py2/dummy/__init__.py",
line 69, in start
threading.Thread.start(self)
File "/home/omer/.pyenv/versions/2.7.8/lib/python2.7/threading.py", line 750,
in start
self.__started.wait()
File "/home/omer/.pyenv/versions/2.7.8/lib/python2.7/threading.py", line 620,
in wait
self.__cond.wait(timeout)
File "/home/omer/.pyenv/versions/2.7.8/lib/python2.7/threading.py", line 339,
in wait
waiter.acquire()
The following code reproduces that traceback every time:
from billiard.pool import ThreadPool
import signal
def handler(signum, frame):
import traceback
print(''.join(traceback.format_stack(frame)))
signal.signal(signal.SIGTERM, handler)
p1 = ThreadPool()
class Foo(object):
def a(self):
print("a")
return 1
def do(self):
return p1.apply_async(self.a)
foo = Foo()
r = foo.do()
print(r.get())
p1.close()
p1.join()
I'm convinced that there is a bug somewhere and it seems that Python is the
source of the bug.
--
___
Python tracker
<http://bugs.python.org/issue22985>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22985] Segfault on time.sleep
Omer Katz added the comment: So why exactly the program terminates on a system call? 2014-12-05 23:44 GMT+02:00 STINNER Victor : > > STINNER Victor added the comment: > > > Program received signal SIGTERM, Terminated. > > SIGTERM is not a segmentation fault. SIGSEGV is for segmentation faults. > > -- > nosy: +haypo > resolution: -> not a bug > status: open -> closed > > ___ > Python tracker > <http://bugs.python.org/issue22985> > ___ > -- ___ Python tracker <http://bugs.python.org/issue22985> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22985] Segfault on time.sleep
New submission from Omer Katz:
I have found what appears to be a segfualt in time.sleep but I'm not sure.
I have verified that the segfualt occurs both on Python 2.7.8 and 3.4.2.
The following program that reproduces the segfault uses my fork of billiard, a
replacement for the multiprocessing module that is used by celery.
In order to install: pip install
git+https://github.com/thedrow/billiard@topic/dill-with-threads
The program:
from billiard.pool import ThreadPool
p1 = ThreadPool()
class Foo(object):
def a(self):
print("a")
return 1
def do(self):
return p1.apply_async(self.a)
foo = Foo()
r = foo.do()
print(r.get())
p1.close()
p1.join()
The output from gdb:
gdb -ex r -q --args python mt_example.py
Reading symbols from python...done.
Starting program: /home/omer/.virtualenvs/billiard/bin/python mt_example.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x731db700 (LWP 2897)]
[New Thread 0x729da700 (LWP 2898)]
[New Thread 0x721d9700 (LWP 2899)]
[New Thread 0x719d8700 (LWP 2900)]
[New Thread 0x711d7700 (LWP 2901)]
[New Thread 0x709d6700 (LWP 2902)]
[New Thread 0x7fffdbfff700 (LWP 2903)]
Program received signal SIGTERM, Terminated.
[Switching to Thread 0x721d9700 (LWP 2899)]
0x771dc1c3 in select () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x771dc1c3 in select () at ../sysdeps/unix/syscall-template.S:81
#1 0x75fd5d5d in floatsleep (secs=) at
Modules/timemodule.c:948
#2 time_sleep (self=, args=) at
Modules/timemodule.c:206
#3 0x004b658c in call_function (oparg=,
pp_stack=0x721d8540) at Python/ceval.c:4033
#4 PyEval_EvalFrameEx (f=f@entry=0x7463eda8, throwflag=throwflag@entry=0)
at Python/ceval.c:2679
#5 0x004b6f6b in PyEval_EvalCodeEx (co=,
globals=, locals=locals@entry=0x0, args=,
argcount=argcount@entry=4, kws=, kwcount=0,
defs=0x740bab68, defcount=1, closure=0x0) at Python/ceval.c:3265
#6 0x004b5b28 in fast_function (nk=, na=4, n=, pp_stack=0x721d8730, func=0x731e2938)
at Python/ceval.c:4129
#7 call_function (oparg=, pp_stack=0x721d8730) at
Python/ceval.c:4054
#8 PyEval_EvalFrameEx (f=f@entry=0x7fffe8000b50, throwflag=throwflag@entry=0)
at Python/ceval.c:2679
#9 0x004b6f6b in PyEval_EvalCodeEx (co=,
globals=, locals=locals@entry=0x0, args=,
argcount=argcount@entry=1, kws=, kwcount=0, defs=0x0,
defcount=0, closure=0x0) at Python/ceval.c:3265
#10 0x004b5b28 in fast_function (nk=, na=1, n=, pp_stack=0x721d8920, func=0x731e28c0)
at Python/ceval.c:4129
#11 call_function (oparg=, pp_stack=0x721d8920) at
Python/ceval.c:4054
#12 PyEval_EvalFrameEx (f=f@entry=0x7fffe8000910, throwflag=throwflag@entry=0)
at Python/ceval.c:2679
#13 0x004b5bc7 in fast_function (nk=, na=1, n=, pp_stack=0x721d8a70, func=0x77e55230)
---Type to continue, or q to quit---
at Python/ceval.c:4119
#14 call_function (oparg=, pp_stack=0x721d8a70) at
Python/ceval.c:4054
#15 PyEval_EvalFrameEx (f=f@entry=0x731f1210, throwflag=throwflag@entry=0)
at Python/ceval.c:2679
#16 0x004b6f6b in PyEval_EvalCodeEx (co=,
globals=, locals=locals@entry=0x0,
args=args@entry=0x731dca68, argcount=,
kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0,
defcount=defcount@entry=0, closure=0x0) at Python/ceval.c:3265
#17 0x00529e70 in function_call (func=0x77e55140,
arg=0x731dca50, kw=0x0) at Objects/funcobject.c:526
#18 0x00422f5a in PyObject_Call (func=func@entry=0x77e55140,
arg=arg@entry=0x731dca50, kw=kw@entry=0x0)
at Objects/abstract.c:2529
#19 0x0042747d in instancemethod_call (func=0x77e55140,
arg=0x731dca50, kw=0x0) at Objects/classobject.c:2602
#20 0x00422f5a in PyObject_Call (func=func@entry=0x74636d70,
arg=arg@entry=0x77f93050, kw=)
at Objects/abstract.c:2529
#21 0x004b07f7 in PyEval_CallObjectWithKeywords (func=0x74636d70,
arg=0x77f93050, kw=) at Python/ceval.c:3902
#22 0x004fae92 in t_bootstrap (boot_raw=0x8899c0) at
./Modules/threadmodule.c:614
#23 0x77bc40a5 in start_thread (arg=0x721d9700) at
pthread_create.c:309
#24 0x771e484d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
--
components: Interpreter Core
messages: 232059
nosy: Omer.Katz
priority: normal
severity: normal
status: open
title: Segfault on time.sleep
type: crash
versions: Python 2.7, Python 3.4
___
Python tracker
<http://bugs.python.org/issue22985>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21807] SysLogHandler closes TCP connection after first message
New submission from Omer Katz:
import logging
import logging.handlers
import socket
logger = logging.getLogger('mylogger')
handler = logging.handlers.SysLogHandler(('',
logging.handlers.SYSLOG_TCP_PORT), socktype=socket.SOCK_STREAM)
formatter = logging.Formatter('%(name)s: [%(levelname)s] %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info("TEST 1")
logger.info("TEST 2")
I have verified that this code only sends 'TEST 1' to Splunk and syslog-ng on
both Python 2.7 and Python 3.4. After that, the connection appears closed. UDP
on the other hand works just fine.
--
components: Library (Lib)
messages: 220961
nosy: Omer.Katz
priority: normal
severity: normal
status: open
title: SysLogHandler closes TCP connection after first message
versions: Python 2.7, Python 3.4
___
Python tracker
<http://bugs.python.org/issue21807>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21145] Add the @cached_property decorator
Omer Katz added the comment: I just checked and werkzeug uses the same implementation as Django & pip. 2014-04-06 14:31 GMT+04:00 Omer Katz : > > Omer Katz added the comment: > > The default implementation should simple: > Once the property is accessed for the first time calculate the result and > never calculate again. > It's what both Django & pip uses. > You can add an optional TTL. > There aren't any other features I can come up with that are common enough. > If needed, one can inherit from cached_property and do whatever is needed. > Eric, Why don't you think a C implementation is needed? It's a simple > operation for sure but it is meant to increase runtime efficiency. > > 2014-04-05 1:11 GMT+04:00 Éric Araujo : > > > > > Éric Araujo added the comment: > > > > It could make sense to add clean, working recipes to e.g. the functools > > documentation. The cached_property in the wiki uses a TTL, other like > > Pyramid's reify decorator make properties that ensure the fget function > is > > called only once per instance, and there may be subtly different variants > > out there. I don't know if there's a universally useful variant that > > should be added to the sdlib right now. (I don't think a C > implementation > > is needed.) > > > > On a related note, the Python docs about desciptors may be missing > > entry-level explanations, as described here: > > http://me.veekun.com/blog/2012/05/23/python-faq-descriptors/ > > > > -- > > nosy: +eric.araujo, rhettinger > > > > ___ > > Python tracker > > <http://bugs.python.org/issue21145> > > ___ > > > > -- > > ___ > Python tracker > <http://bugs.python.org/issue21145> > ___ > -- ___ Python tracker <http://bugs.python.org/issue21145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21145] Add the @cached_property decorator
Omer Katz added the comment: The default implementation should simple: Once the property is accessed for the first time calculate the result and never calculate again. It's what both Django & pip uses. You can add an optional TTL. There aren't any other features I can come up with that are common enough. If needed, one can inherit from cached_property and do whatever is needed. Eric, Why don't you think a C implementation is needed? It's a simple operation for sure but it is meant to increase runtime efficiency. 2014-04-05 1:11 GMT+04:00 Éric Araujo : > > Éric Araujo added the comment: > > It could make sense to add clean, working recipes to e.g. the functools > documentation. The cached_property in the wiki uses a TTL, other like > Pyramid's reify decorator make properties that ensure the fget function is > called only once per instance, and there may be subtly different variants > out there. I don't know if there's a universally useful variant that > should be added to the sdlib right now. (I don't think a C implementation > is needed.) > > On a related note, the Python docs about desciptors may be missing > entry-level explanations, as described here: > http://me.veekun.com/blog/2012/05/23/python-faq-descriptors/ > > -- > nosy: +eric.araujo, rhettinger > > ___ > Python tracker > <http://bugs.python.org/issue21145> > ___ > -- ___ Python tracker <http://bugs.python.org/issue21145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21145] Add the @cached_property decorator
New submission from Omer Katz: cached properties are widely used in various prominent Python projects such as Django and pip (and many more). They are not very complicated and there are proven implementation out there. Unfortunately there are way too many of them. This situation leads me to suggest adding it to the builtins. Possible benefits: * Implementation in C can improve caching performance * Standardized implementation that can be used anywhere -- messages: 215439 nosy: Omer.Katz priority: normal severity: normal status: open title: Add the @cached_property decorator type: enhancement versions: Python 3.5 ___ Python tracker <http://bugs.python.org/issue21145> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
