Charles-François Natali added the comment:
> Here's a patch.
Your patch returns an empty string in case of Linux abstract namespace: this is
wrong, because even though the first char in sun_path is NIL, the path is
non-null, there can be other non-NIL char following.
Example (th
Charles-François Natali added the comment:
I'm making Antoine nosy since he's the author of the "faulty" changeset.
Richard, IIRC, you said somewhere that FD passing failures on OS X could be
made to work by passing a FD at a time, or something like that. What do you
thin
Charles-François Natali added the comment:
+1 from me.
Any info that can help debugging is a good thing (calling getsockname() on an
non-connected socket will raise an error that will get cacthed, but there's not
much we can do about it).
--
nosy: +neo
Charles-François Natali added the comment:
> Is p guaranteed to be size_t aligned?
> If not, unaligned access can segfault (e.g. on Sparc IIRC).
Apparently yes.
--
___
Python tracker
<http://bugs.python.org/i
Charles-François Natali added the comment:
> I'd expect just casting the pointer type before dereferencing:
>
> unsigned char *p;
> ...
> hash = (multiplier * hash) ^ *((Py_uhash_t *)p);
>
> (don't use size_t, use Py_uhash_t)
Is p guaranteed to be size_t aligne
Charles-François Natali added the comment:
> As an interface of ssl socket, server does not have to read, just write
> some data.
> The client side should be able to read the bytes that ther server sent. The
> problem is that client will sometimes raise an unexpected SSLError in
&
Charles-François Natali added the comment:
> Client gets an exception in reading the socket, not in writing. Please run
> the test code and see what happens.
Of course it gets ECONNRESET on subsequent recv(), that's how TCP works.
Just make your handler read from the socket and it w
Charles-François Natali added the comment:
> sometimes RST was sent over the network instead of FIN
Your client sends data, but the server never reads it: when a TCP socket is
closed while there's still data in the input socket buffer, a RST is sent
instead of a FIN. That's nor
Charles-François Natali added the comment:
Note that the patch uses type punning through a union: while GCC allows this,
it's not allowed by ANSI (although since we're using a char [], it's somewhat a
grey area). An aggresive compiler could optimiza the read/write away.
Changes by Charles-François Natali :
--
resolution: -> invalid
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue17677>
___
___
Pyth
Charles-François Natali added the comment:
>> If you could fill a separate issue for that, it would be great.
>
> I will but once this is committed. It'd make things easier.
No problem.
>> BTW, there are other OpenBSD-specific issues on the report, and we
>> don&
Changes by Charles-François Natali :
--
versions: +Python 3.3, Python 3.4
___
Python tracker
<http://bugs.python.org/issue12181>
___
___
Python-bugs-list mailin
Charles-François Natali added the comment:
> New version hopefully good and ready for inclusion.
Looks good to me.
Since I assume you tested your patch on OpenBSD, to me it's ready for commit.
I won't be able to do it myself before two weeks though, so if someone
beats me to
Charles-François Natali added the comment:
I somehow doubt that the gain is worth the trouble, vectored disk I/O is not as
interesting as vectored read/writes to a NIC.
Actually, a quick search returned this link:
http://www.mail-archive.com/dev@httpd.apache.org/msg23763.html
Running the
Charles-François Natali added the comment:
> I don't know how to handle this in a non-hackish way, except by just ignoring
> the issue :-)
Sound fine to me :)
--
resolution: -> invalid
stage: -> committed/rejected
statu
Charles-François Natali added the comment:
> How does the test suite react if you change the order of application to
> resolve symlinks only after dropping the file name from the path?
Note that this will break things, see e.g.
http://bugs.python.org/issue1387483#msg186063
The only ba
New submission from Charles-François Natali:
test_ssl is failing on one the NetBSD buildbots:
http://buildbot.python.org/all/builders/x86%20NetBSD%205.1.2%20%5BSB%5D%203.x/builds/1129/steps/test/logs/stdio
"""
=
Charles-François Natali added the comment:
test_kqueue now passes on NetBSD (can't tell for OpenBSD, all buildbots are
offline).
I'm closing this, feel free to repon in case of problem.
--
resolution: -> fixed
stage: patch review -> committed/rejected
status
Charles-François Natali added the comment:
I've no clue what happened to the issue title (I just replied to the email, and
the title changed)...
--
title: sys.path -> sys.path[0] when executed thru a symbolic link
___
Python tracke
Charles-François Natali added the comment:
> I am not sure about what use cases could be broken by the above change, do
> you have examples?
> Normal use cases of symbolic links have to do with linking entire folders,
> not individual files, and that behaviour would not be broke
Charles-François Natali added the comment:
> You would have expected "/home/kristjan/pydir" since this is the
> apparent directory of the file.
That's questionable.
You usually have the libraries along with the binary: that's for example the
case when you do a CPytho
Charles-François Natali added the comment:
> Here is an updated patch.
_PyThreadState_DeleteExcept uses HEAD_LOCK: ISTM that
PyThreadState_Clear() can trigger arbitrary code execution: if a
thread ends up being created/destroyed, I think we can get a deadlock
when trying to acquire the h
Charles-François Natali added the comment:
> That way the scandir name would be left available for a future version of
> this that yields namedtuples of directory entry details as Martin wants to
> see.
Which might very w'ell be Nick's walkdir, see issue #13229.
BTW, I&#x
Charles-François Natali added the comment:
Richard, your patch is indeed better, and fixes the problem.
With a test it should be ready to go in.
--
___
Python tracker
<http://bugs.python.org/issue17
Charles-François Natali added the comment:
Greg, the original issue was about an OS X failure, and it's still a
problem AFAICT.
However it's definitely not a problem with your patch, but an OS X
kernel bug (we've had another similar issue some time ago I think), so
we might a
Charles-François Natali added the comment:
> i don't think that's it, look at the result output in comment #183756: the
> list of fds is correct, except that they aren't closed as they should be.
Of course the list is correct: the message is printed by the parent
process,
Charles-François Natali added the comment:
> Providing a custom class instantiating two sockets looks like a dead end to
> me though. To say one, what is getsockname() supposed to return? Same for
> detach(), fileno(), 'family' and probably others I can't think of ri
Charles-François Natali added the comment:
Hummm...
For those experiencing failures when /proc is mounted: do you have a
grsecurity-patched kernel?
If RBAC is enabled, /proc/self/fd is empty, so this approach won't work...
--
nosy: +neo
Charles-François Natali added the comment:
> I meant when there is no memory pressure.
http://lwn.net/Articles/326552/
"""
The kernel page cache contains in-memory copies of data blocks
belonging to files kept in persistent storage. Pages which are written
to by a processor, b
Charles-François Natali added the comment:
> What you say is right but whether the kernel supports an hybrid IPv4/6 stack
> or not there's not much we can do about it anyway.
> Exactly what are you suggesting with the ServerSocket class you mentioned?
> What do you expect it
Charles-François Natali added the comment:
> Apart from creating, unlinking and resizing the file I don't think there
> should be any disk I/O.
>
> On Linux disk I/O only occurs when fsync() or close() are called.
What?
Writeback occurs depending on the memory pressure, percent
Charles-François Natali added the comment:
> Through fork, yes, but "shared" rather than "copy-on-write".
There's a subtlety: because of refcounting, just treating a COW object
as read-only (e.g. iteratin on the array) will trigger a copy
anyway...
> I assu
Charles-François Natali added the comment:
I think that's a good idea.
However, there's a problem with the implementation: if one passes "" or None as
address on a dual-stack node, the resulting socket will be either IPv4 bound to
INADDR_ANY, or IPv6 bound to IN6ADDR_AN
Charles-François Natali added the comment:
> Also, does pickle currently handle byte strings larger than 4GB?
The 2.7 failure is indeed a pickle limitation, which should now be fixed by
issue #13555.
> On a machine with 256GB of RAM, it makes more sense to send arrays
> of this size
Changes by Charles-François Natali :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue17555>
___
___
Python-bugs-list mailing list
Unsubscribe:
Charles-François Natali added the comment:
That's due to a leak in forker-registered handlers:
The _afterfork_registry is never cleared, so spawning processes recursively
keeps feeling it, which ends up consuming a huge amount of memory and slowing
process creation greatly.
Could you tr
Charles-François Natali added the comment:
- I don't understand the "running out of FDs" thing. select() is limited to FDs
less than FD_SETSIZE, but I don't see how you could get EMFILE (select() will
return a ValueError)
- is there any platform with sendfile() which
Changes by Charles-François Natali :
--
status: open -> pending
___
Python tracker
<http://bugs.python.org/issue17525>
___
___
Python-bugs-list mailing list
Un
Charles-François Natali added the comment:
Committed, thanks!
--
resolution: -> fixed
stage: commit review -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Changes by Charles-François Natali :
--
keywords: +patch
Added file: http://bugs.python.org/file29561/sendfile_doc.diff
___
Python tracker
<http://bugs.python.org/issue17
Charles-François Natali added the comment:
> The old code deleted the obj in the feeder thread as soon as it was sent at
> lines 247 and 253 -- see Issue #16284. I think that should be retained.
The object is overwritten by the pickled data, so it's not necessary
Charles-François Natali added the comment:
I'm splitting the patches:
- one which adds loads and dumps to ForkingPicler
- the contention reduction patch
I'd like to commit them soon.
--
Added file: http://bugs.python.org/file29559/queues_contention.diff
Added
Charles-François Natali added the comment:
That shouldn't be too complicated, but does Windows have fcomod() & Co?
--
___
Python tracker
<http://bugs.python.or
Charles-François Natali added the comment:
> The docs don't even mention that re.compile() actually uses a cache.
Actually it does:
"""
re.compile(pattern, flags=0)
Note The compiled versions of the most recent patterns passed to re.match(),
re.search() or re.compile()
Changes by Charles-François Natali :
--
title: fix os.senfile() documentation regarding the type of file descriptor ->
fix os.sendfile() documentation regarding the type of file descriptor
___
Python tracker
<http://bugs.python.org/issu
New submission from Charles-François Natali:
os.sendfile() documentation states:
"""
On Solaris, out may be the file descriptor of a regular file or the file
descriptor of a socket. On all other platforms, out must be the file descriptor
of an open socket.
""&qu
Charles-François Natali added the comment:
os.getcwd() just calls the libc getcwd(3), so Python's not the problem here.
it's likely an issue with the CIFS implementation (I guess you're using fuse?).
Could you post the output of:
$ strace python -c "import os; os.getc
Charles-François Natali added the comment:
> Would that be erroneous? It can already happen because of thread switches:
That's not really the same thing. wait() would return True, which is
the right thing to do since the Event has been set at some point.
Here, it would make it possible
Charles-François Natali added the comment:
> I was under the impression that dict access (and therefore attribute
> access for "simple" objects) was guaranteed to be atomic even in
> alternative implementations like Jython and IronPython.
>
> Is this not a language guar
Charles-François Natali added the comment:
Something bothers me:
"""
def wait(self, timeout=None):
if self._flag:
return True
self._cond.acquire()
"""
The _flag is checked without any lock held: although it won't be a
problem with CPython, a
Charles-François Natali added the comment:
> By the way, I forgot to mention it previously, but
> multiprocessing.connection uses a custom pickler (ForkingPickler).
Thanks, I didn't know.
Here's a patch using ForkingPickler.
I did a bit of refactoring to move the pi
Changes by Charles-François Natali :
--
stage: -> commit review
versions: +Python 3.4
Added file: http://bugs.python.org/file29528/queues_contention-1.diff
___
Python tracker
<http://bugs.python.org/issu
Charles-François Natali added the comment:
>> POSIX semaphore don't support negative initial value.
>
> It was an early design decision to go with the Java threading model rather
> than the POSIX model. The Java API supports negative initial values -- the
> same Dijkst
Changes by Charles-François Natali :
--
nosy: -neologix
___
Python tracker
<http://bugs.python.org/issue13564>
___
___
Python-bugs-list mailing list
Unsubscribe:
Charles-François Natali added the comment:
> Then why 'offset' and 'count' parameters have a different data type?
Because offsets can be negative (e.g. for lseek), while a size can't.
That's why 'count' is size_t, not ssize_t.
>> Furthermore, s
Charles-François Natali added the comment:
> It's necessary because sendfile() can fail with EAGAIN.
It can fail with EAGAIN if the input FD is non-blocking, exactly like
the current implementation which calls fp.read(). Furthermore, since
sendfile actually supports only regular
Charles-François Natali added the comment:
> In the meantime I rewrote the original patch and got rid of the
> "use_sendfile" explicit argument in order to attempt to use sendfile() by
> default and fall back on using send() if bytes sent were 0.
"""
Charles-François Natali added the comment:
> As for accepting negative initialization values, it sounds like a reasonable
> request. One reason for rejecting would be if it makes writing a fast
> implementation harder. Also, multiprocessing.Semaphore should be kept
> com
Charles-François Natali added the comment:
>> The transfer won't be faster mainly because it's really I/O bound.
>> But it will use less CPU, only because you're making less syscalls.
>
> Have you actually measured this?
"""
vanilla over Gb/s:
re
Charles-François Natali added the comment:
> IIRC Nick Coghlan had put a bit of work into this a few months ago as an
> external module with a view to seeing if it got traction before putting
> anything into the stdlib. Might be worth pinging him, or looking to see
> what he&#
Charles-François Natali added the comment:
> Specifying a big blocksize doesn't mean the transfer will be faster.
> send/sendfile won't send more than a certain amount of bytes anyways.
The transfer won't be faster mainly because it's really I/O bound.
But it will
Charles-François Natali added the comment:
> Ah ok, I misinterpreted what you wrote then.
> Generally speaking though, you don't need to know the file size: you just
> decide a blocksize (>= 65536 is usually ok) and use sendfile() as you use
> send().
But then you make muc
Charles-François Natali added the comment:
> That's not compatible across POSIX platforms.
What do you mean ?
I juste call fstat() before calling senfile() to find out the file
size, and pass it as `count`.
--
___
Python tracke
Charles-François Natali added the comment:
Here's the result of a benchmark sending a 1GB file over a Gb/s ethernet
network:
vanilla:
real0m9.446s
user0m0.493s
sys 0m1.425s
sendfile:
real0m9.143s
user0m0.055s
sys 0m0.986s
The total time doesn't vary much (the
Charles-François Natali added the comment:
> Here is an experimental patch. The speedup is ... measurable.
It's likely to be more useful for dict and set, to avoid/limit rehashs.
Also, the allocation overhead depends on the implementation, I suspect the gain
would be more important w
Charles-François Natali added the comment:
For the record, I'm posting thse benchmark numbers here (originally from issue
#17025):
"""
with patch:
$ ./python /tmp/multi_queue.py
took 0.7945001125335693 seconds with 1 workers
took 0.7428359985351562 seconds
Changes by Charles-François Natali :
--
status: closed -> open
superseder: reduce multiprocessing.Queue contention ->
___
Python tracker
<http://bugs.python.org/i
Charles-François Natali added the comment:
> No. I only looked at the diff and assumed both changes were for Queue.
OK, great.
> Since you marked issue 10886 as superceded, do you intend to do the
> pickling in put()?
Actually no, I'll reopen it.
I find the performance hit imp
Charles-François Natali added the comment:
> It looks like queues_contention.diff has the line
>
> obj = pickle.dumps(obj)
>
> in both _feed() and put(). Might that be why the third set of benchmarks
> was slower than the second?
_feed() is a Queue method, put
Charles-François Natali added the comment:
So, what do you think?
Is the simple version offloading the serialization to queue enough, or
should we go for a full-blown atomic Connection/Pipe/etc?
I find the performance gain quite appreciable (basically queue didn't
scale at all, now it s
Charles-François Natali added the comment:
Gregory, do you have time to take care of this?
--
nosy: +neologix
___
Python tracker
<http://bugs.python.org/issue16
Charles-François Natali added the comment:
Alright, closing.
--
resolution: -> invalid
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Charles-François Natali added the comment:
Closing.
Let's keep O_NOFOLLOW: it doesn't buy much, and might be useful for some arcane
reason on some weird platform...
--
resolution: -> fixed
stage: -> committed/rejected
status
Charles-François Natali added the comment:
The test assumes that the input flags (EV_ADD, EV_ENABLE...) will be returned
in the output events. It's apparently not the case on OpenBSD and NetBSD (and
probably on OS-X neither, because this check is disabled on this platform), and
I can&
Charles-François Natali added the comment:
> Did you forget to attach the patch?
Oops...
--
Added file: http://bugs.python.org/file29295/tstate_after_fork.diff
___
Python tracker
<http://bugs.python.org/issu
Charles-François Natali added the comment:
> Wouldn't it be better to expose and re-use the HEAD_LOCK and HEAD_UNLOCK
> macros from pystate.c?
I don't like holding locks before calling "alien" code, it's a recipe
for deadlocks: for example, if another thread-local
Charles-François Natali added the comment:
And here's a patch.
--
keywords: +patch
Added file: http://bugs.python.org/file29244/thread_local_concurrent.diff
___
Python tracker
<http://bugs.python.org/is
Charles-François Natali added the comment:
> Note that in threadmodule.c, in local_clear, we are iterating through all
> threads:
>
> In PyDict_DelItem, if the GIL is released and meanwhile, the list of
> threadstates is altered, is that a problem for this loop? So maybe ts
Charles-François Natali added the comment:
I don't know how OS X crash report works, but it seems to have at
least some debug info available, since some ymbols are resolved in the
backtrace.
You might be able to get more info with gdb, with something like:
"""
gdb /path/to/py
Charles-François Natali added the comment:
Thanks, it's surprising this was never noticed before.
Your patch has two issues:
- it doesn't clear the thread state before deleting it, which would leak the
frame, thread-specific dict, etc
- it only clears the thread states after the curr
Changes by Charles-François Natali :
Added file: http://bugs.python.org/file29219/cgi_subprocess-1.diff
___
Python tracker
<http://bugs.python.org/issue16945>
___
___
Changes by Charles-François Natali :
--
nosy: +pitrou
___
Python tracker
<http://bugs.python.org/issue16945>
___
___
Python-bugs-list mailing list
Unsubscribe:
Charles-François Natali added the comment:
> - how do you know the crash really happens because of thread 5?
All other threads are blocked on locks or condition variables, it's
the only runnable thread.
> Another question: are threads being started or stopped while the thread loca
Charles-François Natali added the comment:
> Here is one. Others are in the issue report on GitHub.
Yes, I've seen it, but I'd need a backtrace with line numbers (like
the one you posted above).
thread 5 is crashing, but I don
Charles-François Natali added the comment:
> Note that in my original application where I encountered this (with sqlite),
> the backtrace looks slightly different. It is at shutdown, but not at
> interpreter shutdown - the main thread is still running.
Could you post a traceback of t
Charles-François Natali added the comment:
I'm closing, since issue #17025 proposes to do this as part of performance
optimization.
--
nosy: +neologix
status: open -> closed
superseder: -> reduce multiprocessing.Queue contention
___
Pyt
Charles-François Natali added the comment:
> This shouldn't be a problem in Python 3 where Python threads cannot switch
> during shutdown.
What happens if the GIL is relased during shutdown?
Also, I'm a bit worried about this code:
"""
void
PyThreadState_Clear(
Charles-François Natali added the comment:
Alright, here's what's going on.
When the main thread exits, it triggers the interpreter shutdown, which clears
all the tstates in PyInterpreterState_Clear():
"""
void
PyInterpreterState_Clear(PyInterpreterState *inter
Charles-François Natali added the comment:
The heapq C implementation is apparently not thread-safe:
PyObject_RichCompareBool() and Py_DECREF() can lead to arbitrary python code
being called, which can result in a context switch.
--
nosy: +neologix
Charles-François Natali added the comment:
> Shouldn't you try to make the permission removal atomic?
> Otherwise there's a window of opportunity to exploit the suid bit.
Actually there's already a race even without setuid bit:
http://bugs.python.org/issue15100
All
Charles-François Natali added the comment:
> Barry advised me to open this issue as it's a functional regression from
> Python 2.
But it was relying on a private and non documented feature in Python
2, so it's not really a regression.
> My use case is unit testing code th
Charles-François Natali added the comment:
Could you please give an example of why this would be a useful addition?
I can see the usage of removing a handler, but the user should know which
handlers are registered.
--
nosy: +neologix
___
Python
Charles-François Natali added the comment:
Could you try with recent checkout of python 2.7?
I wonder if this could be an occurrence of issue #13992 fixed by Antoine a
couple months ago.
--
nosy: +neologix
___
Python tracker
<http://bugs.python.
Charles-François Natali added the comment:
> Well, that is not from RFC (or I hadn't find it):) however I'd lie if I'd
> call myself an expert, should I change limit to 4096 then?
It's probably not in the RFC: this just shows that the lim
Charles-François Natali added the comment:
> Have you read and checked what the RFCs about the
> FTP protocol say about maximum line length?
vsftpd seems to use a 4096 limit (and the guy knows his stuff :-):
ftp://vsftpd.beasts.org/users/cevans/untar/vsftpd-3.0.2/defs.h
-
Charles-François Natali added the comment:
Committed.
Brian, thanks for the patch!
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Charles-François Natali added the comment:
You happen to have a script named time.py, so when the subprocess module is
imported, it imports this script instead of the correct time module.
Nothing is wrong, closing.
--
nosy: +neologix
resolution: -> invalid
stage: -> com
Charles-François Natali added the comment:
> Using handle_accept() in my code and remembering to call listen() in my
> asyncore.dispatcher server's constructor did the trick.
>
> I am not sure if we still have a bug here though, since if the subclass
> doesn't define a
Charles-François Natali added the comment:
> For the record, I tried the Connection approach and here is what I ended up
> with.
I don't really like the API.
Having to pass an external lock is IMO a bad idea, it should be a
private instance field.
Also, for consistency we'
Charles-François Natali added the comment:
I'm unable to reproduce it.
Are you using the attached script?
Did you apply the patch by hand, or are you using a recent Python version?
--
___
Python tracker
<http://bugs.python.org/is
701 - 800 of 1864 matches
Mail list logo
