[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: For the record, turns out there was a bit of misunderstanding. I used the term deprecate above to mean warn users (through documentation) that they should not use (a feature) and not in its Python-dev sense of remove (a feature) after a period of warning. I do not think the possibility to mix threading and multiprocessing together should be somehow forcibly disabled. Anyway, since my view does not seem to resonate with core developers I I'll give it a rest for now. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: Hi Gregory, Gregory P. Smith g...@krypto.org added the comment: No Python thread is ever fork safe because the Python interpreter itself can never be made fork safe. Nor should anyone try to make the interpreter itself safe. It is too complex and effectively impossible to guarantee. a) I think the term Guarantee is not meaningful here since the interpreter is probably too complex to guarantee it does not contain other serious problems. b) If no Python thread is ever fork safe, can you illustrate how a trivial Python thread spinning endlessly might deadlock a child forked by another Python thread? I was not able to find reports of deadlocks clearly related to multiprocessing worker threads so they could be practically safe already, to the point other Python-Dev developers would be inclined to bury this as a theoretical problem :) Anyway, there exists at least the problem of forking from the pool worker thread and possibly other issues, so the code should be reviewed. Another latent problem is multiprocessing logging which is disabled by default? There is no general solution to this, fork and threading is simply broken in POSIX and no amount of duct tape outside of the OS kernel can fix it. This is why we should sanitize the multithreading module and deprecate mixing of threading and multiprocessing. I bet most developers using Python are not even aware of this problem. We should make sure they are through documentation. Here is another way to look at the current situation: 1) Don't use threading for concurrency because of the GIL. 2) Don't mix threading with multiprocessing because threading and forking don't mix well. 3) Don't use multiprocessing because it can deadlock. We should make sure developers are aware of (2) and can use (3) safely***. My only desire is that we attempt to do the right thing when possible with the locks we know about within the standard library. Right, with an atfork() mechanism. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: then multiprocessing is completely brain-damaged and has been implemented by a moron. Please do not use this kind of language. Being disrespectful to other people hurts the discussion. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: (BTW: there are religions without god, so whom shall e.g. i praise for the GIL?) Guido? ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: Here is a morning reasoning exercise - please help find the flaws or refine it: 5) Sanitizing worker threads in the multiprocessing module Sanitizing a worker thread in the context of this problem is to make sure it can not create a state that may deadlock another thread that calls fork(); or in other words fork-safe. Keep in mind that in Python os.fork() is never called from a POSIX signal handler. So what are examples of a fork-safe thread? a) A thread that spins endlessly doing nothing in a C for(;;) loop is safe. Another thread may call fork() without restrictions. b) A Python thread that only calls function that do not yield the GIL and that does not acquire locks that are held beyond a Python tick is safe. An example for such a lock is a critical-section lock acquired by a lower level third party library for the duration of a function call. Such a lock will be released by the time os.fork() is called because of the GIL. c) A Python thread that in addition to (2) also acquires a lock that is handled at fork is safe. d) A Python thread that in addition to (2) and (3) calls function that yield the GIL but while the GIL is released only calls async-signal-safe code. This is a bit tricky. We know that it is safe for thread A to fork and call async-signal-safe functions regardless of what thread B has been doing, but I do not know that thread A can fork and call non async-signal-safe functions if thread B was only calling async-signal-safe functions. Nevertheless it makes sense: For example lets assume it isn't true, and that hypothetical thread A forked while thread B was doing the async-signal-safe function safe_foo(), and then thread A called non async-signal-safe function unsafe_bar() and deadlocked. unsafe_bar() could have deadlocked trying to acquire a lock that was acquired by safe_foo(). But if this is so, then it could also happen the other way around. Are there other practical possibilities? Either way, we could double check and white list the async-signal-safe functions we are interested in, in a particular implementation. e) Socket related functions such as bind() accept() send() and recv(), that Python calls without holding the GIL, are all async-signal-safe. This means that in principle we can have a fork-safe worker thread for the purpose of communicating with a forked process using a socket. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: Well, my brain did not deadlock, but after spinning on the problem for a while longer, it now thinks Tomaž Šolc and Steffen are right. We should try to fix the multiprocessing module so it does not deadlock single-thread programs and deprecate fork in multi-threaded programs. Here is the longer version, which is a summary of what people said here in various forms, observations from diving into the code and Googling around: 1) The rabbit hole a) In a multi-threaded program, fork() may be called while another thread is in a critical section. That thread will not exist in the child and the critical section will remain locked. Any attempt to enter that critical section will deadlock the child. b) POSIX included the pthread_atfork() API in an attempt to deal with the problem: http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html c) But it turns out atfork handlers are limited to calling async-signal-safe functions since fork may be called from a signal handler: http://download.oracle.com/docs/cd/E19963-01/html/821-1601/gen-61908.html#gen-95948 This means atfork handlers may not actually acquire or release locks. See opinion by David Butenhof who was involved in the standardization effort of POSIX threads: http://groups.google.com/group/comp.programming.threads/msg/3a43122820983fde d) One consequence is that we can not assume third party libraries are safe to fork in multi-threaded program. It is likely their developers consider this scenario broken. e) It seems the general consensus across the WWW concerning this problem is that it has no solution and that a fork should be followed by exec as soon as possible. Some references: http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html http://austingroupbugs.net/view.php?id=62 http://sourceware.org/bugzilla/show_bug.cgi?id=4737 http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them 2) Python’s killer rabbit The standard library multiprocessing module does two things that force us into the rabbit hole; it creates worker threads and forks without exec. Therefore, any program that uses the multiprocessing module is a multi-threading forking program. One immediate problem is that a multiprocessing.Pool may fork from its worker thread in Pool._handle_workers(). This puts the forked child at risk of deadlock with any code that was run by the parent’s main thread (the entire program logic). More problems may be found with a code review. Other modules to look at are concurrent.futures.process (creates a worker thread and uses multiprocessing) and socketserver (ForkingMixIn forks without exec). 3) God bless the GIL a) Python signal handlers run synchronously in the interpreter loop of the main thread, so os.fork() will never be called from a POSIX signal handler. This means Python atfork prepare and parent handlers may run any code. The code run at the child is still restricted though and may deadlock into any acquired locks left behind by dead threads in the standard library or lower level third party libraries. b) Turns out the GIL also helps by synchronizing threads. Any lock held for the duration of a function call while the GIL is held will be released by the time os.fork() is called. But if a thread in the program calls a function that yields the GIL we are in la la land again and need to watch out step. 4) Landing gently at the bottom a) I think we should try to review and sanitize the worker threads of the multiprocessing module and other implicit worker threads in the standard library. Once we do (and since os.fork() is never run from a POSIX signal handler) the multiprocessing library should be safe to use in programs that do not start other threads. b) Then we should declare the user scenario of mixing the threading and multiprocessing modules as broken by design. c) Finally, optionally provide atfork API The atfork API can be used to refactor existing fork handlers in the standard library, provide handlers for modules such as the logging module that will reduce the risk of deadlock in existing programs, and can be used by developers who insist on mixing threading and forking in their programs. 5) Sanitizing worker threads in the multiprocessing module TODO :) (will try to post some ideas soon) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue874900] malloc
Changes by Nir Aides n...@winpdb.org: -- title: threading module can deadlock after fork - malloc ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue874900 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue874900] threading module can deadlock after fork
Changes by Nir Aides n...@winpdb.org: -- title: malloc - threading module can deadlock after fork ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue874900 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: Would you like to work on a patch to add an atfork mechanism? I will start with an attempt to generate a summary report on this rabbit hole of a problem, the views and suggestions raised by people here and what we may expect from atfork approach, its limitations, etc... I will also take a deeper look into the code. Hopefully my brain will not deadlock or fork while I am at it. more words, I know... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: Sorry, I fail to see how the import graph is related to the correct lock acquisition order. Some locks are created dynamically, for example. Import dependency is a reasonable heuristic to look into for inter-module locking order. The rational is explained in the following pthread_atfork man page: http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html A higher-level package may acquire locks on its own data structures before invoking lower-level packages. Under this scenario, the order specified for fork handler calls allows a simple rule of initialization for avoiding package deadlock: a package initializes all packages on which it depends before it calls the pthread_atfork() function for itself. (The rational section is an interpretation which is not part of the standard) A caveat is that since Python is an object oriented language it is more common than with C that code from a higher level module will be invoked by code from a lower level module, for example by calling an object method that was over-ridden by the higher level module - this actually happens in the logging module (emit method). That's why I asked for a specific API: when do you register a handler? When are they called? When are they reset? Read the pthread_atfork man page. The whole point of atfork is to avoid breaking invariants and introduce invalid state in the child process. If there is one thing we want to avoid, it's precisely reading/writting corrupted data from/to files, so eluding the I/O problem seems foolish to me. Please don't use insulting adjectives. If you think I am wrong, convincing me logically will do. you can avoid breaking invariants using two different strategies: 1) Acquire locks before the fork and release/reset them after it. 2) Initialize the module to some known state after the fork. For some (most?) modules it may be quite reasonable to initialize the module to a known state after the fork without acquiring its locks before the fork; this too is explained in the pthread_atfork man page: Alternatively, some libraries might be able to supply just a child routine that reinitializes the mutexes in the library and all associated states to some known value (for example, what it was when the image was originally executed). A critical section lock that protects in-memory data should not be held for long. Not necessarily. See for example I/O locks and logging module, which hold locks until I/O completion. Oops, I have always used the term critical section to describe a lock that protects data state as tightly as possible, ideally not even across function calls but now I see the Wikipedia defines one to protect any resource including IO. The logging module locks the entire emit() function which I think is wrong. It should let the derived handler take care of locking when it needs to, if it needs to at all. The logging module is an example for a module we should reinitialize after the fork without locking its locks before the fork. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: - what would be the API of this atfork() mechanism (with an example of how it would be used in the library)? The atfork API is defined in POSIX and Gregory P. Smith proposed a Python one above that we can look into. http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html We may need an API to reset a lock. - how do you find the correct order to acquire locks in the parent process? One option is to use the import graph to determine call order of atfork handlers. If a current std library does not fit into this scheme we can possibly fix it when writing its handlers. - what do you do with locks that can be held for arbitrarily long (e.g. I/O locks)? It is likely that such a lock does not need acquiring at the parent, and re-initializing the library in the child handler will do. A critical section lock that protects in-memory data should not be held for long. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: Well, I ping my view that we should: 1) Add general atfork() mechanism. 2) Dive into the std lib and add handlers one by one, that depending on case, either do the lock/init thing or just init the state of the library to some valid state in the child. Once this mechanism is in place and committed with a few obvious handlers such as the one for the logging library, other handlers can be added over time. Following this path we will slowly resolve the problem, handler by handler, without introducing the invalid state problem. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: I believe that the comp.programming.threads post from David Butenhof linked above explains why adding atfork() handlers isn't going to solve this. In Python atfork() handlers will never run from signal handlers, and if I understood correctly, Charles-François described a way to re-initialize a Python lock safely under that assumption. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1625] bz2.BZ2File doesn't support multiple streams
Nir Aides n...@winpdb.org added the comment: Right! I updated the patch and added a test for the aligned stream/buffer case. -- Added file: http://bugs.python.org/file22114/cpython-bz2-streams.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1625 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1625] bz2.BZ2File doesn't support multiple streams
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file22087/cpython-bz2-streams.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1625 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1625] bz2.BZ2File doesn't support multiple streams
Nir Aides n...@winpdb.org added the comment: Wait, the tests seem wrong. I'll post an update later today. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1625 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1625] bz2.BZ2File doesn't support multiple streams
Nir Aides n...@winpdb.org added the comment: False alarm; go ahead with the review. I took a look too early in the morning before caffeine kicked in. Note Lib/test/test_bz2.py was directly upgraded from bz2ms.patch. A note on bz2 behavior: A BZ2Decompressor object is only good for one stream; after that eof is set and it will refuse to continue to the next stream; this seems in line with bzip2 manual: http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html#bzDecompress -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1625 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1625] bz2.BZ2File doesn't support multiple streams
Changes by Nir Aides n...@winpdb.org: -- assignee: niemeyer - nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1625 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1625] bz2.BZ2File doesn't support multiple streams
Nir Aides n...@winpdb.org added the comment: Hi, I attach a patch to Python 3.3 Lib/bz2.py with updated tests: cpython-bz2-streams.patch -- keywords: +needs review stage: needs patch - patch review Added file: http://bugs.python.org/file22087/cpython-bz2-streams.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1625 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: Steffen, can you explain in layman's terms? On Sun, May 15, 2011 at 8:03 PM, Steffen Daode Nurpmeso rep...@bugs.python.org wrote: @ Charles-François Natali wrote (2011-05-15 01:14+0200): So if we really wanted to be safe, the only solution would be to forbid fork() in a multi-threaded program. Since it's not really a reasonable option But now - why this? The only really acceptable thing if you have control about what you are doing is the following: class SMP::Process /*! * \brief Daemonize process. *[.] * \note * The implementation of this function is not trivial. * To avoid portability no-goes and other such problems, * you may \e not call this function after you have initialized * Thread::enableSMP(), * nor may there (have) be(en) Child objects, * nor may you have used an EventLoop! * I.e., the process has to be a single threaded, synchronous one. * [.] */ pub static si32 daemonize(ui32 _daemon_flags=df_default); namespace SMP::POSIX /*! * \brief \fn fork(2). *[.] * Be aware that this passes by all \SMP and Child related code, * i.e., this simply \e is the system-call. * Signal::resetAllSignalStates() and Child::killAll() are thus if * particular interest; thread handling is still entirely up to you. */ pub static sir fork(void); Which kind of programs cannot be written with this restriction? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: Is it possible the following issue is related to this one? http://bugs.python.org/issue10037 - multiprocessing.pool processes started by worker handler stops working -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: I think that generally it is better to deadlock than corrupt data. 2) acquiring locks just before fork is probably one of the best way to deadlock (acquiring a lock we already hold, or acquiring a lock needed by another thread before it releases its own lock). Apart from adding dealock avoidance/recovery mechanisms - which would be far from trivial - I don't see how we could solve this, given that each library can use its own locks, not counting the user-created ones a) We know the correct locking order in Python's std libraries so the problem there is kind of solved. b) We can put the burden of other locks on application developers and since currently no one registers atfork handlers, there is no problem there yet. 4) Python locks differ from usual mutexes/semaphores in that they can be held for quite some time (for example while performing I/O). Thus, acquiring all the locks could take a long time, and users might get irritated if fork takes 2 seconds to complete. We only need a prepare handler to acquire locks that protect data from corruption. A lock synchronizing IO which is held for long periods may possibly be initialized in child without being acquired in a prepare handler; for example, a lock serializing logging messages. In other cases or in general an atfork handler may reset or reinitialize a library without acquiring locks in a prepare handler. 5) Finally, there's a fundamental problem with this approach, because Python locks can be released by a thread other than the one that owns it. Imagine this happens: T1 T2 lock.acquire() (do something without releasing lock) fork() lock.release() This is perfectly valid with the current lock implementation (for example, it can be used to implement a rendez-vous point so that T2 doesn't start processing before T1 forked worker processes, or whatever). But if T1 tries to acquire lock (held by T2) before fork, then it will deadlock, since it will never be release by T2. I think we do not need to acquire rendezvous locks in a prepare handler. Initializing locks in child after fork without acquiring them before the fork may result in corrupted program state and so is probably not a good idea. Yes, but in practise, I think that this shouldn't be too much of a problem. Also note that you can very well have the same type of problem with sections not protected explicitely by locks: for example, if you have a thread working exclusively on an object (maybe part of a threadpool), a fork can very well happen while the object is in an inconsistent state. Acquiring locks before fork won't help that. I think a worker thread that works exclusively on an object does not create the problem: a) If the fork thread eventually needs to read the object then you need synchronization. b) If the worker thread eventually writes data into file or DB then that operation will be completed at the parent process. To summarize I think we should take the atfork path. An atfork handler does not need to acquire all locks, but only those required by library logic, which the handler is aware of, and as a bonus it can be used to do all sort of stuff such as cleaning up, reinitializing a library, etc... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Nir Aides n...@winpdb.org added the comment: Hi, There seem to be two alternatives for atfork handlers: 1) acquire locks during prepare phase and unlock them in parent and child after fork. 2) reset library to some consistent state in child after fork. http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html Option (2) makes sense but is probably not always applicable. Option (1) depends on being able to acquire locks in locking order, but how can we determine correct locking order across libraries? Initializing locks in child after fork without acquiring them before the fork may result in corrupted program state and so is probably not a good idea. On a positive note, if I understand correctly, Python signal handler functions are actually run in the regular interpreter loop (as pending calls) after the signal has been handled and so os.fork() atfork handlers will not be restricted to async-signal-safe operations (since a Python fork is never done in a signal handler). http://pubs.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_04.html http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html It is therefore undefined for the fork handlers to execute functions that are not async-signal-safe when fork() is called from a signal handler. Opinion by Butenhof who was involved in the standardization effort of POSIX threads: http://groups.google.com/group/comp.programming.threads/msg/3a43122820983fde ...so how can we establish correct (cross library) locking order during prepare stage? Nir -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1625] bz2.BZ2File doesn't support multiple streams
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1625 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9504] signal.signal/signal.alarm not working as expected
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9504 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11743] Rewrite PipeConnection and Connection in pure Python
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11743 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9971] Optimize BufferedReader.readinto
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9971 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10037] multiprocessing.pool processes started by worker handler stops working
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10037 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9962] GzipFile doesn't have peek()
Nir Aides n...@winpdb.org added the comment: Should be min(n, 1024) instead of max(...) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9962 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9962] GzipFile doesn't have peek()
Nir Aides n...@winpdb.org added the comment: Right, I missed the change from self.max_read_chunk to 1024 (read_size). Should not peek() limit to self.max_read_chunk as read() does? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9962 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9962] GzipFile doesn't have peek()
Nir Aides n...@winpdb.org added the comment: Hi Antoine, BufferedIOBase is not documented to have peek(): http://docs.python.org/dev/py3k/library/io.html Small note about patch: 1) IOError string says read() on write-only..., should be peek() on write-only... ? 2) Should be min() in self._read(max(self.max_read_chunk, n)) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9962 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7467] The zipfile module does not check files' CRCs, including in ZipFile.testzip
Nir Aides n...@winpdb.org added the comment: I think patch may be simplified. Instead of keeping track of CRC offset, invoke it directly on the 'data' variable being added to _readbuffer. Also the call to _update_crc() before the return from read1() looks redundant. Finally, is it possible to determine end of file if length of 'data' (computed for crc) is 0? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7467 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7467] The zipfile module does not check files' CRCs, including in ZipFile.testzip
Nir Aides n...@winpdb.org added the comment: But you answered my question with code :) self._file_size is now unused and may be removed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7467 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file17356/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: Updated bfs.patch with BSD license and copyright notice. ! Current version patches cleanly and builds with Python revision svn r81201. Issue 7946 and proposed patches were put on hold indefinitely following this python-dev discussion: http://mail.python.org/pipermail/python-dev/2010-May/100115.html I would like to thank the Python developer community and in particular David and Antoine for a most interesting ride. Any party interested in sponsoring further development or porting patch to Python 2.x is welcome to contact me directly at n...@winpdb.org Nir -- Added file: http://bugs.python.org/file17504/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2281] Enhanced cPython profiler with high-resolution timer
Nir Aides n...@winpdb.org added the comment: Nir, would you be interested in looking at this? yes, I'll take a look, but will take me a few days. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2281 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Added file: http://bugs.python.org/file17370/nir-ccbench-linux.log ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: A link to ccbench results comparing old GIL, old GIL with long check interval, new GIL and BFS: http://bugs.python.org/file17370/nir-ccbench-linux.log Summary: Results for ccbench latency and bandwidth test run on Ubuntu Karmic 64bit, q9400 2.6GHz, all Python versions built with computed gotos optimization. Old GIL: Hi level of context switching and reduced performance. ~90ms IO latency with pure Python CPU bound background threads and low IO bandwidth results. Old GIL with sys.setcheckinterval(2500) as done by Zope: Context switching level back to normal. IO latency shoots through the roof. ~950ms (avg) is the maximum recordable value in this test since CPU load duration is 2sec. New GIL: The expected 5ms wait related IO latency and low IO bandwidth. BFS patch: Behaves. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file17330/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: Updated bfs.patch to patch cleanly updated py3k branch. Use: $ patch -p1 bfs.patch -- Added file: http://bugs.python.org/file17356/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file17195/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: Duck, here comes another update to bfs.patch. This one with some cleanups which simplify the code and improve behavior (on Windows XP), shutdown code, comments, and experimental use of TSC for timestamps, which eliminates timestamp reading overhead. TSC (http://en.wikipedia.org/wiki/Time_Stamp_Counter) is a fast way to get high precision timing read. On some systems this is what gettimeofday() uses under the hood while on other systems it will use HPET or another source which is slower, typically ~1usec, but can be higher (e.g. my core 2 duo laptop occasionally goes into a few hours of charging 3usec per HPET gettimeofday() call - god knows why) This overhead is incurred twice for every GIL release/acquire pair and can be eliminated with: 1) Hack the scheduler not to call gettimeofday() when no other threads are waiting to run, or 2) Use TSC on platforms it is available (the Linux BFS scheduler uses TSC). I took cycle.h pointed by the Wikipedia article on TSC for a spin and it works well on my boxes. It is BSD, (un)maintained? and includes implementation for a gazillion of platforms (I did not yet modify configure.in as it recommends). If it breaks on your system please ping with details. Some benchmarks running (Florent's) writes.py on Core 2 Quad q9400 Ubuntu 64bit: bfs.patch - 35K context switches per second, threads balanced, runtime is 3 times that of running IO thread alone: ~/dev/python$ ~/build/python/bfs/python writes.py t1 1.60293507576 1 t2 1.78533816338 1 t1 2.88939499855 2 t2 3.19518113136 2 t1 4.38062310219 3 t2 4.70725703239 3 t1 6.26874804497 4 t2 6.4078810215 4 t1 7.83273100853 5 t2 7.92976212502 5 t1 9.4341750145 6 t2 9.57891893387 6 t1 11.077393055 7 t2 11.164755106 7 t2 12.8495900631 8 t1 12.8979620934 8 t1 14.577999115 9 t2 14.5791089535 9 t1 15.9246580601 10 t2 16.1618289948 10 t1 17.365830183 11 t2 17.7345991135 11 t1 18.9782481194 12 t2 19.2790091038 12 t1 20.4994370937 13 t2 20.5710251331 13 21.0179870129 dabeaz_gil.patch - sometimes runs well but sometimes goes into high level of context switches (250K/s) and produces output such as this: ~/dev/python$ ~/build/python/dabeaz/python writes.py t1 0.742760896683 1 t1 7.50052189827 2 t2 8.63794493675 1 t1 10.1924870014 3 17.9419858456 gilinter2.patch - 300K context switches per second, bg threads starved: ~/dev/python$ ~/build/python/gilinter/python writes.py t2 6.1153190136 1 t2 11.7834780216 2 14.5995650291 -- Added file: http://bugs.python.org/file17330/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Added file: http://bugs.python.org/file17194/nir-ccbench-xp32.log ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file16967/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: I updated bfs.patch with improvements on Windows XP. The update disables priority boosts associated with the scheduler condition on Windows for CPU bound threads. Here is a link to ccbench results: http://bugs.python.org/file17194/nir-ccbench-xp32.log Summary: Windows XP 32bit q9400 2.6GHz Release build (no PG optimizations). Test runs in background, ccbench modified to run both bz2 and sha1. bfs.patch - seems to behave. gilinter2.patch single core: high latency, low IO bandwidth. dabeaz_gil.patch single core: low IO bandwidth. 4 cores: throughput threads starvation (balance), some latency, low IO bandwidth. -- Added file: http://bugs.python.org/file17195/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: Dave, The behavior of your patch on Windows XP/2003 (and earlier) might be related to the way Windows boosts thread priority when it is signaled. Try to increase priority of monitor thread and slice size. Another thing to look at is how to prevent Python CPU bound threads from (starving) messing up scheduling of threads of other processes. Maybe increasing slice significantly can help in this too (50ms++ ?). XP/NT/CE scheduling and thread boosting affect all patches and the current GIL undesirably (in different ways). Maybe it is possible to make your patch work nicely on these systems: http://www.sriramkrishnan.com/blog/2006/08/tale-of-two-schedulers-win_115489794858863433.html Vista and Windows 7 involve CPU cycle counting which results in more sensible scheduling: http://technet.microsoft.com/en-us/magazine/2007.02.vistakernel.aspx -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: On Thu, Apr 29, 2010 at 2:03 AM, David Beazley wrote: Wow, that is a *really* intriguing performance result with radically different behavior than Unix. Do you have any ideas of what might be causing it? Instrument the code and I'll send you a trace. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: On Wed, Apr 28, 2010 at 12:41 AM, Larry Hastings wrote: The simple solution: give up QPC and use timeGetTime() with timeBeginPeriod(1), which is totally reliable but only has millisecond accuracy at best. It is preferable to use a high precision clock and I think the code addresses the multi-core time skew problem (pending testing). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: Dave, there seems to be some problem with your patch on Windows: F:\devz:\dabeaz-wcg\PCbuild\python.exe y:\ccbench.py -b == CPython 3.2a0.0 (py3k) == == x86 Windows on 'x86 Family 6 Model 23 Stepping 10, GenuineIntel' == --- I/O bandwidth --- Background CPU task: Pi calculation (Python) CPU threads=0: 8551.2 packets/s. CPU threads=1: 26.1 ( 0 %) CPU threads=2: 26.0 ( 0 %) CPU threads=3: 37.2 ( 0 %) CPU threads=4: 33.2 ( 0 %) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment:
On Tue, Apr 27, 2010 at 12:23 PM, Charles-Francois Natali wrote:
@nirai
I have some more remarks on your patch:
- /* Diff timestamp capping results to protect against clock differences
* between cores. */
_LOCAL(long double) _bfs_diff_ts(long double ts1, long double ts0) {
I'm not sure I understand. You can have problem with multiple cores when
reading directly the
TSC register, but that doesn't affect gettimeofday. gettimeofday should be
reliable and accurate
(unless the OS is broken of course), the only issue is that since it's wall
clock time, if a process
like ntpd is running, then you'll run into problem
I think gettimeofday() might return different results on different cores as
result of kernel/hardware problems or clock drift issues in VM environments:
http://kbase.redhat.com/faq/docs/DOC-7864
https://bugzilla.redhat.com/show_bug.cgi?id=461640
In Windows the high-precision counter might return different results on
different cores in some hardware configurations (older multi-core processors).
I attempted to alleviate these problems by using capping and by using a python
time counter constructed from accumulated slices, with the assumption that IO
bound threads are unlikely to get migrated often between cores while running. I
will add references to the patch docs.
- did you experiment with the time slice ? I tried some higher values and got
better results,
without penalizing the latency. Maybe it could be interesting to look at it
in more detail (and
on various platforms).
Can you post more details on your findings? It is possible that by using a
bigger slice, you helped the OS classify CPU bound threads as such and improved
synchronization between BFS and the OS scheduler.
Notes on optimization of code taken, thanks.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7946
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: Dave, there seems to be a bug in your patch on Windows XP. It crashes in ccbench.py with the following output: python_d.exe y:\ccbench.py == CPython 3.2a0.0 (py3k) == == x86 Windows on 'x86 Family 6 Model 23 Stepping 10, GenuineIntel' == --- Throughput --- Pi calculation (Python) threads= 1: 840 iterations/s. balance Fatal Python error: ReleaseMutex(mon_mutex) failed threads= 2: 704 ( 83%)0.8167 threads= 3: 840 (100%)1.6706 threads= 4: 840 (100%)2. and the following stack trace: ntdll.dll!7c90120e() [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll] python32_d.dll!Py_FatalError(const char * msg) Line 2033 C python32_d.dll!gil_monitor(void * arg) Line 314 + 0x24 bytes C python32_d.dll!bootstrap(void * call) Line 122 + 0x7 bytes C msvcr100d.dll!_callthreadstartex() Line 314 + 0xf bytesC msvcr100d.dll!_threadstartex(void * ptd) Line 297 C kernel32.dll!7c80b729() -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: the scheduling function bfs_find_task returns the first task that has an expired deadline. since an expired deadline probably means that the scheduler hasn't run for a while, it might be worth it to look for the thread with the oldest deadline and serve it first, instead of stopping at the first one This is by design of BFS as I understand it. Next thread to run is either first expired or oldest deadline: http://ck.kolivas.org/patches/bfs/sched-BFS.txt Once a task is descheduled, it is put back on the queue, and an O(n) lookup of all queued-but-not-running tasks is done to determine which has the earliest deadline and that task is chosen to receive CPU next. The one caveat to this is that if a deadline has already passed (jiffies is greater than the deadline), the tasks are chosen in FIFO (first in first out) order as the deadlines are old and their absolute value becomes decreasingly relevant apart from being a flag that they have been asleep and deserve CPU time ahead of all later deadlines. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file16947/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: Yet another update to bfs.patch. I upload a variation on Florent's write test which prints progress of background CPU bound threads as: thread-name timestamp progress Here are some numbers from Windows XP 32bit with Intel q9400 (4 cores). Builds produced with VS express (alas - no optimizations, so official builds may behave differently). BFS - 33% CPU, 37,000 context switches per second z:\bfs\PCbuild\python.exe y:\writes.py t1 2.34400010109 0 t2 2.4213134 0 t1 4.6713134 1 t2 4.7963134 1 t1 7.0163242 2 t2 7.2036866 2 t1 9.375 3 t2 9.625 3 t1 11.703962 4 t2 12.0309998989 4 t1 14.046313 5 t2 14.421313 5 t1 16.407648 6 t2 16.7809998989 6 t1 18.782648 7 t2 19.125 7 t1 21.157648 8 t2 21.483676 8 t1 23.5 9 t2 23.858676 9 t1 25.858951 10 t2 26.233676 10 t1 28.2349998951 11 28.2189998627 gilinter - starves both bg threads and high rate of context switches. 45% CPU, 203,000 context switches per second z:\gilinter\PCbuild\python.exe y:\writes.py t1 13.0939998627 0 t1 26.421313 1 t1 39.812638 2 t1 53.1559998989 3 57.5470001698 PyCON - starves one bg thread and slow IO thread Py32 - starves IO thread as expected. Note, PyCON, gilinter and py32 starve the bg thread with Dave's original buffered write test as well - http://bugs.python.org/issue7946#msg101116 -- Added file: http://bugs.python.org/file16968/writes.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file16830/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment:
I uploaded an update to bfs.patch which improves behavior in particular on
non-Linux multi-core (4+) machines.
Hi Charles-Francois, Thanks for taking the time to review this patch!
- nothing guarantees that you'll get a msec resolution
Right, the code should behave well with low precision clocks as long as short
(sub-tick) tasks are not synchronized with the slice interval. There is a
related discussion of this problem in schedulers in the section on sub-tick
accounting in: http://ck.kolivas.org/patches/bfs/sched-BFS.txt
On which target systems can we expect not to have high precision clock?
- gettimeofday returns you wall clock time: if a process that modifies time
is running, e.g. ntpd, you'll likely to run into trouble. the value returned
is _not_ monotonic, but clock_gettime(CLOCK_MONOTONIC) is
- inline functions are used, but it's not ANSI
- static inline long double get_timestamp(void) {
struct timeval tv;
GETTIMEOFDAY(tv);
return (long double) tv.tv_sec + tv.tv_usec * 0.01;
}
I added timestamp capping to the code. timestamp is used for waiting and
therefore I think the source should be either CLOCK_REALTIME or gettimeofday().
`tstate-tick_counter % 1000` is replicating the behaviour of the old GIL,
which based its speculative operation on the number of elapsed opcodes (and
which also gave bad latency numbers on the regex workload).
I find this suspicous too. I haven't looked at the patch in detail, but what
does the number of elapsed opcodes offers you over the timesplice expiration
approach ?
More accurate yielding. It is possible a better mechanism can be thought of
and/or maybe it is indeed redundant.
It is thus recommended that a condition wait be enclosed in the equivalent of
a while loop that checks the predicate.
Done.
--
Added file: http://bugs.python.org/file16947/bfs.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7946
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file16710/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: Uploaded an update. -- Added file: http://bugs.python.org/file16830/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file16680/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: I upload a new update to bfs.patch which improves scheduling and reduces overhead. -- Added file: http://bugs.python.org/file16710/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: gilinter.patch has good IO latency in UDP test on my system when built with --with-computed-gotos: In [34]: %timeit -n3 client.work() 0.320 seconds (32782026.509 bytes/sec) 0.343 seconds (30561727.443 bytes/sec) 0.496 seconds (21154075.417 bytes/sec) 0.326 seconds (32171215.998 bytes/sec) 0.462 seconds (22701809.421 bytes/sec) 0.378 seconds (27722146.793 bytes/sec) 0.391 seconds (26826713.409 bytes/sec) 0.315 seconds (5858.720 bytes/sec) 0.281 seconds (37349508.136 bytes/sec) 3 loops, best of 3: 329 ms per loop -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file16663/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: I update bfs.patch. It now builds on Windows (and Posix). -- Added file: http://bugs.python.org/file16679/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file16679/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Added file: http://bugs.python.org/file16680/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: But on a busy system, won't measuring wall clock time rather than CPU time give bogus results? This was the motivation for using clock_gettime(). I tried the wall clock version under load (including on single core system) and it seems to behave. Now it remains to rationalize it :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: I upload an updated bfs.patch. Apply to updated py32 and ignore the error with: $ patch -fp1 bfs.patch $ ./configure Please give understandable benchmark numbers, including an explicit comparison with baseline 3.2, and patched 3.2 (e.g. gilinter.patch) Below. Please also measure single-thread performance, because it looks like you are adding significant work inside the core eval loop Removed most of it now. last bit will be removed soon. Do you need a hi-res clock? gettimeofday() already gives you microseconds. It looks like a bit of imprecision shouldn't be detrimental. I use clock_gettime() to get the thread running time to calculate slice depletion. Wall clock can not help with that. The magic number DEADLINE_FACTOR looks gratuitous (why 1.1^20 ?) To my understanding it controls the CPU load (~6) beyond which threads tend to expire. Since expired threads are handled in FIFO order, IO threads do not preempt them (IO threads are chronically expired). So beyond that load IO threads become less responsive. By the way, I would put COND_SIGNAL inside the LOCK_MUTEX / UNLOCK_MUTEX pair in bfs_yield(). Done. Here are benchmark results of the UDP test as timed with ipython, where client.work() is a single run of the client: System: Core 2 Duo (locked at 2.4 GHz) with Ubuntu Karmic 64 bit. Vanilla Python 3.2: * Note on my system the original problem discussed in this issue report does not manifest since conditions wake up threads according to OS scheduling policy. In [28]: %timeit -n3 client.work() 1.290 seconds (8127084.435 bytes/sec) 1.488 seconds (7045285.926 bytes/sec) 2.449 seconds (4281485.217 bytes/sec) 1.874 seconds (5594303.222 bytes/sec) 1.853 seconds (5659626.496 bytes/sec) 0.872 seconds (12023425.779 bytes/sec) 4.951 seconds (2117942.079 bytes/sec) 0.728 seconds (14409157.126 bytes/sec) 1.743 seconds (6016999.707 bytes/sec) 3 loops, best of 3: 1.53 s per loop gilinter.patch: In [31]: %timeit -n3 client.work() 5.192 seconds (2019676.396 bytes/sec) 1.613 seconds (6500071.475 bytes/sec) 3.057 seconds (3429689.199 bytes/sec) 3.486 seconds (3007596.468 bytes/sec) 4.324 seconds (2424791.868 bytes/sec) 0.964 seconds (10872708.606 bytes/sec) 3.510 seconds (2987722.960 bytes/sec) 1.362 seconds (7698999.458 bytes/sec) 1.013 seconds (10353913.920 bytes/sec) 3 loops, best of 3: 1.96 s per loop PyCON patch: In [32]: %timeit -n3 client.work() 2.483 seconds (4223256.889 bytes/sec) 1.330 seconds (7882880.263 bytes/sec) 1.737 seconds (6036251.315 bytes/sec) 1.348 seconds (7778296.679 bytes/sec) 0.983 seconds (10670811.638 bytes/sec) 1.419 seconds (7387226.333 bytes/sec) 1.057 seconds (9919412.977 bytes/sec) 2.483 seconds (4223205.791 bytes/sec) 2.121 seconds (4944231.292 bytes/sec) 3 loops, best of 3: 1.25 s per loop bfs.patch: In [33]: %timeit -n3 client.work() 0.289 seconds (36341875.356 bytes/sec) 0.271 seconds (38677439.991 bytes/sec) 0.476 seconds (22033958.947 bytes/sec) 0.329 seconds (31872974.070 bytes/sec) 0.478 seconds (21925125.894 bytes/sec) 0.242 seconds (43386204.271 bytes/sec) 0.213 seconds (49195701.418 bytes/sec) 0.309 seconds (33967467.196 bytes/sec) 0.256 seconds (41008076.688 bytes/sec) 3 loops, best of 3: 259 ms per loop Output of cpued.py test: Vanilla Python 3.2, gilinter.patch and PyCON patch all starve the pure Python threads and output the following: $ ~/build/python/python32/python cpued.py t0 0 False t1 0 False t2-interactive 0 True t2-interactive 1 True t2-interactive 2 True t2-interactive 3 True t2-interactive 4 True t2-interactive 5 True t2-interactive 6 True t2-interactive 7 True . . . Output from bfs.patch run: $ ~/build/python/bfs/python cpued.py t0 0 False t1 0 False t2-interactive 0 True t0 1 False t1 1 False t2-interactive 1 True t0 2 False t1 2 False t2-interactive 2 True t0 3 False t1 3 False t2-interactive 3 True . . . Note: I have not tested on other Posix systems, and expect to have some complications on Windows, since its thread timers are low resolution (10ms+), and there are issues with its high-precision wall clock. ...will soon know better. -- Added file: http://bugs.python.org/file16644/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file16634/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file16567/linux-7946.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: Ouch. CLOCK_THREAD_CPUTIME_ID is not a required part of the standard. Only CLOCK_REALTIME is guaranteed to exist. Right, however the man page at kernel.org says the following on CLOCK_THREAD_CPUTIME_ID: Sufficiently recent versions of glibc and the Linux kernel support the following clocks http://www.kernel.org/doc/man-pages/online/pages/man2/clock_getres.2.html The same statement shows up as early as 2003: http://www.tin.org/bin/man.cgi?section=3topic=clock_gettime However, if this is indeed a problem on some systems (none Linux?), then a fall back could be attempted for them. There could also be a problem on systems where the counter exists but has low resolution 10ms+ What platforms do you think this could be a problem on? By the way, it's not obvious cpued tests anything meaningful. I understand the bias you are trying to avoid but constructing artificial test cases is not very useful, because we are playing with heuristics and it's always possible to defeat some expectations. That's why benchmarks should try to model/represent real-world situations. I came up with cpued.py after reading the patches in an attempt to understand how they behave. In this case one thread is pure Python while the other occasionally releases the GIL, both CPU bound. I don't claim this is a real-world situation. However, it is a case in which bfs.patch behaves as expected. I've tried ccbench with your patch and there's a clear regression in latency numbers. Please specify system and test details so I can try to look into it. On my system ccbench behaves as expected: $ ~/build/python/bfs/python ccbench.py == CPython 3.2a0.0 (py3k) == == x86_64 Linux on '' == --- Throughput --- Pi calculation (Python) threads=1: 1252 iterations/s. threads=2: 1199 ( 95 %) threads=3: 1178 ( 94 %) threads=4: 1173 ( 93 %) regular expression (C) threads=1: 491 iterations/s. threads=2: 478 ( 97 %) threads=3: 472 ( 96 %) threads=4: 477 ( 97 %) SHA1 hashing (C) threads=1: 2239 iterations/s. threads=2: 3719 ( 166 %) threads=3: 3772 ( 168 %) threads=4: 3464 ( 154 %) --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 1 ms.) CPU threads=1: 0 ms. (std dev: 1 ms.) CPU threads=2: 0 ms. (std dev: 1 ms.) CPU threads=3: 0 ms. (std dev: 1 ms.) CPU threads=4: 0 ms. (std dev: 1 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 6 ms. (std dev: 0 ms.) CPU threads=2: 2 ms. (std dev: 2 ms.) CPU threads=3: 1 ms. (std dev: 0 ms.) CPU threads=4: 5 ms. (std dev: 7 ms.) Background CPU task: SHA1 hashing (C) CPU threads=0: 0 ms. (std dev: 1 ms.) CPU threads=1: 0 ms. (std dev: 1 ms.) CPU threads=2: 0 ms. (std dev: 1 ms.) CPU threads=3: 1 ms. (std dev: 1 ms.) CPU threads=4: 1 ms. (std dev: 0 ms.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: It's a dual-core Linux x86-64 system. But, looking at the patch again, the reason is obvious: #define CHECK_SLICE_DEPLETION(tstate) (bfs_check_depleted || (tstate tick_counter % 1000 == 0)) `tstate-tick_counter % 1000` is replicating the behaviour of the old GIL, which based its speculative operation on the number of elapsed opcodes (and which also gave bad latency numbers on the regex workload). The flag_check_depleted is there to work around this problem. It is raised by waiters which timeout. What distribution and version of GNU/Linux are you using? As for the CLOCK_THREAD_CPUTIME_ID clock, support was added to FreeBSD recently in version 7.1, which I guess is not good enough: http://www.freebsd.org/releases/7.1R/relnotes.html I did not find yet anything on Solaris. Do you know of an alternative way to measure thread running time on Posix? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment:
Well, on initial check the scheduler seems to work well with regular
gettimeofday() wall clock instead of clock_gettime().
:)
/* Return thread running time in seconds (with nsec precision). */
static inline long double get_thread_timestamp(void) {
return get_timestamp(); // wall clock via gettimeofday()
/*struct timespec ts;
clock_gettime(CLOCK_THREAD_CPUTIME_ID, ts);
return (long double) ts.tv_sec + ts.tv_nsec * 0.1;*/
}
Does it make things better on your system?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7946
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: Removed file: http://bugs.python.org/file16644/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: Uploaded an updated bfs.patch The latency problem was related to the --with-computed-gotos flag. I fixed it and it seems to work fine now. I also switched to gettimeofday() so it should work now on all Posix with high resolution timer. -- Added file: http://bugs.python.org/file16663/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: I upload bfs.patch To apply the patch use the following commands on updated python 3.2: $ patch -fp1 bfs.patch $ ./configure The patch replaces the GIL with a scheduler. The scheduler is a simplified implementation of the recent kernel Brain F**k Scheduler by the Linux hacker Con Kolivas: http://ck.kolivas.org/patches/bfs/sched-BFS.txt The goal of the Brain Fuck Scheduler, referred to as BFS from here on, is to completely do away with the complex designs of the past for the cpu process scheduler and instead implement one that is very simple in basic design. The main focus of BFS is to achieve excellent desktop interactivity and responsiveness without heuristics and tuning knobs that are difficult to understand, impossible to model and predict the effect of, and when tuned to one workload cause massive detriment to another. Con Kolivas is the hacker whose work inspired the current CFS scheduler of the Linux Kernel. On my core 2 duo laptop it performs as follows compared to the other patches: 1) Florent's writenums() test: ~same 2) UDP test: x6 faster 3) cpued test: works as expected, while the other patches starve the pure python threads. cpued test spins 3 threads, 2 of them pure python and the 3rd does time.sleep(0) every ~1ms: import threading import time def foo(n): while n 0: 'y' in 'x' * n n -= 1 def bar(sleep, name): for i in range(100): print (name, i, sleep) for j in range(300): foo(1500) if sleep: time.sleep(0) t0 = threading.Thread(target=bar, args=(False, 't0')) t1 = threading.Thread(target=bar, args=(False, 't1')) t2 = threading.Thread(target=bar, args=(True, 't2-interactive')) list(map(threading.Thread.start, [t0, t1, t2])) list(map(threading.Thread.join, [t0, t1, t2])) The patch is still work in progress. In particular: 1) I still need to add support for Windows. 2) It currently requires Posix clock_gettime() and assumes good timer resolution. 3) I only verified it builds on Ubuntu Karmic 64bit. 4) I still need to optimize it and address cleanup. The scheduler is very simple, straight forward and flexible, and it addresses the tuning problems discussed recently. I think it can be a good replacement to the GIL, since Python really needs a scheduler, not a lock. -- Added file: http://bugs.python.org/file16634/bfs.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: I tried Florent's modification to the write test and did not see the effect on my machine with an updated revision of Python32. I am running Ubuntu Karmic 64 bit. 7s - no background threads. 20s - one background thread. According to the following documentation the libc condition is using scheduling policy when waking a thread and not FIFO order: The following documentation suggests ordering in Linux is not FIFO: http://www.opengroup.org/onlinepubs/95399/functions/pthread_cond_timedwait.html#tag_03_518_08_06 I upload a quick and dirty patch (linux-7946.patch) to the new GIL just to reflect this by avoiding the timed waits. On my machine it behaves reasonably both with the TCP server and with the write test, but so does unpatched Python 3.2. I noticed high context switching rate with dave's priority GIL - with both tests it goes above 40K/s context switches. -- Added file: http://bugs.python.org/file16567/linux-7946.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Nir Aides n...@winpdb.org added the comment: I updated the patch with a small fix and increased the ticks countdown-to-release considerably. This seems to help the OS classify CPU bound threads as such and actually improves IO performance. -- Added file: http://bugs.python.org/file16570/linux-7946.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7946] Convoy effect with I/O bound threads and New GIL
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7946 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Nir Aides n...@winpdb.org added the comment: I actually meant how would you simulate zlib's absence on a system in which it is present? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Nir Aides n...@winpdb.org added the comment: The related scenario is a system without zlib. How do you suggest simulating this in test? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Nir Aides n...@winpdb.org added the comment: Unconsumed data is compressed data. If the part which handles unconsumed data does not work when zlib is available, then the existing tests would fail. In any case the unconsumed buffer is an implementation detail of zipfile. I see a point in adding a test to make sure zipfile behaves as expected when zlib is not available, but how? Also, on which systems is zlib missing? I don't see this mentioned in the zlib docs. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Nir Aides n...@winpdb.org added the comment: Uploaded an updated patch with read() which calls underlying stream enough times to satisfy required read size. -- Added file: http://bugs.python.org/file15941/zipfile_7610_py27_v5.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Nir Aides n...@winpdb.org added the comment: Right, removed MAX_N from read(); remains in read1(). If good, what versions of Python is this patch desired for? -- Added file: http://bugs.python.org/file15949/zipfile_7610_py27_v6.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Nir Aides n...@winpdb.org added the comment: I do not find the existing phrasing in the IO docs ambiguous, but since it is obviously possible to misinterpret it it would be good to clarify it. Can you suggest an alternate phrasing that would be clearer? Replace 'may' with 'will' or 'shall' everywhere the context indicates a mandatory requirement. Since this possibly affects the entire Python documentation, does it make sense to discuss this on python-dev? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Nir Aides n...@winpdb.org added the comment: May be a good idea to clear this up in the documentation. http://en.wiktionary.org/wiki/may#Verb (modal auxiliary verb, defective) To have permission to. Used in granting permission and in questions to make polite requests. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Nir Aides n...@winpdb.org added the comment: I uploaded an update for Python 2.7. * you should probably write `n = sys.maxsize` instead of `n = 1 31 - 1` sys.maxsize is 64 bit number on my system but the maximum value accepted by zlib's decompress() seems to be INT_MAX defined in pyport.h which equals the number I used. * ZipExtFile.read() should support `n=None` as a synonym to `n=-1` (read everything) Added * `bytes` as a variable name isn't very good since it's the built-in name for bytestrings in py3k Changed (old habits die hard). * in ZipExtFile.read(), it seems you have removed the adjustment for encrypted files (see `adjust read size for encrypted files since the first 12 bytes [etc.]`) Yes, was moved to the constructor. * is there a situation where the decompressor might return less bytes than expected? (after all compression doesn't /always/ compress, in unfavourable chunks of data it might actually expand things a bit) The documentation of io.BufferedIOBase.read() reads multiple raw reads may be issued to satisfy the byte count. I understood this language to mean satisfying read size is optional. Isn't it? -- Added file: http://bugs.python.org/file15882/zipfile_7610_py27_v4.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Nir Aides n...@winpdb.org added the comment: If the sequence of readaheads is ['a\r', '\nb\n'], the first use of the pattern will consume 'a', then the peek(2) will trigger a read() and the next use of the pattern will consume '\r\n'. I updated the patch and enhanced a little the inline comments to clear this up and added the suggested test. I also ventured into a rewrite of the read() function since it seems to contain various latent bugs / problems, probably the result of ages of slow refactoring. For example, decompressor flush() depends on self.eof flag which is never set and the decrypter is fed by the raw buffer which is in turn set by the decompresser, thus theoretically feeding decrypted data back to the decrypter. I also made it completely buffered so now small reads are significantly faster. If this patch is good, i'll prepare a patch for python 3.x and 2.6 -- Added file: http://bugs.python.org/file15822/zipfile_7610_py27.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Nir Aides n...@winpdb.org added the comment: Right, I was reading the 3.1 docs by mistake. I updated the patch. This time universal newlines are supported. On my dataset (75MB 650K lines log file) the readline() speedup is x40 for 'r' mode and x8 for 'rU' mode, and you can get an extra bump by using the io module wrappers. I added tests for interleaved use of readline() and read() (which add ~2 seconds to running time) -- Added file: http://bugs.python.org/file15739/zipfile_7610_py27.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Nir Aides n...@winpdb.org added the comment: Thank you. Are you sure the Shortcut common case in readline() is useful? BufferedIOBase.readline() in itself should be rather fast. On my dataset the shortcut speeds up readline() 400% on top of the default C implementation. I can take a look to why the C implementation is slow (although it is documented as slowish). Also, I'm not sure what happens in readline() in universal mode when the chunk ends with a '\r' and there's a '\n' in the following chunk (see the ugly check that your patch removes). Is there a test for that? The regular pattern returns either a line chunk or a newline (sequence) but not both. To read a line there is therefore a minimum of two peek() calls. One for the line content and the last for the newline. Since the peek is called with a value of 2, the newline sequence \r\n should be retrieved as is. There is no test for that. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Nir Aides n...@winpdb.org added the comment: I uploaded a possible patch for Python 2.7. The patch converts ZipExtFile into subclass of io.BufferedIOBase and drops most of the original implementation. However, the patch breaks current newline behavior of ZipExtFile. I figured this was reasonable because zipfile newline behavior: a) was introduced in Python 2.6 b) was incompatible with behavior of io module files. c) was not documented. Reading zip content as text with newline functionality may be done with io.TextIOWrapper. A bonus of sub classing io.BufferedIOBase is significantly better read performance. -- keywords: +patch Added file: http://bugs.python.org/file15723/zipfile_7610_py27.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7610] Cannot use both read and readline method in same ZipExtFile object
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7610 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6452] urllib2.Request() will not work with long authorization headers
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6452 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7544] Fatal error on thread creation in low memory condition
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7544 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7544] Fatal error on thread creation in low memory condition
Nir Aides n...@winpdb.org added the comment: Memory can be pre-allocated by thread_PyThread_start_new_thread() before thread is spawned. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7544 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7481] Failing to start a thread leaves zombie thread in initial state
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7481 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7540] urllib2 request does not update content length after new add_data
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7540 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7564] test_ioctl fails sometimes
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7564 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7519] CompileParser can't read files with BOM markers
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7519 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7503] multiprocessing AuthenticationError digest sent was rejected when pickling proxy
Changes by Nir Aides n...@winpdb.org: -- nosy: +nirai ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7503 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
