Re: [Python-Dev] Point of building without threads?
Yury V. Zaytsev, 08.01.2013 10:28: > Moreover, these days there is a clear trend towards OpenMP, so it has > become even harder to pressure the manufacturers to fix threads, because > they have 101 argument why you should port your code to OpenMP instead. I can't see OpenMP being an *alternative* to threads. You can happily acquire and release the GIL from OpenMP threads, for example. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
Yury V. Zaytsev wrote: > On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote: > > > > I guess a long time ago, threading support in operating systems wasn't > > very widespread, but these days all our supported platforms have it. > > Is it still useful for production purposes to configure > > --without-threads? Do people use this option for something else than > > curiosity of mind? > > I hope that the intent behind asking this question was more of being > curious, rather then considering dropping --without-threads: > unfortunately, multithreading was, still is and probably will remain > troublesome on many supercomputing platforms. > > Often, once a new supercomputer is launched, as a developer you get a > half-baked C/C++ compiler with threading support broken to the point > when it's much easier to not use it altogether [*] rather than trying to > work around the compiler quirks. Out of curiosity: Do these incomplete compilers have any problem with either stdint.h or static inline functions in header files? Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
On 1/10/2013 8:54 AM, Nick Coghlan wrote: On Wed, Jan 9, 2013 at 7:01 AM, Yury V. Zaytsev wrote: Anyways, the point is that it's a recurrent problem, and we hit it every single time with each new machine, so I hope you can understand why I decided to post, even though the original message appeared back in May. Thank you for speaking up. Easier porting to new platforms is certainly one of the reasons we keep that capability, and it's helpful to have it directly confirmed that there *are* users out there that benefit from it :) If there is not currently a comment in the setup code explaining this, perhaps it would be a good idea, lest some future developer be tempted to delete the option. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
On Wed, Jan 9, 2013 at 7:01 AM, Yury V. Zaytsev wrote: > Anyways, the point is that it's a recurrent problem, and we hit it every > single time with each new machine, so I hope you can understand why I > decided to post, even though the original message appeared back in May. Thank you for speaking up. Easier porting to new platforms is certainly one of the reasons we keep that capability, and it's helpful to have it directly confirmed that there *are* users out there that benefit from it :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
On Tue, 2013-01-08 at 15:09 +0100, Antoine Pitrou wrote: > > The original e-mail is quite old (it was sent in May) :-) I'm sorry about that! I've just stumbled upon this thread and got scared that --without-threads might be going away soon... We've just been porting Python to a new supercomputer, so that we can use Python interface to our simulation software and, at the moment, this is the only way we can get it to work due to the deficiencies of the available software stack (which will hopefully be fixed in the future). Anyways, the point is that it's a recurrent problem, and we hit it every single time with each new machine, so I hope you can understand why I decided to post, even though the original message appeared back in May. -- Sincerely yours, Yury V. Zaytsev ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
2013/1/8 Trent Nelson : > On Tue, Jan 08, 2013 at 06:15:45AM -0800, Stefan Krah wrote: >> Trent Nelson wrote: >> > All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. >> > Without it, they all wedge in some way or another. (That should be >> > fixed*/investigated, but, until then, yeah, --without-threads allows >> > for a slightly more useful (but still broken) test suite run on >> > these platforms.) >> > >> > [*]: I suspect the problem with at least OpenBSD is that their >> > userland pthreads implementation just doesn't cut it; there >> > is no hope for the really technical tests that poke and >> > prod at things like correct signal handling and whatnot. >> >> For OpenBSD the situation should be fixed in the latest release: >> >>http://www.openbsd.org/52.html#new >> >> I haven't tried it myself though. > > Interesting! I'll look into upgrading the existing Snakebite > OpenBSD slaves (they're both at 5.1). Oooh yes, many bugs has been fixed by the implementation of threads in the kernel (rthreads in OpenBSD 5.2)! Just one recent example. On OpenBSD 4.9, FD_CLOEXEC doesn't work with fork()+exec() whereas it works with exec(); on OpenBSD 5.2, both cases work as expected. http://bugs.python.org/issue16850#msg179294 Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
On Tue, Jan 08, 2013 at 06:15:45AM -0800, Stefan Krah wrote: > Trent Nelson wrote: > > All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. > > Without it, they all wedge in some way or another. (That should be > > fixed*/investigated, but, until then, yeah, --without-threads allows > > for a slightly more useful (but still broken) test suite run on > > these platforms.) > > > > [*]: I suspect the problem with at least OpenBSD is that their > > userland pthreads implementation just doesn't cut it; there > > is no hope for the really technical tests that poke and > > prod at things like correct signal handling and whatnot. > > For OpenBSD the situation should be fixed in the latest release: > >http://www.openbsd.org/52.html#new > > I haven't tried it myself though. Interesting! I'll look into upgrading the existing Snakebite OpenBSD slaves (they're both at 5.1). Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
Trent Nelson wrote: > All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. > Without it, they all wedge in some way or another. (That should be > fixed*/investigated, but, until then, yeah, --without-threads allows > for a slightly more useful (but still broken) test suite run on > these platforms.) > > [*]: I suspect the problem with at least OpenBSD is that their > userland pthreads implementation just doesn't cut it; there > is no hope for the really technical tests that poke and > prod at things like correct signal handling and whatnot. For OpenBSD the situation should be fixed in the latest release: http://www.openbsd.org/52.html#new I haven't tried it myself though. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
Le Tue, 8 Jan 2013 09:02:00 -0500, Trent Nelson a écrit : > [ Weird, I can't see your original e-mail Antoine; hijacking > Yury's reply instead. ] The original e-mail is quite old (it was sent in May) :-) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
[ Weird, I can't see your original e-mail Antoine; hijacking Yury's reply instead. ] On Tue, Jan 08, 2013 at 01:28:25AM -0800, Yury V. Zaytsev wrote: > On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote: > > > > I guess a long time ago, threading support in operating systems wasn't > > very widespread, but these days all our supported platforms have it. > > Is it still useful for production purposes to configure > > --without-threads? Do people use this option for something else than > > curiosity of mind? All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. Without it, they all wedge in some way or another. (That should be fixed*/investigated, but, until then, yeah, --without-threads allows for a slightly more useful (but still broken) test suite run on these platforms.) [*]: I suspect the problem with at least OpenBSD is that their userland pthreads implementation just doesn't cut it; there is no hope for the really technical tests that poke and prod at things like correct signal handling and whatnot. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
Le Tue, 08 Jan 2013 10:28:25 +0100, "Yury V. Zaytsev" a écrit : > On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote: > > > > I guess a long time ago, threading support in operating systems > > wasn't very widespread, but these days all our supported platforms > > have it. Is it still useful for production purposes to configure > > --without-threads? Do people use this option for something else than > > curiosity of mind? > > I hope that the intent behind asking this question was more of being > curious, rather then considering dropping --without-threads: > unfortunately, multithreading was, still is and probably will remain > troublesome on many supercomputing platforms. I was actually asking this question in the hope that we could perhaps simplify our range of build options (and the corresponding C #define's), but you made a convincing point that we should keep the --without-threads option :-) Thank you Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote: > > I guess a long time ago, threading support in operating systems wasn't > very widespread, but these days all our supported platforms have it. > Is it still useful for production purposes to configure > --without-threads? Do people use this option for something else than > curiosity of mind? I hope that the intent behind asking this question was more of being curious, rather then considering dropping --without-threads: unfortunately, multithreading was, still is and probably will remain troublesome on many supercomputing platforms. Often, once a new supercomputer is launched, as a developer you get a half-baked C/C++ compiler with threading support broken to the point when it's much easier to not use it altogether [*] rather than trying to work around the compiler quirks. Of course, the situation improves over the lifetime of each particular computer, but usually, when everything is halfway working, the computer itself becomes obsolete, so there is not much point in using it anymore. Moreover, these days there is a clear trend towards OpenMP, so it has become even harder to pressure the manufacturers to fix threads, because they have 101 argument why you should port your code to OpenMP instead. HTH. [*]: Another usual candidates for being broken beyond repair are the linker, especially when it comes to shared libraries, and support for advanced C++ language features, such as templates... -- Sincerely yours, Yury V. Zaytsev ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
On Thu, 10 May 2012 20:23:08 +0200 Stefan Krah wrote: > Antoine Pitrou wrote: > > On Wed, 9 May 2012 11:26:29 +0200 > > Stefan Krah wrote: > > > Antoine Pitrou wrote: > > > > > _decimal is about 12% faster without threads, because the expensive > > > > > thread local context can be disabled. > > > > > > > > If you cached the last thread id along with the corresponding context, > > > > perhaps it could speed things up in most scenarios? > > > > > > Nice. This reduces the speed difference to about 4%! > > > > Note that you don't need the actual thread id, the Python thread state > > is sufficient: PyThreadState_GET should be a simply variable lookup in > > release builds. > > I've tried both ways now and the speed gain is roughly the same. > > Perhaps the interpreter as a whole is slightly faster --without-threads? > That would explain the remaining speed difference of 4%. It may be. Can you try other benchmarks? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
Stefan Krah wrote: > > > Nice. This reduces the speed difference to about 4%! > > > > Note that you don't need the actual thread id, the Python thread state > > is sufficient: PyThreadState_GET should be a simply variable lookup in > > release builds. > > I've tried both ways now and the speed gain is roughly the same. > > Perhaps the interpreter as a whole is slightly faster --without-threads? > That would explain the remaining speed difference of 4%. Actually this seems to be the case: In the benchmark floats are also about 3% faster without threads. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
Antoine Pitrou wrote: > On Wed, 9 May 2012 11:26:29 +0200 > Stefan Krah wrote: > > Antoine Pitrou wrote: > > > > _decimal is about 12% faster without threads, because the expensive > > > > thread local context can be disabled. > > > > > > If you cached the last thread id along with the corresponding context, > > > perhaps it could speed things up in most scenarios? > > > > Nice. This reduces the speed difference to about 4%! > > Note that you don't need the actual thread id, the Python thread state > is sufficient: PyThreadState_GET should be a simply variable lookup in > release builds. I've tried both ways now and the speed gain is roughly the same. Perhaps the interpreter as a whole is slightly faster --without-threads? That would explain the remaining speed difference of 4%. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
On Wed, 9 May 2012 11:26:29 +0200 Stefan Krah wrote: > Antoine Pitrou wrote: > > > _decimal is about 12% faster without threads, because the expensive > > > thread local context can be disabled. > > > > If you cached the last thread id along with the corresponding context, > > perhaps it could speed things up in most scenarios? > > Nice. This reduces the speed difference to about 4%! Note that you don't need the actual thread id, the Python thread state is sufficient: PyThreadState_GET should be a simply variable lookup in release builds. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
Antoine Pitrou wrote: > > _decimal is about 12% faster without threads, because the expensive > > thread local context can be disabled. > > If you cached the last thread id along with the corresponding context, > perhaps it could speed things up in most scenarios? Nice. This reduces the speed difference to about 4%! Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
On Tue, 8 May 2012 19:40:32 +0200 Stefan Krah wrote: > Antoine Pitrou wrote: > > I guess a long time ago, threading support in operating systems wasn't > > very widespread, but these days all our supported platforms have it. > > Is it still useful for production purposes to configure > > --without-threads? Do people use this option for something else than > > curiosity of mind? > > _decimal is about 12% faster without threads, because the expensive > thread local context can be disabled. If you cached the last thread id along with the corresponding context, perhaps it could speed things up in most scenarios? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
Antoine Pitrou wrote: > I guess a long time ago, threading support in operating systems wasn't > very widespread, but these days all our supported platforms have it. > Is it still useful for production purposes to configure > --without-threads? Do people use this option for something else than > curiosity of mind? _decimal is about 12% faster without threads, because the expensive thread local context can be disabled. On OpenBSD threading leads to strange problems like delayed signals in the REPL http://bugs.python.org/issue8714 . Without threads these problems don't occur. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
> > I guess a long time ago, threading support in operating systems wasn't very > widespread, but these days all our supported platforms have it. > Is it still useful for production purposes to configure --without-threads? Do > people use this option for something else than curiosity of mind? For EVE Online, we started out not using threads but relying solely on tasklets. We only added thread supports perhaps five years ago. Other embedded projects _might_ be omitting thread support for a leaner interpreter, but I'm not sure the difference is that large. K ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
On Mon, May 7, 2012 at 9:49 PM, Antoine Pitrou wrote: > I guess a long time ago, threading support in operating systems wasn't > very widespread, but these days all our supported platforms have it. > Is it still useful for production purposes to configure > --without-threads? Do people use this option for something else than > curiosity of mind? Gentoo (of course) allows users to build Python without threads; I'm not aware of anyone depending on that, but I sent out a quick question to gentoo-dev. Cheers, Dirkjan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
> I guess a long time ago, threading support in operating systems wasn't > very widespread, but these days all our supported platforms have it. > Is it still useful for production purposes to configure > --without-threads? Do people use this option for something else than > curiosity of mind? At work, I'm working on embedded systems (television set top boxes) with a Linux kernel with the GNU C library, and we do use threads! I'm not sure that Python runs on slower/smaller systems because they have other constrains like having very few memory, maybe no MMU and not using the glibc but µlibc for example. There is the "python-on-a-chip" project. It is written from scratch and is very different from CPython. I don't think that it uses threads. http://code.google.com/p/python-on-a-chip/ Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Point of building without threads?
Hello, I guess a long time ago, threading support in operating systems wasn't very widespread, but these days all our supported platforms have it. Is it still useful for production purposes to configure --without-threads? Do people use this option for something else than curiosity of mind? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
