[sage-devel] Re: Slow pty's (was Re: libdivecomputer interfaces?)
On Nov 13, 3:00 am, Jef Driesen jefdrie...@telenet.be wrote: On 10/06/10 19:25, Linus Torvalds wrote: Greg, Alan, Hirofumi-san, I thought we long since (ie back last fall) fixed the latency problems withpty's, but there does seem to be something very fishy going on there still. On Thu, Jun 10, 2010 at 8:01 AM, Linus Torvalds torva...@linux-foundation.org wrote: On Sat, May 29, 2010 at 12:53 PM, Jef Driesenjefdrie...@telenet.be wrote BTW, now that I have your attention, could you maybe help me with a linux kernel problem I'm experiencing in this area? I reported the problem on LKML but got no response: http://www.divesoftware.org/libdc/simulator.html http://groups.google.com/group/linux.kernel/browse_thread/thread/5a2b... [ Hmm.. Testing.. ] Yeah, it's slow. Your test thing takes one and a quarter minutes for me. That's ridiculous. And no, we shouldn't need the low-latency flag, we're supposed to do this all automatically correctly. I'll talk to the tty people. This is clearly not a regression (it's been going on forever, I suspect), but taking over a minute to transfer just over half a MB of data over aptyseems crazy. Maybe it's not a kernel problem, and it's something done wrong by rx/sx/socat, I haven't looked at what they do. But since setting low_latency apparently helps (I didn't test that part, but I did test ridiculously slow), it sounds very much like something is still wrong in the kernel unless there is some really subtle timing issue in user space. From Jef's original lkml report linked to above: You can reproduce the problem by running these commands in three different terminals: # Terminal 1: Setup thepty's. socatPTY,link=/tmp/ttyS0PTY,link=/tmp/ttyS1 # Terminal 2: Send some data. dd if=/dev/urandom of=input.bin bs=538368 count=1 sx input.bin/tmp/ttyS0/tmp/ttyS0 # Terminal 2: Receive the data data. time rx output.bin/tmp/ttyS1/tmp/ttyS1 and yeah, it's pretty clear to see. A perf report on that receiving side just shows queue_delayed_work_on(), but that doesn't mean much. It's clearly just sleeping all the time... Any ideas? Linus Just out of curiosity, is there any progress on this issue? There was some discussion on NOHZ related changes in the remainder of this thread, but they don't appear to have fixed the problem I reported above. I still need to patch my kernel to set the low-latency flag to get decent performance. over on sage-devel, we discuss a similar issue. Sage (sagemath.org) relies on pty's (via the pexpect Python module http://www.noah.org/wiki/pexpect) to make different computer algebra systems talk to each other, and we find that on recent kernels provided by Debian and Ubuntu (2.6.32- ish) the performance of these channels is very slow, compared to older kernels (e.g. 2.6.18). http://groups.google.com/group/sage-devel/browse_thread/thread/d4b2ca2fe7ee0678 Just in case, Dmitrii I wish I could look into this myself, but unfortunately my kernel experience is still too limited. But if there is anything that I could do to help, just let me know. Thanks for your time. Jef -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Interrupting a doc test creating zombies?
Hi! I did some changes in expect.py, but apparently not so well, because then sage -t -verbose -force_lib devel/sage/sage/interfaces/expect.py sometimes (not always) resulted in a dead lock. But at least, the test was still interruptible by Ctrl-C. So, I thought everything was fine, but ps ax | grep python revealed that it wasn't: The sage-cleaner script was still present, but marked as S. And there were three further python processes marked as D. While the sage-cleaner process could be removed by kill, this won't work for the other three python processes. My question is not about the dead lock (I guess I can solve it myself), but about the zombie processes: Have you encountered before that interrupting a hanging test is problematic? Where could I start looking at for debugging the creation of zombies? Cheers, Simon -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Interrupting a doc test creating zombies?
On 2010-11-22 09:43, Simon King wrote: So, I thought everything was fine, but ps ax | grep python revealed that it wasn't: The sage-cleaner script was still present, but marked as S. And there were three further python processes marked as D. I think Z means zombie and D means that a process is waiting for I/O (I read D as disk even though it officially means non-interruptible sleep). I agree that interrupting doctests doesn't work as it should. It seems that the doctest process itself cannot be interrupted, it just looks for KeyboardInterrupts in the doctested process (well, that's my guess, I have not looked at the code). Anyway: if you find a good way of reproducing this problem, let me know. I am currently doing a big rewrite of the interrupt functionality in Sage, see #9678 (hopefully ready for Sage Days 27). Jeroen. -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Severe slowness of singular interface on some machines partially caused by _synchronize
for reference: I'm running Ubuntu 10.10 on a quad core, 2.6.35-22-generic 1. This is not an issue with select. This is an issue with the low_latency pty stuff that Simon mentioned. I tested this by modifying pexpect to use true nonblocking io (ie set the child_fd to O_NONBLOCK, and then just read() and ignore EAGAIN), and there still was a delay before read would return something other than EAGAIN. 2. There's nothing wrong with the singular interface. 3. Possible fix: for singular and gap (and anything else that doesn't need a full tty, ie doesn't use fancy screen stuff like top), perhaps we should use subprocess.Popen(... stdout=PIPE, stdin=PIPE, stderr=PIPE) and then communicate through the pipes. There shouldn't be any latency. I'm currently modifying pexpect to do this and will report my findings. - Alex -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Interrupting a doc test creating zombies?
Hi Jeroen!
On 22 Nov., 09:55, Jeroen Demeyer jdeme...@cage.ugent.be wrote:
I think Z means zombie and D means that a process is waiting for I/O
(I read D as disk even though it officially means non-interruptible
sleep).
OK, could very well be. There was a file created, but when I wanted to
remove it or touch it, rm resp. tmp was hanging as well, and could not
be deleted.
Anyway: if you find a good way of reproducing this problem, let me know.
Tricky...
It would not be a good idea to do this using my expect.py-patch: I
found that my error came from a race condition, thus can not be
reliably triggered.
But perhaps the following scenario fails consistently:
1. The test starts with creating a small file tmp.py containing,
for example
sage: singular._eval_line_using_file(def i; for(i=1;i=3;i++)
{ i=1;};)
''
or some other test that would run infinitely *and* uses the temporary
file associated with a an interface.
alarm(3) (or a sufficiently small number).
2. The test then says alarm(2)
3. It starts a sub-process sage -t tmp.py
4. Since the sub-process would run infinitely, it will receive a
KeyboardInterrupt, due to alarm(2).
5. Since the _local_tmpfile of the gap sub-sub-process is still open,
I reckon gap and the one python sub process will initially be locked.
And then the question is how one can test whether sage-cleaner can
cope with the locked processes. I have no idea. And I am even not sure
if the above scenario would reliably create a process that is D and
can not be terminated.
I am currently doing a big rewrite of the interrupt functionality in
Sage, see #9678 (hopefully ready for Sage Days 27).
Good!
Cheers,
Simon
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
Re: [sage-devel] Re: Severe slowness of singular interface on some machines partially caused by _synchronize
Am 22.11.2010 um 14:12 schrieb Alex Leone: for reference: I'm running Ubuntu 10.10 on a quad core, 2.6.35-22-generic 1. This is not an issue with select. This is an issue with the low_latency pty stuff that Simon mentioned. I tested this by modifying pexpect to use true nonblocking io (ie set the child_fd to O_NONBLOCK, and then just read() and ignore EAGAIN), and there still was a delay before read would return something other than EAGAIN. 2. There's nothing wrong with the singular interface. 3. Possible fix: for singular and gap (and anything else that doesn't need a full tty, ie doesn't use fancy screen stuff like top), perhaps we should use subprocess.Popen(... stdout=PIPE, stdin=PIPE, stderr=PIPE) and then communicate through the pipes. There shouldn't be any latency. I'm currently modifying pexpect to do this and will report my findings. Thanks for all your efforts about the performance of our interface. For Singular, there exists the libsingular interface. I would appreciate every help making it the perfect Singular interface, instead of optimizing the generic pexpect interface. Cheers, Michael - Alex -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org --- Dr. rer. nat. Michael Brickenstein Mathematisches Forschungsinstitut Oberwolfach gGmbH Schwarzwaldstr. 9 - 11 77709 Oberwolfach Tel.: 07834/979-31 Fax: 07834/979-38 -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: building sage took more then 24h now
I have a similarly modest system and my experience with UBUNTU 10.10 has been that the standard set of packages takes about 8 hours, maybe 6 1/2 of build and 1 1/2 of testing and document building. Anecdotal ? Yes, but 24h + to build some unknown package(s) is no surprise to me (-: You could monitor the log(s) as this build progresses. You MIGHT see some re-curing error that is causing numerous re-tries, etc. Same comment on the scripts, they may retry without limit on (partial)failure. It might be worth editing to limit the re-tries, example Atlas gives up after 5. Disclaimer; I am a newbie, regard my input as NOT advice (-: On Nov 16, 3:21 am, Johannes dajo.m...@web.de wrote: hi list, On Sunday started to install all experimental packages by sage -experimental | grep --color -vE 'INSTALLED|NOT|Type' | xargs sage -i and it's still building. Is this normal, or did i run into a endless loop? my configuration: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz 2GB Ram Linux neo 2.6.32-25-generic #44-Ubuntu SMP Fri Sep 17 20:26:08 UTC 2010 i686 GNU/Linux greatz Johannes -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Porting of LaTeX-Module for other TeX Systems
Hi List, just was wondering if there is some interest in porting the LaTeX-module for other systems too, as I am a ConTeXt user and love the simple and python stylish coding of Sage I would like to work on a fork of the LaTeX-module that could be natively used in ConTeXt, are there some others that like to work on it with me if not I hope to get later by questions some help here. Greetings -- Martin 'golodhrim' Scholz e-Mail: scholz@googlemail.com We don't really understand it, so we'll give it to the programmers. -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Severe slowness of singular interface on some machines partially caused by _synchronize
Hi Alex and Michael! On 22 Nov., 14:40, Michael Brickenstein brickenst...@mfo.de wrote: Am 22.11.2010 um 14:12 schrieb Alex Leone: for reference: I'm running Ubuntu 10.10 on a quad core, 2.6.35-22-generic ... 2. There's nothing wrong with the singular interface. The commands a=gap(1) a=gap(2) require to wait for a Gap prompt twice. The commands a=singular(1) a=singular(2) require to wait for a Singular prompt five times. So, there *is* something wrong with the Singular interface: The overhead is considerably more than with the other interfaces! And I do believe that this is worth a ticket (I think it is the last of the three tickets that I opened for this thread). 3. Possible fix: for singular and gap (and anything else that doesn't need a full tty, ie doesn't use fancy screen stuff like top), perhaps we should use subprocess.Popen(... stdout=PIPE, stdin=PIPE, stderr=PIPE) and then communicate through the pipes. There shouldn't be any latency. I'm currently modifying pexpect to do this and will report my findings. Would this be possible without too much effort? This sounds great! But the Singular interface having more overhead than other interfaces is orthogonal to the pexpect vs. pipes issue. For Singular, there exists the libsingular interface. I would appreciate every help making it the perfect Singular interface, instead of optimizing the generic pexpect interface. 1. Sage uses some pexpect interfaces that can not be replaced by C API right now, and some will never be replacable by a C API: Maxima and GAP belong to the first category IIRC; Maple, Magma and other interfaces belong to the second category (unless Magma changes to GPL...) So, optimizing the generic pexpect interface, as suggested by Alex, is a *very* interesting project. 2. IMHO, making libsingular the perfect Singular interface would include to re-write any Sage code that currently relies on the pexpect interface. And this would certainly be a lot of stuff to re-write. Best regards, Simon -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: SAGE64 + linux, SAGE64 in general
On 11/22/10 07:55 AM, Peter Jeremy wrote:
On 2010-Nov-20 19:09:00 -0800, John H Palmierijhpalmier...@gmail.com wrote:
Is that the compiler flag -m64 has no effect on other systems? Some
spkg-install files just check whether SAGE64 is set, not the platform,
and then they add -m64 to the flags. So if that does anything on
linux, then SAGE64 can be used on linux.
'-m64' is recognized by gcc and the SunPRO compilers. It's not recognized
by the HP Tru64 C compiler (which is natively 64-bit). I can't comment
on any other compilers.
-m64 is not reconised by the HP compiler on HP-UX (where one needs +DA2.OW or
+DD64 depending on the CPU type).
Neither is -m64 reconised by IBM's compiler on AIX, where one needs -q64.
So there are a few cases where -m64 is not appropriate. At some point in the
future, who knows what other systems or compilers will not want -m64?
Generically adding '-m64' when SAGE64 is set is definitely wrong because
not all compilers will support that flag. In addition, gcc treats x86
and x86_64 as different variants of the one architecture - so gcc on a
32-bit platform can compile x86_64 code. On 32-bit x86 Linux and *BSD,
using '-m64' will cause gcc to build x86_64 code - which the kernel won't
be able to execute - so this is highly undesirable.
But if SAGE64 is set to yes, then adding an appropriate flag should not be a
problem.
If SAGE64 is set to anything other than yes, and a compiler flag like -m64 was
added, then clearly that would be stupid.
Ideally, all skpg files should inherit {C,CPP,CXX,FC,LD}FLAGS from the
environment (adding spkg-specific options if required). This would
allow SAGE64 to be processed in one spot fairly early on in the build
- adding '-m64' or equivalent to {C,FCC,FC,LD}FLAGS, which is then
inherited by all succeeding spkg builds. This would remove a lot of
special-casing from the build.
Agreed.
I tried that before
http://trac.sagemath.org/sage_trac/ticket/7818
so every package would get a common set of flags, unless they were set
otherwise. Those flags were -g -Wall for gcc, but changed for non-GNU compilers
and 64-bit builds.
But doing this removes the -fno-strict-aliasing option from the gcc options that
Cython uses, which broke things all over the place.
IMHO, it is a fault of Cython. But that ticket caused so much hassle, I have
given up trying to fix it.
The CFLAG64 option is not an ideal solution but came about by the following set
of circumstances.
1) Micheal made the variable add -m64 *only* on OS X.
2) I change some instance of it so it worked on both Solaris and OS X, since I
knew both the Sun and GNU compilers accepted this option. At this point, I gave
no consideration to other platforms.
3) Later I since realised that (2) was a bit dumb, as potentially one could use
another operating system, and potentially the flag might not be -m64. AIX and
HP-UX are examples of this, but there may be others now or in the future.
So more recent changes made by me should allow any user-specificed flag(s) to be
set for 64-bit builds, but the will default to -m64 if SAGE64=yes.
Dave
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
Re: [sage-devel] Re: building sage took more then 24h now
i stopped it after 3 days. took to long just for some experiments. greatz Am 22.11.2010 15:48, schrieb RegB: I have a similarly modest system and my experience with UBUNTU 10.10 has been that the standard set of packages takes about 8 hours, maybe 6 1/2 of build and 1 1/2 of testing and document building. Anecdotal ? Yes, but 24h + to build some unknown package(s) is no surprise to me (-: You could monitor the log(s) as this build progresses. You MIGHT see some re-curing error that is causing numerous re-tries, etc. Same comment on the scripts, they may retry without limit on (partial)failure. It might be worth editing to limit the re-tries, example Atlas gives up after 5. Disclaimer; I am a newbie, regard my input as NOT advice (-: On Nov 16, 3:21 am, Johannes dajo.m...@web.de wrote: hi list, On Sunday started to install all experimental packages by sage -experimental | grep --color -vE 'INSTALLED|NOT|Type' | xargs sage -i and it's still building. Is this normal, or did i run into a endless loop? my configuration: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz 2GB Ram Linux neo 2.6.32-25-generic #44-Ubuntu SMP Fri Sep 17 20:26:08 UTC 2010 i686 GNU/Linux greatz Johannes -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Severe slowness of singular interface on some machines partially caused by _synchronize
On Mon, Nov 22, 2010 at 9:00 AM, Simon King simon.k...@uni-jena.de wrote: Hi Alex and Michael! On 22 Nov., 14:40, Michael Brickenstein brickenst...@mfo.de wrote: Am 22.11.2010 um 14:12 schrieb Alex Leone: for reference: I'm running Ubuntu 10.10 on a quad core, 2.6.35-22-generic ... 2. There's nothing wrong with the singular interface. The commands a=gap(1) a=gap(2) require to wait for a Gap prompt twice. The commands a=singular(1) a=singular(2) require to wait for a Singular prompt five times. So, there *is* something wrong with the Singular interface: The overhead is considerably more than with the other interfaces! And I do believe that this is worth a ticket (I think it is the last of the three tickets that I opened for this thread). 3. Possible fix: for singular and gap (and anything else that doesn't need a full tty, ie doesn't use fancy screen stuff like top), perhaps we should use subprocess.Popen(... stdout=PIPE, stdin=PIPE, stderr=PIPE) and then communicate through the pipes. There shouldn't be any latency. I'm currently modifying pexpect to do this and will report my findings. Would this be possible without too much effort? This sounds great! But the Singular interface having more overhead than other interfaces is orthogonal to the pexpect vs. pipes issue. For Singular, there exists the libsingular interface. I would appreciate every help making it the perfect Singular interface, instead of optimizing the generic pexpect interface. 1. Sage uses some pexpect interfaces that can not be replaced by C API right now, and some will never be replacable by a C API: Maxima and GAP belong to the first category IIRC; Maple, Magma and other interfaces belong to the second category (unless Magma changes to GPL...) So, optimizing the generic pexpect interface, as suggested by Alex, is a *very* interesting project. 2. IMHO, making libsingular the perfect Singular interface would include to re-write any Sage code that currently relies on the pexpect interface. And this would certainly be a lot of stuff to re-write. That's not clear to me. Maybe the full singular command prompt could be used via a library interface? After all, it is just a C++ program. The C interface to GAP that I wrote just involves rewriting the GAP read-eval-print loop (the command line interface) and turning it into C library calls. I literally just stick some data in where the read would put it, then tell GAP that it just read something in, so go evaluate it, etc. I mentioned this to Martin Albrecht (author of libsingular), and he said it would be more difficult to do that with Singular. But that doesn't mean it is impossible. It might be better to write something like this and not change any of Sage, than to have to rewrite a lot of Sage. -- William -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Alternate Lapack Versions?
In my development, I'm attempting to parallelize some code. However, the bottleneck is a call to numpy.linalg.lapack_lite.zgesv, that is the point where numpy calls LAPACK to solve my complex system of linear equations. Ideally I'd like to parallelize this, and I know the ScaLAPACK has a parallel version, pzgesv. Is ScaLAPACK supported by Sage, or are there any plans to support it in the future? If not, I'll have to look for workarounds. Thanks, Ethan Van Andel -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Alternate Lapack Versions?
In my development, I'm attempting to parallelize some code. However, the bottleneck is a call to numpy.linalg.lapack_lite.zgesv, that is the point where numpy calls LAPACK to solve my complex system of linear equations. Ideally I'd like to parallelize this, and I know the ScaLAPACK has a parallel version, pzgesv. Is ScaLAPACK supported by Sage, or are there any plans to support it in the future? If not, I'll have to look for workarounds. Technically sage doesn't ever call lapack directly. Most lapack calls are from numpy I think. So you would have to get numpy to compile against it. You probably could make a creative use of the SAGE_ATLAS variable to point to your implementation of f77blas/cblas/lapack you may have to create links appropriately. Francois -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Importing stuff from the doc namespace in the sage namespace
What is the best way to read variables from SAGE_ROOT/devel/sage/doc/common/builder.py from within the sage library? The file SAGE_ROOT/devel/sage/doc/common/builder.py contains things like SAGE_DOC = os.environ['SAGE_DOC'] LANGUAGES = ['en', 'fr'] SPHINXOPTS = PAPER = OMIT= [introspect] # docs/dirs to omit when listing and building 'all' and this needs to be read from SAGE_ROOT/devel/sage/sage/misc/sagedoc.py How to do this? (this relates to ticket #10200) Jeroen. -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Importing stuff from the doc namespace in the sage namespace
On Mon, Nov 22, 2010 at 12:30 PM, Jeroen Demeyer jdeme...@cage.ugent.be wrote: How to do this? (this relates to ticket #10200) You could add SAGE_ROOT/devel/sage/doc/common/ to sys.path, import the module, and then remove it from sys.path. --Mike -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: building sage took more then 24h now
On Mon, Nov 22, 2010 at 6:48 AM, RegB 2regburg...@earthlink.net wrote: I have a similarly modest system and my experience with UBUNTU 10.10 has been that the standard set of packages takes about 8 hours, maybe 6 1/2 of build and 1 1/2 of testing and document building. Anecdotal ? Yes, but 24h + to build some unknown package(s) is no surprise to me (-: You could monitor the log(s) as this build progresses. You MIGHT see some re-curing error that is causing numerous re-tries, etc. Same comment on the scripts, they may retry without limit on (partial)failure. It might be worth editing to limit the re-tries, example Atlas gives up after 5. Disclaimer; I am a newbie, regard my input as NOT advice (-: 6-8 hours seems like a long time for anything but an ancient system, I'd expect a core 2 duo at that speed to take more like 3-4 hours, (less if make -j3 is enabled). If ATLAS needs to be configured than that can more than double the time though... On Nov 16, 3:21 am, Johannes dajo.m...@web.de wrote: hi list, On Sunday started to install all experimental packages by sage -experimental | grep --color -vE 'INSTALLED|NOT|Type' | xargs sage -i and it's still building. Is this normal, or did i run into a endless loop? my configuration: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz 2GB Ram Linux neo 2.6.32-25-generic #44-Ubuntu SMP Fri Sep 17 20:26:08 UTC 2010 i686 GNU/Linux greatz Johannes -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Importing stuff from the doc namespace in the sage namespace
On 2010-11-22 21:41, Mike Hansen wrote:
On Mon, Nov 22, 2010 at 12:30 PM, Jeroen Demeyer jdeme...@cage.ugent.be
wrote:
How to do this? (this relates to ticket #10200)
You could add SAGE_ROOT/devel/sage/doc/common/ to sys.path, import the
module, and then remove it from sys.path.
--Mike
Wow. It turns out that SAGE_ROOT/devel/sage/doc/common is already in
the sys.path, so {{{from builder import OMIT}}} actually works!
Thanks a great lot!
Jeroen.
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
[sage-devel] Re: Severe slowness of singular interface on some machines partially caused by _synchronize
Hi William, On 22 Nov., 20:44, William Stein wst...@gmail.com wrote: 2. IMHO, making libsingular the perfect Singular interface would include to re-write any Sage code that currently relies on the pexpect interface. And this would certainly be a lot of stuff to re-write. That's not clear to me. Maybe the full singular command prompt could be used via a library interface? After all, it is just a C++ program. What I meant was: There is a difference between being able to do sth. and doing sth. I am rather sure that there is much code in Sage and probably third party code as well that uses Singular via pexpect interface. It is one thing to say to the authors of that code: Meanwhile you can do all this via libsingular (as much as I understood, it *is* now possible). But it is a completely different thing to make the authors change all their code so that it *does* use libsingular. Hence, if a small change in the singular pexpect interface reduces the overhead by 2/3 then a lot of code would benefit - thus, such small change is worth-while. The C interface to GAP that I wrote just involves rewriting the GAP read-eval-print loop (the command line interface) and turning it into C library calls. I literally just stick some data in where the read would put it, then tell GAP that it just read something in, so go evaluate it, etc. I didn't know that there was so much progress - I remember someone mentioning that certain internal aspects of GAP would make the creation of libgap difficult. Here is my personal list of pros and contras. Improving the text interfaces by replacing pexpect: - Such radical change (as suggested by Alex) would probably involve much new library code to be written. + AFAIK, text interfaces *will* remain important in Sage + Lots of existing code would benefit. Improving pexpect by reducing the latency time of pseudo terminals: - I doubt that Sage can change it, as this sounds like the job of a system administrator. + Users might be happy if they can find an explanation in the manual how they (or their sys admin) can improve the performance of all existing interface-intensive Sage programs. Reducing the particular overhead of the Singular interface: + The changes I am suggesting are small (doc tests pass, so, I guess I can post a patch tomorrow) + Lots of existing code would benefit. Make libsingular applicable in a broader range of functionality: + (a BIG plus) If something can be done via C API then it is certainly better than a text interface. - Existing code would not benefit. Everything has pros. So, simply one has to decide about one's priority. Mine is to reduce the latency time on my machine and to reduce the singular interface overhead, since that seems to involve both the most impact on my work in cohomology and the least effort. Cheers, Simon -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Slowness of www.sagenb.org
On 11/21/10 01:23 PM, tuxiano wrote: Please, don't disable www.sagenb.org I'm not an administrator of my office PC so I can't install SAGE and my company filters connections not on port 80 or 8080 so www.sagenb.org is the only Sage implementation that I can use. Perhaps other people are in my condition. Do they filter port 8000? That is not an uncommon port for HTTP traffic? http://t2nb.math.washington.edu:8000/ I used to run an SSH server on port 443 when my company filtered port 22!! If your company does filter 8080. is possible I could make http://t2nb.math.washington.edu:8000/ also work as http://t2nb.math.washington.edu:8080/ Thanks Tiziano On Nov 21, 3:18 am, Jeroen Demeyerjdeme...@cage.ugent.be wrote: On 2010-11-20 22:45, William Stein wrote: (1) I simply totally disablewww.sagenb.org, Maybe keepwww.sagenb.orgas it is but remove the link from the Sage front page? Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Severe slowness of singular interface on some machines partially caused by _synchronize
I wrote a small script (see attached) that reads ~1 second of the stdout of
a command opened with subprocess.Popen(). For some reason singular doesn't
like this and refuses to show any output. However gap works fine:
singular output:
{{{
SAGE_ROOT=/home/alex/progs/src/sage-4.6.1.alpha2
(sage subshell) P5Q-E:sage-4.6.1.alpha2 alex$ python test_Popen.py
/home/alex/progs/src/sage-4.6.1.alpha2/local/bin/Singular-3-1-1 -t
Starting subprocess with args=
['/home/alex/progs/src/sage-4.6.1.alpha2/local/bin/Singular-3-1-1', '-t']
p_pid=30070 p_stdinfd=4 p_stdoutfd=5 p_stderrfd=None
p.terminate()
returncode=-15
}}}
gap output:
{{{
SAGE_ROOT=/home/alex/progs/src/sage-4.6.1.alpha2
(sage subshell) P5Q-E:sage-4.6.1.alpha2 alex$ python test_Popen.py gap
Starting subprocess with args=
['gap']
p_pid=30073 p_stdinfd=4 p_stdoutfd=5 p_stderrfd=None
Got:
Got:
# ## ### ###
# ##
## # #
### # ## #
## # # ## ##
## ## #####
## # ##
# # # ###
# ### ###
# ### ## ##
# ### ## #
#
# # #
#
## # #
#
## #
##### #
### #
# # # #
Information at: http://www.gap-system.org
Try '?help' for help. See also '?copyright' and '?authors'
Loading the library. Please be patient, this may take a while.
Got:
GAP4, Version: 4.4.12 of 17-Dec-2008, x86_64-unknown-linux-gnu-gcc
Got:
gap
p.terminate()
returncode=-15
}}}
So it seems singular needs a tty.
- Alex
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
import errno
import fcntl
import os
import subprocess
import sys
import time
def set_nonblocking(fd):
fl = fcntl.fcntl(fd, fcntl.F_GETFL)
fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
def check_completion(p, timeout=0.5):
endt = time.time() + timeout
while time.time() endt:
if p.poll() is not None:
return True
return False
def stop_subprocess(p):
if check_completion(p):
return
for _ in range(4):
print p.terminate()
p.terminate()
if check_completion(p):
return
for _ in range(4):
print p.kill()
p.kill()
if check_completion(p):
return
print Could not kill subprocess! Please kill -9 %d % (p.pid)
def _main():
if len(sys.argv) 2:
print Usage: %s [--strace] command ... % (sys.argv[0])
sys.exit(2)
if sys.argv[1] == '--strace':
args = [strace] + sys.argv[2:]
else:
args = sys.argv[1:]
print Starting subprocess with args=
print args
p = subprocess.Popen(args,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
shell=True)
p_stdinfd = p.stdin.fileno()
p_stdoutfd = p.stdout.fileno()
print p_pid=%s p_stdinfd=%s p_stdoutfd=%s p_stderrfd=%s % (p.pid,
p_stdinfd, p_stdoutfd, p.stderr.fileno() if p.stderr else p.stderr)
set_nonblocking(p_stdoutfd)
endt = time.time() + 1.0
while time.time() endt:
try:
s = os.read(p_stdoutfd, 4096)
if not s: # length zero string returned - EOF.
print Got EOF
break
print Got:
print + \n.join(s.splitlines())
except OSError as e:
if e.errno == errno.EAGAIN or e.errno == errno.EWOULDBLOCK:
continue
raise
stop_subprocess(p)
print returncode=%s % (p.returncode)
if __name__ == '__main__':
_main()
Re: [sage-devel] Re: Severe slowness of singular interface on some machines partially caused by _synchronize
On Mon, Nov 22, 2010 at 2:56 PM, Alex Leone acle...@gmail.com wrote: I wrote a small script (see attached) that reads ~1 second of the stdout of a command opened with subprocess.Popen(). For some reason singular doesn't like this and refuses to show any output. However gap works fine: pexpect (and other expect variants) were written specifically to avoid deadlocks which you are most likely seeing with Singular. For more info on this, check out http://effbot.org/pyfaq/how-do-i-run-a-subprocess-with-pipes-connected-to-both-input-and-output.htm --Mike -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Severe slowness of singular interface on some machines partially caused by _synchronize
This should be the problem:
1. If you start singular normally, it prints version info, etc to stdout and
then waits for stdin. So if we read stdout, we should get something
immediately after the process is started.
2. We set stdout to nonblocking (in the python process), so read() doesn't
block.
3. I ran strace to see what was happening. For some reason singular is
trying to read from stdin before it prints anything to stdout.
relevant strace details from the python process:
{{{
pipe([3, 4])= 0
pipe([5, 6])= 0
pipe([7, 8])= 0
fcntl(8, F_GETFD) = 0
fcntl(8, F_SETFD, FD_CLOEXEC) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7f8aaf3eb9d0) = 30213
close(8)= 0
close(3)= 0
close(6)= 0
...
fcntl(5, F_GETFL) = 0 (flags O_RDONLY)
fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
read(5, 0x1a9c8a4, 4096)= -1 EAGAIN (Resource temporarily
unavailable)
... (this read is repeated while the python process tries to read any output
from the singular process for about a second)
...
kill(30213, SIGTERM)= 0
...
--- SIGCHLD (Child exited) @ 0 (0) ---
...
}}}
relevant strace details from the clone()-ed singular process:
{{{
close(4)= 0
close(5)= 0
close(7)= 0
dup2(3, 0) = 0
dup2(6, 1) = 1
dup2(6, 2) = 2
close(3)= 0
close(6)= 0
execve(/home/alex/progs/src/sage-4.6.1.alpha2/local/bin/Singular-3-1-1,
[Singular-3-1-1, -t], [/* 81 vars */]) = 0
...
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff17936e60) = -1 EINVAL (Invalid
argument)
...
fstat(0, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7ff5da40b000
read(0, 0x7ff5da40b000, 4096) = ? ERESTARTSYS (To be restarted)
--- SIGTERM (Terminated) @ 0 (0) ---
}}}
The above output was produced by
{{{
strace -o strace_python_singular_-t_noshell.txt -ff -F python test_Popen.py
Singular-3-1-1 -t
}}}
- Alex
On Mon, Nov 22, 2010 at 3:02 PM, Mike Hansen mhan...@gmail.com wrote:
pexpect (and other expect variants) were written specifically to avoid
deadlocks which you are most likely seeing with Singular. For more
info on this, check out
http://effbot.org/pyfaq/how-do-i-run-a-subprocess-with-pipes-connected-to-both-input-and-output.htm
--Mike
- Alex
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
Re: [sage-devel] Re: Severe slowness of singular interface on some machines partially caused by _synchronize
Sorry, I meant that deadlock *shouldn't* be problem. The problem is the call to read(0, )... by the singular process before it writes anything to stdout (eg write(1, )). - Alex -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Alternate Lapack Versions?
On 11/22/10 1:48 PM, Ethan Van Andel wrote: In my development, I'm attempting to parallelize some code. However, the bottleneck is a call to numpy.linalg.lapack_lite.zgesv, that is the point where numpy calls LAPACK to solve my complex system of linear equations. Ideally I'd like to parallelize this, and I know the ScaLAPACK has a parallel version, pzgesv. Is ScaLAPACK supported by Sage, or are there any plans to support it in the future? If not, I'll have to look for workarounds. I would definitely start by asking on the numpy list about this. It sounds interesting. Jason -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Porting of LaTeX-Module for other TeX Systems
On Mon, 22 Nov 2010 at 03:45PM +, Martin 'golodhrim' Scholz wrote: just was wondering if there is some interest in porting the LaTeX-module for other systems too, as I am a ConTeXt user and love the simple and python stylish coding of Sage I would like to work on a fork of the LaTeX-module that could be natively used in ConTeXt, are there some others that like to work on it with me if not I hope to get later by questions some help here. I would certainly like to support ConTeXt. I don't know much about it, though. I don't think we need a fork; if the code for ConTeXt support gets written, we'll just included in with the usual SageTeX distribution. I know that PGF/TikZ support plain TeX, LaTeX, and ConTeXt, so this is certainly possible. I won't have time to work on this myself in the near future, but patches are welcome! Dan -- --- Dan Drake - http://mathsci.kaist.ac.kr/~drake --- signature.asc Description: Digital signature
Re: [sage-devel] Re: Severe slowness of singular interface on some machines partially caused by _synchronize
Am 22.11.2010 um 23:56 schrieb Alex Leone: Loading the library. Please be patient, this may take a while. Got: GAP4, Version: 4.4.12 of 17-Dec-2008, x86_64-unknown-linux-gnu-gcc Got: gap p.terminate() returncode=-15 }}} So it seems singular needs a tty. - Alex sage -singular --help Singular version 3-1-0 -- a CAS for polynomial computations. Usage: Singular-3-1-0 [options] [file1 [file2 ...]] Options: ... -t --no-tty Do not redefine the terminal characteristics ... Cheers, Michael -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org test_Popen.py --- Dr. rer. nat. Michael Brickenstein Mathematisches Forschungsinstitut Oberwolfach gGmbH Schwarzwaldstr. 9 - 11 77709 Oberwolfach Tel.: 07834/979-31 Fax: 07834/979-38 -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
