Re: OT processes and cpu use
On Mon, Feb 19, 2007 at 09:56:26PM +, Tyler Smith wrote: I have a general question about processes in GNU/Linux. I'm working with PAUP, which is a command-line tool for calculating phylogenies from various data types (gene sequencies, proteins etc.). This sort of analysis routinely takes hours to days of processing. I find it is convenient to run paup in *shell* mode inside emacs -nw, allowing me to easily yank commands from various files into the paup process. Since I routinly run on a 486, I have some hands-on experience getting the best performance on long-running jobs (e.g. bzip2 of a huge tar). What you should look at is minimizing the number of other processes running. Each one takes time slices in addition to whatever it is doing. Each time-slice means the processor has to change its context. It all takes time. If you run paup from within emacs that means that emacs takes some slices. If its within X, then thats a whole loaf of slices. You're compute-bound so don't do anything that takes computing. If you need to work on your thesis, either use another computer or do it from a text console rather than X. Or use another computer. What about the possibility of upgrading the CPU on your motherboard? Look at your programme and see why its needing so much system time. Each system call means a context switch and other overhead. Then look at the program in more detail, e.g. profile it, and tighten it up. In what language is it written? Switch to Fortran for part or all? Good luck. Doug. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT processes and cpu use
On Fri, 23 Feb 2007 01:59:14 -0500 Kevin Mark [EMAIL PROTECTED] wrote: On Wed, Feb 21, 2007 at 03:27:57PM -0500, Celejar wrote: On Wed, 21 Feb 2007 03:15:45 GMT Tyler Smith [EMAIL PROTECTED] wrote: [snip] Actually, I do pop into firefox and xpdf once in a while to check a reference, which is not possible without X, but your point is well You could actually pipe the pdf through a pdf to text / html converter (e.g. poppler-utils or xpdf-utils) and use a TUI html browser (e.g. lynx, links, links2) from the cli. You forgot one more thing: fb I use fbi and fbgs. fbgs can 'view' ps and pdf. Cool. I had browsed through the archives lookng for a pdf reader that didn't depend on X but I missed 'fbi', presumably since I was using aptitude which searches only package names by default. Lazy, I know :). I notice now, however, that 'fbi' doesn't provide the virtual package 'pdf-viewer'; the only ones that do are 'evince', the xpdf packages, 'kpdf', 'gv', 'kghostview', and 'viewpdf'. Is this a bug in 'pdf-viewer'? Celejar -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT processes and cpu use
On Fri, Feb 23, 2007 at 09:45:36AM -0500, Celejar wrote: On Fri, 23 Feb 2007 01:59:14 -0500 Kevin Mark [EMAIL PROTECTED] wrote: On Wed, Feb 21, 2007 at 03:27:57PM -0500, Celejar wrote: On Wed, 21 Feb 2007 03:15:45 GMT Tyler Smith [EMAIL PROTECTED] wrote: [snip] Actually, I do pop into firefox and xpdf once in a while to check a reference, which is not possible without X, but your point is well You could actually pipe the pdf through a pdf to text / html converter (e.g. poppler-utils or xpdf-utils) and use a TUI html browser (e.g. lynx, links, links2) from the cli. You forgot one more thing: fb I use fbi and fbgs. fbgs can 'view' ps and pdf. Cool. I had browsed through the archives lookng for a pdf reader that didn't depend on X but I missed 'fbi', presumably since I was using aptitude which searches only package names by default. Lazy, I know :). I notice now, however, that 'fbi' doesn't provide the virtual package 'pdf-viewer'; the only ones that do are 'evince', the xpdf packages, 'kpdf', 'gv', 'kghostview', and 'viewpdf'. Is this a bug in 'pdf-viewer'? Celejar That's a good question. I've seem IIRC in the mailcap db different tests for X and non-X but nothing for an FB environment. Because FB is not TUI and not GUI. And I'm not sure of a test to use to check for it. So many be it could be marked as 'Provides: pdf-viewer', but I'd expect some confusion about it. -- | .''`. == Debian GNU/Linux == | my web site: | | : :' : The Universal |mysite.verizon.net/kevin.mark/| | `. `' Operating System| go to counter.li.org and | | `-http://www.debian.org/ |be counted! #238656 | | my keysever: subkeys.pgp.net | my NPO: cfsg.org | signature.asc Description: Digital signature
Re: OT processes and cpu use
On Wed, 21 Feb 2007 03:15:45 GMT Tyler Smith [EMAIL PROTECTED] wrote: [snip] Actually, I do pop into firefox and xpdf once in a while to check a reference, which is not possible without X, but your point is well You could actually pipe the pdf through a pdf to text / html converter (e.g. poppler-utils or xpdf-utils) and use a TUI html browser (e.g. lynx, links, links2) from the cli. Celejar -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT processes and cpu use
On Wed, Feb 21, 2007 at 03:27:57PM -0500, Celejar wrote: On Wed, 21 Feb 2007 03:15:45 GMT Tyler Smith [EMAIL PROTECTED] wrote: [snip] Actually, I do pop into firefox and xpdf once in a while to check a reference, which is not possible without X, but your point is well You could actually pipe the pdf through a pdf to text / html converter (e.g. poppler-utils or xpdf-utils) and use a TUI html browser (e.g. lynx, links, links2) from the cli. You forgot one more thing: fb I use fbi and fbgs. fbgs can 'view' ps and pdf. -- | .''`. == Debian GNU/Linux == | my web site: | | : :' : The Universal |mysite.verizon.net/kevin.mark/| | `. `' Operating System| go to counter.li.org and | | `-http://www.debian.org/ |be counted! #238656 | | my keysever: subkeys.pgp.net | my NPO: cfsg.org | signature.asc Description: Digital signature
Re: OT processes and cpu use
On Tue, 20 Feb 2007 01:47:21 GMT Tyler Smith [EMAIL PROTECTED] wrote: [snip] Thanks for the suggestion. As I'm working with one machine I do eliminate all processes I don't need, but I keep X, fluxbox, and a terminal or two open so I can work on my thesis in emacs while I'm waiting for paup to run. I imagine if I reconfig my startup stuff I could login without X, start running paup, then start X while I'm working, and kill X when I'm done, leaving paup undisturbed. I'll take a look at this. Thanks! Even if you start paup when you are already in X, you can start it from within GNU screen, which will allow you to detach it from the console and leave it running (and then to kill X), and then to reattach it later. Celejar -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT processes and cpu use
Tyler Smith [EMAIL PROTECTED] writes: [snip] Sorry for reposting, but I should add that my concern stems in part from top reporting that 27% of the CPU is being used by users, and 73% by system, which conflicts with the line item for paup, a user process, which shows 99% CPU use. So I'm not sure if paup is using 99% or 27% of the CPU: Cpu(s): 26.7%us, 73.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 14385 tyler 25 0 141m 140m 660 R 99.7 9.3 82:34.35 paup This is a bit odd, as others have indicated this merely means that this process spends over 70% of it's time in system mode. This means that it spends a lot of time producing output or communicating with other processes or what have you. If this program does purely numerical analysis it should not be doing this, 'strace' or 'valgrind' might tell you more. -- ..__o Øyvind .. _`\(,http://www.darkside.no/olr/ .. (_)/(_) ... biciclare necesse est ...
Re: OT processes and cpu use
On Tue, Feb 20, 2007 at 01:47:21AM +, Tyler Smith wrote: On 2007-02-19, Andrew Sackville-West [EMAIL PROTECTED] wrote: All I can suggest is that you eliminate *every* process that you don't need in order to do your thing. If you're running emacs -nw, then you must be running X in some form, do you need it? If not, get rid of it -- X will on occaision demand some cpu. If it works out to a couple % over a couple days, you're looking at an h our or so.=20 Thanks for the suggestion. As I'm working with one machine I do eliminate all processes I don't need, but I keep X, fluxbox, and a terminal or two open so I can work on my thesis in emacs while I'm waiting for paup to run. I imagine if I reconfig my startup stuff I could login without X, start running paup, then start X while I'm working, and kill X when I'm done, leaving paup undisturbed. I'll take a look at this. Thanks! I don't doubt that you've got a pretty light-weight setup there, but you haven't mentioned anything that requires you to run X yet... :) just pointing it out. A signature.asc Description: Digital signature
Re: OT processes and cpu use
On 2/19/07, Tyler Smith [EMAIL PROTECTED] wrote: When I check with top, it appears that paup is using 99% of the CPU when I do this, which is as it should be. I just wanted to check with more experienced people to verify that running paup from within emacs is not going to incur a performance hit. The analysis I have running at the moment (heuristic search) will take more than 5 hours, and if that works properly I'll be continuing on with a follow-up (bootstrap) that will take several days, so tweaking performance is a non-trivial concern. You can examine the performance hit pretty directly. If you've got a sufficiently short job (on the order of an hour or two), you can try running it both within and apart from emacs. For a one hour job, every minute extra that it takes in emacs should translate to about two hours over a five-day job. That's assuming that everything scales nicely, of course. You might also want to run once outside of emacs and check top -- you might be incurring more context switches when running in emacs, or incurring more page faults, which could account for the high proportion of system time. -- Michael A. Marsh http://www.umiacs.umd.edu/~mmarsh http://mamarsh.blogspot.com -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT processes and cpu use
Øyvind Røtvold wrote:
Tyler Smith [EMAIL PROTECTED] writes:
[snip]
Sorry for reposting, but I should add that my concern stems in part
from top reporting that 27% of the CPU is being used by users, and 73%
by system, which conflicts with the line item for paup, a user
process, which shows 99% CPU use. So I'm not sure if paup is using 99%
or 27% of the CPU:
Cpu(s): 26.7%us, 73.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
14385 tyler 25 0 141m 140m 660 R 99.7 9.3 82:34.35 paup
This is a bit odd, as others have indicated this merely means that
this process spends over 70% of it's time in system mode. This means
Yep.
that it spends a lot of time producing output or communicating with
other processes or what have you. If this program does purely
numerical analysis it should not be doing this, 'strace' or 'valgrind'
might tell you more.
Umm, not a valid conclusion, though the idea to try to optimize
is reasonable. He may be doing lots of disc reading, all of which
is in cache.
Mike
--
p=p=%c%s%c;main(){printf(p,34,p,34);};main(){printf(p,34,p,34);}
Oppose globalization and One World Governments like the UN.
This message made from 100% recycled bits.
You have found the bank of Larn.
I can explain it for you, but I can't understand it for you.
I speak only for myself, and I am unanimous in that!
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT processes and cpu use
Andrew Sackville-West wrote:
I don't doubt that you've got a pretty light-weight setup there, but
you haven't mentioned anything that requires you to run X yet... :)
just pointing it out.
Even if he shut down X, he isn't going to see any speedup. He's
already using all the CPU.
Mike
--
p=p=%c%s%c;main(){printf(p,34,p,34);};main(){printf(p,34,p,34);}
Oppose globalization and One World Governments like the UN.
This message made from 100% recycled bits.
You have found the bank of Larn.
I can explain it for you, but I can't understand it for you.
I speak only for myself, and I am unanimous in that!
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT processes and cpu use
On 2007-02-20, Mike McCarty [EMAIL PROTECTED] wrote: Andrew Sackville-West wrote: I don't doubt that you've got a pretty light-weight setup there, but you haven't mentioned anything that requires you to run X yet... :) just pointing it out. Even if he shut down X, he isn't going to see any speedup. He's already using all the CPU. Actually, I do pop into firefox and xpdf once in a while to check a reference, which is not possible without X, but your point is well made. I should get more comfortable managing multiple processes from a single terminal, as an alternative to alt-tab bouncing around xterms. I tried running Paup in an xterm outside of emacs, and it showed the same split between user and system cpu%, which actually fluctuates between almost 99%us and +/-20%us. So whatever it's doing, it's not because of emacs. I also checked the paup output itself, which is more informative than I realized. It appears to examine about 35,000,000 trees a minute under either configuration (xterm or inside emacs), so that doesn't seem to make much difference there. After a nine hour timed analysis analysis it reported having used only 8:48 of CPU time. This was while I was working on some trees from previous analyses with Inkscape, so I guess I can live with the lost 12 minutes in that case. I think you guys have confirmed my suspicion that I'm not going to gain too much by stripping down any further, at least not if I'm going to continue to work on this machine while the analysis runs. Thanks for the informative responses, it's reassuring for those of us working in the isolation of our own homes to get a bit of feedback! -- Regards, Tyler Smit -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT processes and cpu use
On 2007-02-19, Tyler Smith [EMAIL PROTECTED] wrote: Hi, I have a general question about processes in GNU/Linux. I'm working with PAUP, which is a command-line tool for calculating phylogenies from various data types (gene sequencies, proteins etc.). This sort of analysis routinely takes hours to days of processing. I find it is convenient to run paup in *shell* mode inside emacs -nw, allowing me to easily yank commands from various files into the paup process. When I check with top, it appears that paup is using 99% of the CPU when I do this, which is as it should be. I just wanted to check with more experienced people to verify that running paup from within emacs is not going to incur a performance hit. The analysis I have running at the moment (heuristic search) will take more than 5 hours, and if that works properly I'll be continuing on with a follow-up (bootstrap) that will take several days, so tweaking performance is a non-trivial concern. Any comments on my current set-up or suggestions for ways to tweak performance in any way would be most welcome! Sorry for reposting, but I should add that my concern stems in part from top reporting that 27% of the CPU is being used by users, and 73% by system, which conflicts with the line item for paup, a user process, which shows 99% CPU use. So I'm not sure if paup is using 99% or 27% of the CPU: Cpu(s): 26.7%us, 73.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 14385 tyler 25 0 141m 140m 660 R 99.7 9.3 82:34.35 paup -- Regards, Tyler Smit -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT processes and cpu use
On Mon, Feb 19, 2007 at 09:56:26PM +, Tyler Smith wrote: Hi, Hi, I have a general question about processes in GNU/Linux. I'm working with PAUP, which is a command-line tool for calculating phylogenies from various data types (gene sequencies, proteins etc.). This sort of analysis routinely takes hours to days of processing. I find it is convenient to run paup in *shell* mode inside emacs -nw, allowing me to easily yank commands from various files into the paup process. When I check with top, it appears that paup is using 99% of the CPU when I do this, which is as it should be. I just wanted to check with If your process is already taking the whole cpu, i don't see how you could improve its performance. more experienced people to verify that running paup from within emacs is not going to incur a performance hit. The analysis I have running at the moment (heuristic search) will take more than 5 hours, and if if you're really looking at many days of processing, it may be worth your while to run some test in different configurations and timing them. Saving a few % here and there maybe worthwhile. All I can suggest is that you eliminate *every* process that you don't need in order to do your thing. If you're running emacs -nw, then you must be running X in some form, do you need it? If not, get rid of it -- X will on occaision demand some cpu. If it works out to a couple % over a couple days, you're looking at an h our or so. .02 A signature.asc Description: Digital signature
Re: OT processes and cpu use
Tyler Smith wrote:
On 2007-02-19, Tyler Smith [EMAIL PROTECTED] wrote:
Sorry for reposting, but I should add that my concern stems in part
from top reporting that 27% of the CPU is being used by users, and 73%
by system, which conflicts with the line item for paup, a user
process, which shows 99% CPU use. So I'm not sure if paup is using 99%
or 27% of the CPU:
This is not a conflict. There are no such thing as user processes
and system processes. You are looking at different kinds of breakdown.
Cpu(s): 26.7%us, 73.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
14385 tyler 25 0 141m 140m 660 R 99.7 9.3 82:34.35 paup
What this says is that paup is using all (99.7%) of the CPU, and that
of this 26.7% is spent executing in user mode, and 73.3% is
spent executing in supervisor mode. Ordinary programs switch
the operating mode of the processor by making system calls.
The time allocated to the process is partly in user mode, and
partly in system mode. If you read the manual page for time
you'll get a little more information.
HTH
Mike
--
p=p=%c%s%c;main(){printf(p,34,p,34);};main(){printf(p,34,p,34);}
Oppose globalization and One World Governments like the UN.
This message made from 100% recycled bits.
You have found the bank of Larn.
I can explain it for you, but I can't understand it for you.
I speak only for myself, and I am unanimous in that!
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT processes and cpu use
On 2007-02-20, Mike McCarty [EMAIL PROTECTED] wrote: Tyler Smith wrote: On 2007-02-19, Tyler Smith [EMAIL PROTECTED] wrote: Sorry for reposting, but I should add that my concern stems in part from top reporting that 27% of the CPU is being used by users, and 73% by system, which conflicts with the line item for paup, a user process, which shows 99% CPU use. So I'm not sure if paup is using 99% or 27% of the CPU: This is not a conflict. There are no such thing as user processes and system processes. You are looking at different kinds of breakdown. Cpu(s): 26.7%us, 73.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 14385 tyler 25 0 141m 140m 660 R 99.7 9.3 82:34.35 paup What this says is that paup is using all (99.7%) of the CPU, and that of this 26.7% is spent executing in user mode, and 73.3% is spent executing in supervisor mode. Ordinary programs switch the operating mode of the processor by making system calls. The time allocated to the process is partly in user mode, and partly in system mode. If you read the manual page for time you'll get a little more information. HTH Thanks, that does help. I went through the docs for 'top' but didn't find what I needed. -- Regards, Tyler Smit -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: OT processes and cpu use
On 2007-02-19, Andrew Sackville-West [EMAIL PROTECTED] wrote: All I can suggest is that you eliminate *every* process that you don't need in order to do your thing. If you're running emacs -nw, then you must be running X in some form, do you need it? If not, get rid of it -- X will on occaision demand some cpu. If it works out to a couple % over a couple days, you're looking at an h our or so.=20 Thanks for the suggestion. As I'm working with one machine I do eliminate all processes I don't need, but I keep X, fluxbox, and a terminal or two open so I can work on my thesis in emacs while I'm waiting for paup to run. I imagine if I reconfig my startup stuff I could login without X, start running paup, then start X while I'm working, and kill X when I'm done, leaving paup undisturbed. I'll take a look at this. Thanks! -- Regards, Tyler Smit -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
