Volker Armin Hemmann <[email protected]> posted
[email protected], excerpted below, on Mon,
18 May 2009 17:56:19 +0200:
>> Wil Reichert wrote:
>> >
>> > I've got PORTAGE_NICENESS="15" set in my make.conf. Makes all
>> > emerges unnoticable on my system.
>
> 19 should be better. Once upon a time it marked portage as 'sched_batch'
> which was theoretically speeding up compiling.
Indeed. A nice of 19, the lowest priority possible, puts stuff in batch
mode. While it's lower priority and doesn't get as many time-slices to
work with, when it does get them, they are longer. This is perfect for
something that's not particularly interactive but IS quite CPU intensive,
as switching tasks at each timeslice is inefficient, so doing it less
frequently means more time is actually spent on-task. The tradeoff is
responsiveness, but for background/batch priority tasks, responsiveness
is assumed not to be an issue, so they can get the long timeslices to
increase efficiency with very little downside at all.
There's also ionice, which (depending on i/o scheduler) is normally
automatically set within the best-effort class based on normal/cpu nice.
However, setting ionice -c3 for a task puts it in idle priority class,
meaning it won't get disk i/o time unless no other program (not also of
idle class) has used the disk for a defined grace period. But to my
knowledge, ionice idle has no "batch benefit" as does normal nice.
But I didn't know until reading Nikos' post that there was a way to set
that with portage -- and have it apply to child processes as it would
need to do, as well. Thus I've not tried this and am simply copying it
here from his post, so everything's in one place:
PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"
There's also something that can be done with the kernel (from 2.6.27
IIRC?? maybe .26 ??), that effectively gives each username its own CPU
quota (with root by default getting double the normal user default).
This means that no matter how a user prioritizes his tasks or what he's
trying to run, even a load of several hundred, if another user has stuff
to run, they get to use their quota and the user trying to hog simply
can't.
In menuconfig, general setup, ensure group CPU scheduler is enabled, then
enable scheduling for sched_other (at least, the RR/FIFO option is
special case, see its help for a pointer at the documentation), and if
need be, set the basis for grouping tasks to user_id, but that's the
default I think anyway.
(There's also control group scheduling which is somewhat different and
more complicated. It's for servers running multiple containers
instances, etc, so way beyond the level we're talking here. I'm
mentioning it just to say don't have them both on at the same time unless
you know what you are doing. If you're learning anything from this at
all, stick with the simple stuff and choose the user based scheduling.)
Once you have a kernel configured for that up and running, the knobs to
tweak it are in the /sys/kernel/uids/<user_ID>/cpu_share files, where
<user_ID> is the numeric user_ID for that user. The user_ID subdirs (and
their cpu_share files) will be dynamically created and erased as those
users are seen to be running anything by the kernel. Thus, all the
system daemons have files for whatever user they are run as, root has it
when anything is running as root, each human user has theirs when
anything is running as them, etc.
If you read the cpu_share files, you will see ordinary users are set to
1024 by default, while the root user (ID of 0) gets 2048 by default,
double the time of anyone else. This is a quite sane default setup, as
it means that you should always be able to login as root and kill a
process if it's hogging resources.
Now, if you are using portage (I assume the other PMs have something
similar) and are using FEATURES=userpriv, portage will spend most of its
high-CPU compiling time as the user "portage". FEATURES=userfetch and
usersync can be used as well, to have portage use its user for that too.
Of course, it must be root to run the install bit and when actually
qmerging to the live filesystem, but that's a very small part of its
normal work. With the kernel configured as described above, you now have
the kernel enforcing share so it can't hog things.
If you need, you can set your own user to 2048 or possibly 4096, 2-4
times the normal user share, the same or twice what root gets. That
should still give root enough time to do what it needs to do as long as
you only do that for a single user, but I'd be very cautious about doing
it for more than one user or setting > 4096.
Of course, you can also tighten down on hogging users, with one
limitation -- remember the user_id subdir goes away if there's nothing
left running as that user, and of course, over a reboot. Thus, you can't
set it as permanently as you might wish. There was a helper script that
was supposed to detect a userlogin and write a configured share to that
share file automatically, thus making it a basically permanent setting,
but I never could get the thing to work right here. The trigger that was
supposed to cause it to execute automatically never seemed to trigger, so
the script never ran.
Thus, it's often easier to adjust (presumably increasing) the share of
your normal human user when desired, than it is to adjust (presumably
decreasing, for portage) the share of a user that may be coming and
going, as the portage user would be, unless you're running an emerge
--emptytree @system @world (as I'm doing ATM, having just upgraded to gcc
4.4.0) or something.
With those, you should be able to set something sane for portage. Note
that if you have enough memory (I'd recommend 2 gigs at least better 4
gigs), you can point PORTAGE_TMPDIR at a tmpfs as well. That'll help
keep the I/O down while also /dramatically/ decreasing your merge times,
but you do need a decent amount of memory or it might make trying to work
with the system at the same time worse instead of better.
Using all of the above on a quad-core (or dual dual-cores, as mine), even
a portage load average of hundreds shouldn't be a problem, tho the memory
for it might be -- you don't want to go too far into swap. On a dual-
core (or dual-single-cores, as I ran for some time), you'll normally be
fine, but will notice a bit more jerkiness on the mouse and may have
issues with video or visualization, etc. With a single-core, you'll
still notice it to some extent, but it should still be usable, with
patience.
--
Duncan - List replies preferred. NoHTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
