Antti Harri wrote:
On Sun, 16 Jul 2006, Nick Holland wrote:
nope, you can still likely use multiple partitions. Break your backup
job into smaller chunks, put each chunk on its own partition. Or put
each machine on its own partition. Or ...
Interesting ideas. I didn't think that having the same amount of files
in many partitions will reduce the total time to fsck, does it really
work that way although it goes through the same amount of files?
I haven't fully benchmarked it, but my inclination is that several small
is faster than one big. If scattered across multiple disks, definitely
(they can be done in parallel...assuming you have the RAM!), for one
disk, the partitions are done sequentially, not concurrently.
Further, if you keep only the partitions mounted that you are using AT
THAT MOMENT, only those partitions will need an fsck should an event happen.
BTW: Yes, the dmesg could very well have helped. If your disks were
not being handled properly or you had insufficient RAM, you can have
HORRIBLE problems with fsck performance, adding to your fsck time by a
non-trivial multiple. Your times sound excessive to me, but then, I
don't think I have that many files on a single partition.
Unfortunately the computer isn't at hand right now. I'll check the
amount of RAM and add some more if there isn't much. Would changing
BUFCACHEPCT
help too? Because the computer is dedicated backup server so it can
take up all the memory as far as I'm concerned.
BUFCACHEPCT may help...assuming you don't force the machine to start
swapping! Easy thing to benchmark...
Guideline is 1M RAM for 1G of disk to keep fsck from swapping. This
seems to be a decent rule. On the other hand, IF you have huge
quantities of tiny files, this may skew things in worse directions...or
may not. I'm not a file system person, so everything I say could be
wrong. :)
One idea which has been suggested is to use softupdates, and simply
"force" mounting of the volume at boot, and periodically, fsck the
thing on your schedule, to reclaim lost disk space. Yes, when you do
run the fsck, you will spend a lot of time waiting for it, but you
will be able to schedule it.
Hmm, actually I am using softupdates. Doesn't it *ever* get corrupted with
softupdates even though there is a crash?
Theory or reality?? :)
Supposedly, "... the disk image is always kept consistent."
http://www.mckusick.com/softdep/
Reality: as your disk probably has some write cache, don't bet on
that... things wont be written to the disk in the order expected. On the
other hand...look at what fsck does: it just goes through the disk and
tries to make sense of the file system...it does NOT magically fix all
your data that was being written to disk when the power went out.
So...if you trip over the power cord or have a not-so-bright squirrel
take out your power mid-write, data will be corrupted, and all fsck will
do is make sure totally unrecognizable data is not taking up space on
the disk.
So...does it get corrupted? Yes. Does running fsck fix that? Not
really. fsck just hides some of the evidence. :)
Keep in mind, partitions need not all be mounted in /etc/fstab, they
can be manually mounted "later" in rc.local. Why does your backup
machine have to boot "fast"? (I got one with way too little RAM, it
needs to use swap to fsck, but that's ok...I'm not in a hurry for this
machine to come up). Doing something else with it? Ok, just put the
backup partition as noauto in /etc/fstab, and fsck and mount (or just
force-mount) the partition in /etc/rc.local. Now, whatever it was
that was bothering you about booting so slowly is up quickly, and the
backup partition will get mounted in due time.
Well, I have it set up so that it comes up once a day and after it
finishes doing backups it shuts down itself. So if it crashes and starting
up takes too much time the backup job won't fit the window it's supposed
to. I'm still working on the server and trying to find the best
solution for my needs. Luckily there hasn't been much use for the
backups since there hasn't been any real accidents or failures either ;-)
I presume you are turning the machine off for reducing power
consumption? How about this: turn off at end of job, turn on at
beginning of job, and set to auto-power-on at application of power?
That way, if the power is interrupted, it will come up and fsck as soon
as the power comes back on, so it will be ready for the backup window.
If the machine crashes (which it really shouldn't, of course), you are
SOL, but that might motivate you to fix whatever caused the crash (and
you are in the same position with the crash, regardless, as it won't
power itself down, unless you are killing the power externally).
Nick.
