On 06/11/2013 4:36 pm, Pádraig Brady wrote:
On 06/11/2013 07:20 AM, Joseph D. Wagner wrote:
Currently, when --remove (-u) is specified, shred overwrites the file
name once for each character, so a file name of 0123456789 would be
overridden 10 times. While this may be the most secure, it is also
the
most time consuming as each of the 10 renames has its own fsync.
Also,
renaming may not be as effective on some journaled file systems. This
patch adds the option --wipename (-w) which accepts the options: *
perchar - overwrite file name once per character; same as current. *
once - overwrite file name once total. * none - skip overwrite of
file
name entirely; just unlink. If --remove is specified but not
--wipename, perchar is assumed, preserving current behavior.
Specifying
--wipename implies --remove. In theory, this should provide improved
performance for those who choose it, especially when deleting many
small files. I am currently testing performance on my system, but I
wanted to get the ball rolling by soliciting your comments and your
receptiveness to accepting this patch. Thoughts?
Thanks for the patch.
While on the face of it, the extra control seems beneficial,
I'm not convinced. The main reason is that this gives
some extra credence to per file shreds, which TBH are
not guaranteed due to journalling etc.
I see performance as an important consideration when
shredding large amounts of data like a disk device.
However single file performance is less of a concern.
The normal use case for shred would be for single files,
or at the device level. shredding many files is not the
normal use case to worry about IMHO. If one was worried
about securely shredding many files, it's probably best
to have those files on a separate file system, and shred
that at a lower level.
In any case if you really were OK with just unlinking files
after shredding the data, that can be done in a separate operation:
find | xargs shred
find | xargs rm
So I'm 60:40 against adding this option.
thanks,
Pádraig.
I thought about running two separate operations, as you suggested.
However, my problem with that would be the loss of an atomic
transaction. What if something happens midway through the shred? I
would not know which files were shredded, and I would have to start
over. Worse, if running from a script, it might execute the unlinking
without having completed the shred. While I could create all sorts of
sophisticated code to check these things, it would be a lot easier if I
could simply rely on the mechanisms already built into shred.
I can understand your concern about a tool being misused. If adding a
warning to the usage output would help alleviate your concerns, I would
be happy to draft one and add it to my patch. However, I do not
believe
people should be denied a tool due to its potential misuse. Would you
deny people the use of an iron due to its risk of misuse or injury? My
personal philosophy is to give them the tool with instructions and
warnings. If the user disregards this information, it is not my
problem. In my case, I am using shred to purge information from file
systems that cannot be taken offline. Given the specific file system,
its configuration, and modest sensitivity of the information, the
decision was made that this is an acceptable risk. I believe I should
be able to assume those risks, without being denied optimizations
because they are not considered best practices for the majority of use
cases.
As for the performance improvement itself, the result is measurably
significant. I wrote a script that creates 100,000 files, and then
measures the performance of shredding those files using the different
wipename options in my patch. Exact results and the exact script are
below.
I am hoping these hard numbers and my kind, persuasive arguments will
convince you to change your mind, and accept my patch.
Thank you for your time and consideration.
Joseph D. Wagner
## perchar ##
real678m33.468s
user0m9.450s
sys 3m20.001s
## once ##
real151m54.655s
user0m3.336s
sys 0m32.357s
## none ##
real107m34.307s
user0m2.637s
sys 0m21.825s
perchar: 11 hours 18 minutes 33.468 seconds
once: 2 hours 31 minutes 54.655 seconds
* a 346% improvement over perchar
none: 1 hour 47 minutes 34.307 seconds
* a 530% improvement over perchar
* a 41% improvement over once
Exact script below.
#!/bin/bash
# Generate random files
FILESNEEDED=$((10 - `ls -1 | wc -l`))
for (( i=1; i ${FILESNEEDED}; i++)); do
FILENAME=`tr -dc [:alpha:] /dev/urandom | head -c 16`
truncate -s 4k ${FILENAME}
done
# Wipename perchar
echo ## perchar ##
time find . -type f -print0 | xargs -0 -n16384 ../shred -n1
--wipename=perchar
# Generate random files
FILESNEEDED=$((10 - `ls -1 | wc -l`))
for (( i=1; i ${FILESNEEDED}; i++)); do
FILENAME=`tr -dc [:alpha:] /dev/urandom | head -c 16`
truncate -s 4k ${FILENAME}
done
# Wipename once
echo ## once ##
time find .
