On 2020-06-17 03:14, Albretch Mueller wrote:
HDDs have their internal caching mechanism and I have heard that the
Linux kernel uses RAM very effitiently, but to my understanding RAM
being only 3-4 times faster doesn't make much sense, so I may be doing
or understanding something not entirely right.
does dd actually hit the bare metal drive or is it just reaching the
disks cache
This is what I am consistently getting from my code doing intesive IO
on the RAM drive:
// __ write speed test
#
# time dd if=/dev/zero of="${_RAM_MNT}"/zero bs=4k count=100000
100000+0 Datensätze ein
100000+0 Datensätze aus
409600000 Bytes (410 MB, 391 MiB) kopiert, 0,756943 s, 541 MB/s
real 0m0,760s
user 0m0,048s
sys 0m0,680s
#
// __ read speed test
#
# time dd if="${_RAM_MNT}"/zero of=/dev/null bs=4k count=100000
100000+0 Datensätze ein
100000+0 Datensätze aus
409600000 Bytes (410 MB, 391 MiB) kopiert, 0,36381 s, 1,1 GB/s
real 0m0,368s
user 0m0,016s
sys 0m0,344s
#
// __ my code reports while using a ramdisk:
...
// __ |0| files not found or empty!
// __ |8616768| bytes in |48| files offset processed (weighted and
checked) in |720057| (ms), |11| (bytes/ms)
real 12m0,396s
user 9m12,596s
sys 3m11,524s
For background information on asynchronous input/ output, see:
https://en.wikipedia.org/wiki/Asynchronous_I/O
As others have mentioned:
1. dd(1) with an option of 'bs=4k' will give lower performance that
dd(1) with a larger blocksize. (I believe the Linux kernel uses 128 kiB
blocks for I/O; this value should work well on any Linux computer. But,
I use 'bs=1M' to make calculations easier.)
2. AIUI dd(1) uses asynchronous (buffered) I/O unless told otherwise.
If the total amount of data written fits into RAM, dd(1) with
asynchronous writes will give higher performance that dd(1) with
synchronous writes. If the total amount of data written is much larger
than RAM, the differences are less pronounced.
You seem to be comparing apples and oranges:
1. The dash(1) and bash(1) built-in 'time' reports "accumulated user
and system times for the shell and for processes run from the shell". I
tend to focus on the first value ("wall clock time").
2. dd(1) reports the number of full and partial records in, the number
of full and partial records out, bytes copied, Megabytes copied,
Mibabytes copied, copy time, and throughput.
3. The statistics reported by your code are different values, units,
and/or format. This makes it hard to make comparisons against the dd(1)
statistics. I suggest that you rework your code to collect and output
statistics in the same format as dd(1).
4. It is unclear if your dd(1) invocations are representative of the
I/O performed by your code. I suggest that you match block size, block
count, and I/O mode for both dd(1) and your code.
<snip>
$
_HDD_DIR="/home/lbrtchx/cmllpz/prjx/kd/java/IO/ffst_bytes/logs/ffst_diffs_files"
$
$ time dd if=/dev/zero of="${_HDD_DIR}"/zero bs=4k count=100000
100000+0 Datensätze ein
100000+0 Datensätze aus
409600000 Bytes (410 MB, 391 MiB) kopiert, 1,92086 s, 213 MB/s
real 0m1,924s
user 0m0,064s
sys 0m1,468s
<snip>
$ time dd if="${_HDD_DIR}"/zero of=/dev/null bs=4k count=100000
100000+0 Datensätze ein
100000+0 Datensätze aus
409600000 Bytes (410 MB, 391 MiB) kopiert, 0,521688 s, 785 MB/s
real 0m0,525s
user 0m0,040s
sys 0m0,308s
<snip>
# time dd if=/dev/zero of="${_HDD_DIR}"/zero bs=4k count=100000
100000+0 Datensätze ein
100000+0 Datensätze aus
409600000 Bytes (410 MB, 391 MiB) kopiert, 1,94642 s, 210 MB/s
real 0m1,954s
user 0m0,048s
sys 0m1,484s
#
<snip>
# time dd if="${_HDD_DIR}"/zero of=/dev/null bs=4k count=100000
100000+0 Datensätze ein
100000+0 Datensätze aus
409600000 Bytes (410 MB, 391 MiB) kopiert, 0,735103 s, 557 MB/s
real 0m0,744s
user 0m0,036s
sys 0m0,324s
#
// __ my code reports while using a hdd access:
...
// __ |0| files not found or empty!
// __ |8616768| bytes in |48| files offset processed (weighted and
checked) in |2412518| (ms), |3| (bytes/ms)
real 40m12,727s
user 10m42,576s
sys 8m41,180s
Again, apples and oranges. A minor point -- use one account for
testing, not one user account and the root account.
Improving the efficiency and performance of software is both art and
science. I suggest that you look for a profiling tools for whatever
programming language/ environment you are using (Java?). Such tools can
help you identify bottlenecks, evaluate alternatives, and make informed
decisions.
David
