Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-18 Thread Michael Stone
On Thu, Jun 18, 2020 at 05:28:11PM +0300, Reco wrote: Hi. On Thu, Jun 18, 2020 at 08:57:48AM -0400, Michael Stone wrote: On Thu, Jun 18, 2020 at 08:50:49AM +0300, Reco wrote: > On Wed, Jun 17, 2020 at 05:54:51PM -0400, Michael Stone wrote: > > On Wed, Jun 17, 2020 at 11:45:53PM +0300,

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-18 Thread Reco
Hi. On Thu, Jun 18, 2020 at 08:57:48AM -0400, Michael Stone wrote: > On Thu, Jun 18, 2020 at 08:50:49AM +0300, Reco wrote: > > On Wed, Jun 17, 2020 at 05:54:51PM -0400, Michael Stone wrote: > > > On Wed, Jun 17, 2020 at 11:45:53PM +0300, Reco wrote: > > > > Long story short, if you need a

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-18 Thread Michael Stone
On Thu, Jun 18, 2020 at 08:50:49AM +0300, Reco wrote: Hi. On Wed, Jun 17, 2020 at 05:54:51PM -0400, Michael Stone wrote: On Wed, Jun 17, 2020 at 11:45:53PM +0300, Reco wrote: > Long story short, if you need a primitive I/O benchmark, you're better > with both dsync and nocache. Not

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Reco
Hi. On Wed, Jun 17, 2020 at 05:54:51PM -0400, Michael Stone wrote: > On Wed, Jun 17, 2020 at 11:45:53PM +0300, Reco wrote: > > Long story short, if you need a primitive I/O benchmark, you're better > > with both dsync and nocache. > > Not unless that's your actual workload, IMO. Almost

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Andy Smith
Hello, On Wed, Jun 17, 2020 at 12:17:58PM +0200, Albretch Mueller wrote: > also, if in order to use RAID 10 you need 4 drives Linux mdadm can do RAID-10 with 2 or more devices (also doesn't have to be an even number). > (but the dollar per Gb is approaching $0.02) and you get 1.5 > faster

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Michael Stone
On Wed, Jun 17, 2020 at 11:45:53PM +0300, Reco wrote: Long story short, if you need a primitive I/O benchmark, you're better with both dsync and nocache. Not unless that's your actual workload, IMO. Almost nothing does sync i/o; simply using conv=fdatasync to make sure that the cache is

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Reco
On Wed, Jun 17, 2020 at 11:02:14PM +0200, to...@tuxteam.de wrote: > On Wed, Jun 17, 2020 at 11:45:53PM +0300, Reco wrote: > > [...] > > > Long story short, if you need a primitive I/O benchmark, you're better > > with both dsync and nocache. > > Thanks for actually looking over dd's shoulder

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Reco
On Wed, Jun 17, 2020 at 01:23:41PM -0700, David Christensen wrote: > On 2020-06-17 12:26, Reco wrote: > > > On Wed, Jun 17, 2020 at 12:10:51PM -0700, David Christensen wrote: > > > 2. AIUI dd(1) uses asynchronous (buffered) I/O unless told otherwise. > > > > You seem to confuse asynchronous and

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread tomas
On Wed, Jun 17, 2020 at 11:45:53PM +0300, Reco wrote: [...] > Long story short, if you need a primitive I/O benchmark, you're better > with both dsync and nocache. Thanks for actually looking over dd's shoulder :-) Cheers -- t signature.asc Description: Digital signature

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Reco
Hi. On Wed, Jun 17, 2020 at 10:33:51PM +0200, to...@tuxteam.de wrote: > So to test disk write speed, 'dsync' seems the way to go. When dumping > to a device, there are no metadata (am I right there?), so probably > again you want 'dsync'. > > I don't know what 'nocache' would do for

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread tomas
On Wed, Jun 17, 2020 at 01:23:41PM -0700, David Christensen wrote: [...] > I was referring to the 'fdatasync', 'fsync', 'dsync', 'sync', and > 'nocache' options to dd(1). Given the terse manual page, and a > unwillingness to crawl the dd(1) and/or kernel code, I can only > guess at my

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread David Christensen
On 2020-06-17 12:26, Reco wrote: On Wed, Jun 17, 2020 at 12:10:51PM -0700, David Christensen wrote: 2. AIUI dd(1) uses asynchronous (buffered) I/O unless told otherwise. You seem to confuse asynchronous and cached I/O too. From Linux kernel POV, *asynchronous* I/O is a pair of

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Reco
Hi. On Wed, Jun 17, 2020 at 12:10:51PM -0700, David Christensen wrote: > 2. AIUI dd(1) uses asynchronous (buffered) I/O unless told otherwise. You seem to confuse asynchronous and cached I/O too. >From Linux kernel POV, *asynchronous* I/O is a pair of io_submit/io_getevents syscalls,

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread David Christensen
On 2020-06-17 03:14, Albretch Mueller wrote: HDDs have their internal caching mechanism and I have heard that the Linux kernel uses RAM very effitiently, but to my understanding RAM being only 3-4 times faster doesn't make much sense, so I may be doing or understanding something not entirely

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Albretch Mueller
> Your test dataset is too small and you aren't flushing the cache before > exiting dd, so you are largely seeing the time it takes to write to cache, > not to disk. > But that gives the RAID10 system 220 IOPs, still nowhere near the 100,000 > IOPs of a single SSD. > I suggest that you google a

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Anders Andersson
On Wed, Jun 17, 2020 at 12:15 PM Albretch Mueller wrote: > > HDDs have their internal caching mechanism and I have heard that the > Linux kernel uses RAM very effitiently, but to my understanding RAM > being only 3-4 times faster doesn't make much sense, so I may be doing > or understanding

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Dan Ritter
Albretch Mueller wrote: > also, if in order to use RAID 10 you need 4 drives (but the dollar > per Gb is approaching $0.02) and you get 1.5 faster performance, what > is the economy of "bying more RAM" if it is so much more expensive? > > Any comparison on HDD, SSD and RAM including pros and

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Dan Ritter
Albretch Mueller wrote: > HDDs have their internal caching mechanism and I have heard that the > Linux kernel uses RAM very effitiently, but to my understanding RAM > being only 3-4 times faster doesn't make much sense, so I may be doing > or understanding something not entirely right. > >

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Michael Stone
On Wed, Jun 17, 2020 at 12:14:55PM +0200, Albretch Mueller wrote: HDDs have their internal caching mechanism and I have heard that the Linux kernel uses RAM very effitiently, but to my understanding RAM being only 3-4 times faster doesn't make much sense, so I may be doing or understanding

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Linux-Fan
Albretch Mueller writes: [...] does dd actually hit the bare metal drive or is it just reaching the disks cache This is what I am consistently getting from my code doing intesive IO on the RAM drive: // __ write speed test # time dd if=/dev/zero of="${_RAM_MNT}"/zero bs=4k count=10

Re: Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Albretch Mueller
also, if in order to use RAID 10 you need 4 drives (but the dollar per Gb is approaching $0.02) and you get 1.5 faster performance, what is the economy of "bying more RAM" if it is so much more expensive? Any comparison on HDD, SSD and RAM including pros and cons which is worth reading?

Could RAM possibly be just 3-4 times faster than bare hdd writes and reads? or, is the Linux kernel doing its 'magic' in the bg? or, ...

2020-06-17 Thread Albretch Mueller
HDDs have their internal caching mechanism and I have heard that the Linux kernel uses RAM very effitiently, but to my understanding RAM being only 3-4 times faster doesn't make much sense, so I may be doing or understanding something not entirely right. does dd actually hit the bare metal