Re: [SOLVED] Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
On 29/07/21 10:28, Dr Rainer Woitok wrote: > I just directly formatted one of my two Philips 128G USB 3.0 sticks with > "mkntfs" and the write performance without VeraCrypt did not improve. > Further searching the web I found that vendors were rarely specifying > the writing speeds of their USB sticks at all, and if they did, there > was a tendency to exaggerate. See for instance Could this be the 512B / 4K sector size problem? If performance is horrible, it could well be a stick / partition mismatch. I never investigated, but if the stick is faking a 512B sector size, so the partition is using 4K blocks aligned on sector 1, it will kill write speed, and in time will kill the stick, too! Cheers, Wol
[SOLVED] Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
Michael and All, Long time ago, on Wednesday, 2021-03-31 12:21:27 +0100, wrote: > ... > OK, unless you made a typo and the "minutes" were meant to say seconds, this > is ridiculously slow. Yes, it really were minutes. > You could run some tests to see what is causing the delay. The veracrypt > algos & cipher iterations, the fuse based ntfs-3g, or the USB stick's > controller. > > However if, as I understand it, all other variables are the same and the only > change was to replace your Verbatim 64G USB 2.0 sticks with Philips 128G USB > 3.0 sticks, then the slow writes point to the Philips devices being the > culprit. I just directly formatted one of my two Philips 128G USB 3.0 sticks with "mkntfs" and the write performance without VeraCrypt did not improve. Further searching the web I found that vendors were rarely specifying the writing speeds of their USB sticks at all, and if they did, there was a tendency to exaggerate. See for instance https://www.lifewire.com/sandisk-extreme-pro-solid-state-flash-drive-review-4689258 Checking what SanDisk USB sticks I could easily obtain locally I finally picked two SanDisk Ultra 128 GB USB 3.0 sticks, and it turned out the writing speed was just amazing, with and without VeraCrypt. So basical- ly, USB 3.0 denotes a standard rather than a (write) speed, and it pays to search the web for benchmark tests before buying. Thanks for all the help and support the participants on this list are providing ... :-) Sincerely, Rainer
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
On Wed, 31 Mar 2021 07:09:03 -0500, Dale wrote: > I wish I could view man pages like I used to in Konqueror. It displays > like a webpage and is much easier to search through. I miss that too. I use mankier.com these days, which gives similar benefits. I have a shortcut set up in chromium so typing "man dd" opens the page in mankier.com. -- Neil Bothwick Life's a cache, and then you flush... pgpXpSNRLU1Pd.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
On Wed, 31 Mar 2021 07:23:27 -0500, Dale wrote: > > Yes, I also liked the old Konqueror interface. Searching for > > keywords e.g. "progress" within man pages works if you preface the > > keyword with "/": > > > > /progress > > > > will find it and "n" or "Shift+n" will jump forward and backward to > > any other instances in the man page. > > That doesn't work here. I can type in /progress but it just shows up at > the bottom. If I try "n" or shift+n I just get a n or N. Maybe my man > page uses something different. It's a feature of whichever pager you use, rather than man itself. Less does it, as does most, but others may differ. -- Neil Bothwick Having children will turn you into your parents. pgpBmNLFYGPYU.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
Michael wrote: > On Wednesday, 31 March 2021 13:23:27 BST Dale wrote: >> Michael wrote: >>> On Wednesday, 31 March 2021 13:09:03 BST Dale wrote: Michael wrote: > On Wednesday, 31 March 2021 12:37:49 BST Dale wrote: >> I checked the man page, I see nothing about the show progress option. > Are you sure? > > This is what I see here on line 47: > > "status=LEVEL > > The LEVEL of information to print to stderr; 'none' suppresses > everything but error messages, 'noxfer' suppresses the final > transfer > statistics, 'progress' shows periodic transfer statistics" > > I find this useful in seeing the transfer speed drop in real time as the > cache gets saturated. Ahhh, I didn't see the status part. It's sort of hiding in a subsection. At least I know now that the version I have installed has this option. I wish I could view man pages like I used to in Konqueror. It displays like a webpage and is much easier to search through. Thanks for pointing that out. Dale :-) :-) >>> Yes, I also liked the old Konqueror interface. Searching for keywords >>> e.g. >>> "progress" within man pages works if you preface the keyword with "/": >>> >>> /progress >>> >>> will find it and "n" or "Shift+n" will jump forward and backward to any >>> other instances in the man page. >> That doesn't work here. I can type in /progress but it just shows up at >> the bottom. > Yes, it shows at the bottom until you hit enter to execute the search. Then > it highlights the next instance of the searched string. Just like Vim/Vi > does. Hmm ... I wonder if I have set up some special environment parameter > on > my systems and forgotten about it. :-/ > > *cough cough* I didn't hit enter. Just did and it worked. Now to get that info to stick in this old dog's brain for next time. ROFL Dale :-) :-)
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
On Wednesday, 31 March 2021 13:23:27 BST Dale wrote: > Michael wrote: > > On Wednesday, 31 March 2021 13:09:03 BST Dale wrote: > >> Michael wrote: > >>> On Wednesday, 31 March 2021 12:37:49 BST Dale wrote: > I checked the man page, I see nothing about the show progress option. > >>> > >>> Are you sure? > >>> > >>> This is what I see here on line 47: > >>> > >>> "status=LEVEL > >>> > >>> The LEVEL of information to print to stderr; 'none' suppresses > >>> everything but error messages, 'noxfer' suppresses the final > >>> transfer > >>> statistics, 'progress' shows periodic transfer statistics" > >>> > >>> I find this useful in seeing the transfer speed drop in real time as the > >>> cache gets saturated. > >> > >> Ahhh, I didn't see the status part. It's sort of hiding in a > >> subsection. At least I know now that the version I have installed has > >> this option. > >> > >> I wish I could view man pages like I used to in Konqueror. It displays > >> like a webpage and is much easier to search through. > >> > >> Thanks for pointing that out. > >> > >> Dale > >> > >> :-) :-) > > > > Yes, I also liked the old Konqueror interface. Searching for keywords > > e.g. > > "progress" within man pages works if you preface the keyword with "/": > > > > /progress > > > > will find it and "n" or "Shift+n" will jump forward and backward to any > > other instances in the man page. > > That doesn't work here. I can type in /progress but it just shows up at > the bottom. Yes, it shows at the bottom until you hit enter to execute the search. Then it highlights the next instance of the searched string. Just like Vim/Vi does. Hmm ... I wonder if I have set up some special environment parameter on my systems and forgotten about it. :-/ signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
Michael wrote: > On Wednesday, 31 March 2021 13:09:03 BST Dale wrote: >> Michael wrote: >>> On Wednesday, 31 March 2021 12:37:49 BST Dale wrote: I checked the man page, I see nothing about the show progress option. >>> Are you sure? >>> >>> This is what I see here on line 47: >>> >>> "status=LEVEL >>> >>> The LEVEL of information to print to stderr; 'none' suppresses >>> everything but error messages, 'noxfer' suppresses the final >>> transfer >>> statistics, 'progress' shows periodic transfer statistics" >>> >>> I find this useful in seeing the transfer speed drop in real time as the >>> cache gets saturated. >> Ahhh, I didn't see the status part. It's sort of hiding in a >> subsection. At least I know now that the version I have installed has >> this option. >> >> I wish I could view man pages like I used to in Konqueror. It displays >> like a webpage and is much easier to search through. >> >> Thanks for pointing that out. >> >> Dale >> >> :-) :-) > Yes, I also liked the old Konqueror interface. Searching for keywords e.g. > "progress" within man pages works if you preface the keyword with "/": > > /progress > > will find it and "n" or "Shift+n" will jump forward and backward to any other > instances in the man page. That doesn't work here. I can type in /progress but it just shows up at the bottom. If I try "n" or shift+n I just get a n or N. Maybe my man page uses something different. That said, I haven't tried Konqueror in a while so I found it, it gives a error but I can type in man:dd and it shows up. It has a search tool. Last time I tried it, wouldn't even come up. That was a while ago tho. Guess it got fixed. Dale :-) :-)
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
On Wednesday, 31 March 2021 13:09:03 BST Dale wrote: > Michael wrote: > > On Wednesday, 31 March 2021 12:37:49 BST Dale wrote: > >> I checked the man page, I see nothing about the show progress option. > > > > Are you sure? > > > > This is what I see here on line 47: > > > > "status=LEVEL > > > > The LEVEL of information to print to stderr; 'none' suppresses > > everything but error messages, 'noxfer' suppresses the final > > transfer > > statistics, 'progress' shows periodic transfer statistics" > > > > I find this useful in seeing the transfer speed drop in real time as the > > cache gets saturated. > > Ahhh, I didn't see the status part. It's sort of hiding in a > subsection. At least I know now that the version I have installed has > this option. > > I wish I could view man pages like I used to in Konqueror. It displays > like a webpage and is much easier to search through. > > Thanks for pointing that out. > > Dale > > :-) :-) Yes, I also liked the old Konqueror interface. Searching for keywords e.g. "progress" within man pages works if you preface the keyword with "/": /progress will find it and "n" or "Shift+n" will jump forward and backward to any other instances in the man page. signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
Michael wrote: > On Wednesday, 31 March 2021 12:37:49 BST Dale wrote: > >> I checked the man page, I see nothing about the show progress option. > Are you sure? > > This is what I see here on line 47: > > "status=LEVEL > The LEVEL of information to print to stderr; 'none' suppresses > everything but error messages, 'noxfer' suppresses the final transfer > statistics, 'progress' shows periodic transfer statistics" > > I find this useful in seeing the transfer speed drop in real time as the > cache > gets saturated. Ahhh, I didn't see the status part. It's sort of hiding in a subsection. At least I know now that the version I have installed has this option. I wish I could view man pages like I used to in Konqueror. It displays like a webpage and is much easier to search through. Thanks for pointing that out. Dale :-) :-)
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
On Wednesday, 31 March 2021 12:37:49 BST Dale wrote: > I checked the man page, I see nothing about the show progress option. Are you sure? This is what I see here on line 47: "status=LEVEL The LEVEL of information to print to stderr; 'none' suppresses everything but error messages, 'noxfer' suppresses the final transfer statistics, 'progress' shows periodic transfer statistics" I find this useful in seeing the transfer speed drop in real time as the cache gets saturated. signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
Michael wrote: > You may want to run some tests on the sticks you have, if only to bottom out > what their performance is on different PCs and USB ports: > > dd if=/dev/zero of=/run/media///TESTFILE bs=512 > count=60 oflag=direct conv=notrunc,fsync status=progress > > Use a large enough file to make sure the USB controller cache gets saturated. > > You could use a ramdisk/tmpfs as an input file. > > If you write directly to the device as Dale suggested it will wipe data, so > keep a backup of anything you need first. I checked the man page, I see nothing about the show progress option. You can bet I'll try that next time tho. I saw a video of someone else using it and it is a lot easier than having to switch Konsoles and type in more commands. Thanks for sharing that option, that isn't in the freaking manual I might add. This is one time where telling someone to read the manual wouldn't work. ROFL Awesome!! Dale :-) :_)
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
On Tuesday, 30 March 2021 18:11:56 BST Dr Rainer Woitok wrote:
> On Saturday, 2020-12-05 19:07:51 +0100, I myself wrote:
>
> ("> >" refers to Michael )
>
> > Michael,
> >
> > On Friday, 2020-11-27 19:07:17 +, you wrote:
> > > ...
> > > A 4k block size is recommended for ntfs-3g which is the default sector
> > > created by fdisk and friends on Linux these days. This will align your
> > > partition optimally. In addition, mkfs.ntfs will use 4096 bytes as the
> > > default cluster size, so you should be good in that respect.
> > >
> > > Another setting you may want to try is mounting the USB with
> > > 'big_writes' -
> > > check the man page. This should help particularly with large files,
> > > which
> > > will use larger blocks up to 128KB when copying data to the NTFS.
> >
> > Both, the VeraCrypt command line (--fs-options=big_writes) and the Vera-
> > Crypt GUI (under "Settings --> Preferences") allow setting this mount
> > option. But
> >
> >$ mount | grep veracrypt
> >
> > never shows it, initially causing me to erroneously believe it wasn't
> > set and to try finding on the web another way of setting it. By pure
> > chance I finally found out that
> >
> >$ ps -ef | grep veracrypt
> >
> > lists a "/usr/sbin/mount.ntfs" task which shows the options really in
> > effect. However, I haven't yet had the time to test the effect of this
> > option when writing plenty of really big files. I will report on that
> > later.
>
> Well, it's been quite a while, due to my being almost permanently con-
> fronted with more pressing tasks ... :-(
>
> To sum up my experience with my new 128 GB Philips USB 3.0 sticks: while
> the Philips sticks are significantly faster for reading operations than
> my old 64 GB Verbatim ones (probably USB 2.0), writing operations to the
> Philips sticks are unbearably slow, regardless of whether I created a
> normal unencrypted NTFS filesystem on them or an encrypted NTFS filesys-
> tem using VeraCrypt. Writing to the USB stick while at the same time
> reading from it in a different terminal window caused commands like "cd"
> or "ls" to simply stall. Thus while running
>
>$ cp --preserve=timestamps -ru $source_dir .
>
> in one terminal window, I ran
>
>$ while true
>
>> do n=$(ps -ef|g 'cp --preserve'|g -v grep)
>>
>>if [[ "$n" = "${o-}" ]]
>>then sleep 10
>>else o="$n"
>>
>> echo "$n"
>>
>>fi
>>
>> done
>
> in another, to get the wall clock times when copying a new file began.
> That way I found that copying a 30 MB file took about 40 minutes.
OK, unless you made a typo and the "minutes" were meant to say seconds, this
is ridiculously slow.
You could run some tests to see what is causing the delay. The veracrypt
algos & cipher iterations, the fuse based ntfs-3g, or the USB stick's
controller.
However if, as I understand it, all other variables are the same and the only
change was to replace your Verbatim 64G USB 2.0 sticks with Philips 128G USB
3.0 sticks, then the slow writes point to the Philips devices being the
culprit.
Some years ago I tested some USB 2.0 sticks of various sizes, from 256M up to
32G and recall the smaller the USB stick the faster the write performance, so
differences in writing speed are normal. The writing speed you're describing
however is a clear indication of something being wrong.
> So what are my options?
>
>- Stay away from Philips USB 3.0 sticks?
>
>- Stay away from Philips USB sticks in general?
Without knowing the internals, a brand may offer only an unwarranted
assumption of performance. We saw Western Digital disks being sold as CMR,
while having SMR internals. A brand could switch OEM suppliers, or
components, making benchmarking unreliable.
>- Stay away from USB 3.0 sticks in general?
USB 3.0 is faster and USB 3.2 when available will be even faster. So use
whatever the USB ports on your PC offer.
>- Stay away from Filesystem in User Space using a non-stable 5.10 or
> 5.11 kernel (currently I'm using stable 5.4.97)?
>
>- Stay away from Gentoo?
>
>- Stay away from Linux in general and go back to OTOS (aka the Only
> True Operating System aka Windoze)?
>
>- ...?
In-kernel fs drivers are measurably faster than fuse based fs for well
understood reasons. However, if needs must and the fs you require is not
available on Linux, then some compromise will be required.
> Any ideas and comments welcome ...
>
> Sincerely,
> Rainer
You may want to run some tests on the sticks you have, if only to bottom out
what their performance is on different PCs and USB ports:
dd if=/dev/zero of=/run/media///TESTFILE bs=512
count=60 oflag=direct conv=notrunc,fsync status=progress
Use a large enough file to make sure the USB controller cache gets saturated.
You could use a ramdisk/tmpfs as an input file.
If you write directly to the devic
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
On March 30, 2021 10:11:56 AM PDT, Dr Rainer Woitok
wrote:
>On Saturday, 2020-12-05 19:07:51 +0100, I myself wrote:
>
>("> >" refers to Michael )
>
>> Michael,
>>
>> On Friday, 2020-11-27 19:07:17 +, you wrote:
>>
>> > ...
>> > A 4k block size is recommended for ntfs-3g which is the default
>sector created
>> > by fdisk and friends on Linux these days. This will align your
>partition
>> > optimally. In addition, mkfs.ntfs will use 4096 bytes as the
>default cluster
>> > size, so you should be good in that respect.
>> >
>> > Another setting you may want to try is mounting the USB with
>'big_writes' -
>> > check the man page. This should help particularly with large
>files, which
>> > will use larger blocks up to 128KB when copying data to the NTFS.
>>
>> Both, the VeraCrypt command line (--fs-options=big_writes) and the
>Vera-
>> Crypt GUI (under "Settings --> Preferences") allow setting this
>mount
>> option. But
>>
>>$ mount | grep veracrypt
>>
>> never shows it, initially causing me to erroneously believe it
>wasn't
>> set and to try finding on the web another way of setting it. By
>pure
>> chance I finally found out that
>>
>>$ ps -ef | grep veracrypt
>>
>> lists a "/usr/sbin/mount.ntfs" task which shows the options really
>in
>> effect. However, I haven't yet had the time to test the effect of
>this
>> option when writing plenty of really big files. I will report on
>that
>> later.
>
>Well, it's been quite a while, due to my being almost permanently
>con-
>fronted with more pressing tasks ... :-(
>
>To sum up my experience with my new 128 GB Philips USB 3.0 sticks:
>while
>the Philips sticks are significantly faster for reading operations
>than
>my old 64 GB Verbatim ones (probably USB 2.0), writing operations to
>the
>Philips sticks are unbearably slow, regardless of whether I created
>a
>normal unencrypted NTFS filesystem on them or an encrypted NTFS
>filesys-
>tem using VeraCrypt. Writing to the USB stick while at the same
>time
>reading from it in a different terminal window caused commands like
>"cd"
>or "ls" to simply stall. Thus while running
>
> $ cp --preserve=timestamps -ru $source_dir .
>
>in one terminal window, I ran
>
> $ while true
> > do n=$(ps -ef|g 'cp --preserve'|g -v grep)
> >if [[ "$n" = "${o-}" ]]
> >then sleep 10
> >else o="$n"
> > echo "$n"
> >fi
> > done
>
>in another, to get the wall clock times when copying a new file
>began.
>That way I found that copying a 30 MB file took about 40 minutes.
>
>So what are my options?
>
> - Stay away from Philips USB 3.0 sticks?
>
> - Stay away from Philips USB sticks in general?
>
> - Stay away from USB 3.0 sticks in general?
>
> - Stay away from Filesystem in User Space using a non-stable 5.10 or
> 5.11 kernel (currently I'm using stable 5.4.97)?
>
> - Stay away from Gentoo?
>
> - Stay away from Linux in general and go back to OTOS (aka the Only
> True Operating System aka Windoze)?
>
> - ...?
>
>Any ideas and comments welcome ...
>
>Sincerely,
> Rainer
There are a number of things which might be going on here.
To start with, you can get the kernel, user, and wall clock run times for
commands by prefixing it with "time". So:
time cp
Will get you more precise answers with much less effort.
As for the performance of the USB drive in question, there are a few things
that might be tripping it up.
Firstly, writing flash memory is significantly slower than reading it. Some
drives deal with this by having some kind of internal cache mechanism. Many
deal with it by using a pile of smaller chips instead of one big one and
striping the writes. If the Phillips drive just used a few large chips
instead, then it's just slow to write to and there isn't much you can do about
it. I've seen a lot of cheaper drives that are that way.
Double check that the alignment and block size are correct for the drive's
internal structure. That can cause some pretty massive performance hits if
it's incorrect.
You can also check the output of the dmesg command for any errors the system is
encountering with regard to the drive.
I don't know of any reason to stay away from usb 3.0 on Linux, but if you have
USB 3 devices on both ends and try to hook them together with a USB 2.0 or 1.1
rated cable that could easily cause some problems... I assume you're plugging
the drive straight into the machine's socket. If you're using the front panel
though try one of the ones on the back. There may be something up with the
case wiring.
I've never had a Phillips USB stick, so maybe do some tests with another brand
of stick and see if it has the same problem. Kingston or SanDisk or something.
One of the ones where memory is their primary focus.
You could definitely check performance on a different OS. There may be
driver-related performance issues on this model of drive or even this specific
drive.
Instead of NTFS
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
Dr Rainer Woitok wrote:
>
> Well, it's been quite a while, due to my being almost permanently con-
> fronted with more pressing tasks ... :-(
>
> To sum up my experience with my new 128 GB Philips USB 3.0 sticks: while
> the Philips sticks are significantly faster for reading operations than
> my old 64 GB Verbatim ones (probably USB 2.0), writing operations to the
> Philips sticks are unbearably slow, regardless of whether I created a
> normal unencrypted NTFS filesystem on them or an encrypted NTFS filesys-
> tem using VeraCrypt. Writing to the USB stick while at the same time
> reading from it in a different terminal window caused commands like "cd"
> or "ls" to simply stall. Thus while running
>
>$ cp --preserve=timestamps -ru $source_dir .
>
> in one terminal window, I ran
>
>$ while true
>> do n=$(ps -ef|g 'cp --preserve'|g -v grep)
>>if [[ "$n" = "${o-}" ]]
>>then sleep 10
>>else o="$n"
>> echo "$n"
>>fi
>> done
>
> in another, to get the wall clock times when copying a new file began.
> That way I found that copying a 30 MB file took about 40 minutes.
>
> So what are my options?
>
>- Stay away from Philips USB 3.0 sticks?
>
>- Stay away from Philips USB sticks in general?
>
>- Stay away from USB 3.0 sticks in general?
>
>- Stay away from Filesystem in User Space using a non-stable 5.10 or
> 5.11 kernel (currently I'm using stable 5.4.97)?
>
>- Stay away from Gentoo?
>
>- Stay away from Linux in general and go back to OTOS (aka the Only
> True Operating System aka Windoze)?
>
>- ...?
>
> Any ideas and comments welcome ...
>
> Sincerely,
> Rainer
>
>
Have you tried using dd to test the speed? It doesn't even need a file
system as it writes directly to the device. I've done this in the past
and it tells if it is a file system issue or a hardware issue. One
thing here, it will destroy ANY and ALL data on it if you let it run
until it finishes. Make sure you have nothing you want to save on it
when doing this. I might add, I like this over rsync --progress because
it doesn't have a file system in the middle.
Commands I use to test this.
dd if=/dev/zero of=/dev/sd bs=4k conv=notrunc oflag=direct #disables cache
Don't forget to complete the disk device in the "of=" section and you
can leave off the comment as well when typing it in. You may also want
to adjust the other settings or leave out some to test things. See if
you can figure out a way to tweek it a bit. Sometimes you can add those
to the mount options to make it work better, so I've read anyway.
To monitor the speed, I found this years ago and it still worked a few
months ago when I was testing a stick that was giving me problems.
Turned out, the stick was dying a slow death. The dd test finished it,
with errors.
watch -n 10 kill -USR1
I use Konsole here which has tabs but you may use something similar that
will work just as well. I start the dd command in one tab, then go to
other tab and find the process number for dd there with ps and grep. I
then use the above and replace "" with the number, leave
out the <> as well, just the number itself. When you go to the tab
where dd is running, it updates about every ten seconds with speed and
other info. I've found it normal to start out fast and then slow down
as it goes. Since it has no moving parts, no clue why it does that.
Someone else may see this and have a much better method but that has
worked for me in the past. As we know, some USB stick and other memory
type cards can get fussy or go bad.
Hope that helps or at least gives some ideas.
Dale
:-) :-)
P. S. I just copy what folks post, I don't claim to understand the
inner workings of this. LOL I do know, it destroys data tho. Always
be careful with dd and if and 0.
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
On Saturday, 2020-12-05 19:07:51 +0100, I myself wrote:
("> >" refers to Michael )
> Michael,
>
> On Friday, 2020-11-27 19:07:17 +, you wrote:
>
> > ...
> > A 4k block size is recommended for ntfs-3g which is the default sector
> > created
> > by fdisk and friends on Linux these days. This will align your partition
> > optimally. In addition, mkfs.ntfs will use 4096 bytes as the default
> > cluster
> > size, so you should be good in that respect.
> >
> > Another setting you may want to try is mounting the USB with 'big_writes' -
> > check the man page. This should help particularly with large files, which
> > will use larger blocks up to 128KB when copying data to the NTFS.
>
> Both, the VeraCrypt command line (--fs-options=big_writes) and the Vera-
> Crypt GUI (under "Settings --> Preferences") allow setting this mount
> option. But
>
>$ mount | grep veracrypt
>
> never shows it, initially causing me to erroneously believe it wasn't
> set and to try finding on the web another way of setting it. By pure
> chance I finally found out that
>
>$ ps -ef | grep veracrypt
>
> lists a "/usr/sbin/mount.ntfs" task which shows the options really in
> effect. However, I haven't yet had the time to test the effect of this
> option when writing plenty of really big files. I will report on that
> later.
Well, it's been quite a while, due to my being almost permanently con-
fronted with more pressing tasks ... :-(
To sum up my experience with my new 128 GB Philips USB 3.0 sticks: while
the Philips sticks are significantly faster for reading operations than
my old 64 GB Verbatim ones (probably USB 2.0), writing operations to the
Philips sticks are unbearably slow, regardless of whether I created a
normal unencrypted NTFS filesystem on them or an encrypted NTFS filesys-
tem using VeraCrypt. Writing to the USB stick while at the same time
reading from it in a different terminal window caused commands like "cd"
or "ls" to simply stall. Thus while running
$ cp --preserve=timestamps -ru $source_dir .
in one terminal window, I ran
$ while true
> do n=$(ps -ef|g 'cp --preserve'|g -v grep)
>if [[ "$n" = "${o-}" ]]
>then sleep 10
>else o="$n"
> echo "$n"
>fi
> done
in another, to get the wall clock times when copying a new file began.
That way I found that copying a 30 MB file took about 40 minutes.
So what are my options?
- Stay away from Philips USB 3.0 sticks?
- Stay away from Philips USB sticks in general?
- Stay away from USB 3.0 sticks in general?
- Stay away from Filesystem in User Space using a non-stable 5.10 or
5.11 kernel (currently I'm using stable 5.4.97)?
- Stay away from Gentoo?
- Stay away from Linux in general and go back to OTOS (aka the Only
True Operating System aka Windoze)?
- ...?
Any ideas and comments welcome ...
Sincerely,
Rainer
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
Michael, On Friday, 2020-11-27 19:07:17 +, you wrote: > ... > A 4k block size is recommended for ntfs-3g which is the default sector > created > by fdisk and friends on Linux these days. This will align your partition > optimally. In addition, mkfs.ntfs will use 4096 bytes as the default cluster > size, so you should be good in that respect. > > Another setting you may want to try is mounting the USB with 'big_writes' - > check the man page. This should help particularly with large files, which > will use larger blocks up to 128KB when copying data to the NTFS. Both, the VeraCrypt command line (--fs-options=big_writes) and the Vera- Crypt GUI (under "Settings --> Preferences") allow setting this mount option. But $ mount | grep veracrypt never shows it, initially causing me to erroneously believe it wasn't set and to try finding on the web another way of setting it. By pure chance I finally found out that $ ps -ef | grep veracrypt lists a "/usr/sbin/mount.ntfs" task which shows the options really in effect. However, I haven't yet had the time to test the effect of this option when writing plenty of really big files. I will report on that later. Sincerely, Rainer
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
Hi Rainer, On Friday, 27 November 2020 16:01:29 GMT Dr Rainer Woitok wrote: > Since the USB sticks contain symbolic links and have to be accessible > from both, Linux and Windows they are NTFS formatted, and according to > "mkntfs(8)" the sector size can be at most 4096, while the cluster size > is limited to 2097152, that is 2G. However, when NTFS formatting an USB > stick from within TrueCrypt/VeraCrypt or directly in Windows the maximum > cluster size is 64K, with the only difference that Windows calls it > "allocation unit size". Ohh! STOP RIGHT THERE! :-) I mistakenly thought you were using FAT. NTFS on linux uses the ntfs-3g driver, which relies on FUSE. This 'Filesystem in Userspace' is inevitably slower than kernel filesystem drivers, because it has to jump through hoops and libs, acting as a virtual filesystem. CPU usage will also be higher as a result, than when using a native kernel filesystem driver. A 4k block size is recommended for ntfs-3g which is the default sector created by fdisk and friends on Linux these days. This will align your partition optimally. In addition, mkfs.ntfs will use 4096 bytes as the default cluster size, so you should be good in that respect. Another setting you may want to try is mounting the USB with 'big_writes' - check the man page. This should help particularly with large files, which will use larger blocks up to 128KB when copying data to the NTFS. Also, read the FAQs under the heading "Performance" for more useful information: https://www.tuxera.com/community/ntfs-3g-faq/ Hope this helps. signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
Michael, On Thursday, 2020-11-26 00:10:00 +, you wrote: > ... > Check dmesg to see if initialisation of the USB 3.0 drive throws up any > errors. No errors. > Then check 'lsusb -t' to make sure it has been recognised as a USB > 3.0. "lsusb -tv" showed the stick to be USB 3.0. > ... >Partitioning the USB drive to use 128KB sectors > and > then aligning the fs on it should improve matters. Since the USB sticks contain symbolic links and have to be accessible from both, Linux and Windows they are NTFS formatted, and according to "mkntfs(8)" the sector size can be at most 4096, while the cluster size is limited to 2097152, that is 2G. However, when NTFS formatting an USB stick from within TrueCrypt/VeraCrypt or directly in Windows the maximum cluster size is 64K, with the only difference that Windows calls it "allocation unit size". So I think above you were talking about 128K clusters rather than sect- ors. I'll give that a try and will reformat the USB sticks using the maximum cluster size of 64K. But I don't see a way to "align" the file system on these USB sticks. > I found this article which mentions an experiment with ext4 fs. Thanks for the link you sent in your other mail and thanks for pointing all this out :-) Sincerely, Rainer
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
On Thursday, 26 November 2020 00:10:00 GMT Michael wrote: > On Wednesday, 25 November 2020 17:37:15 GMT Dr Rainer Woitok wrote: > > Greetings, > > > > since my old 64 GB Verbatim USB sticks became too small, I bought two > > new 128 GB Philips sticks. Because I need to read and write them on > > both, a stand-alone Windows laptop (not connected to the internet) runn- > > ing Windows Vista and Cygwin and my Gentoo laptop, I encrypted them with > > old TrueCrypt on the Windows box, using them under Gentoo in TrueCrypt > > compatibility mode. > > > > This worked well with the Verbatim USB sticks (which probably are USB > > 2.0), but while reading the new USB 3.0 Philips USB sticks is signific- > > antly faster than reading the old Verbatim USB sticks, writing to them > > is slow as hell under Gentoo. And writing to the Philips USB sticks on > > the old Vista laptop with USB 2.0 ports clearly outperforms writing to > > them using the Gentoo laptop's USB 3.0 ports. > > > > This could be a problem with TrueCrypt/VeraCrypt or with somehow miscon- > > figured USB ports. To check for the latter I provide below all kernel > > configuration variables I regard USB related in the hope that some know- > > > ledgable people might find a glitch in there: > Check dmesg to see if initialisation of the USB 3.0 drive throws up any > errors. Then check 'lsusb -t' to make sure it has been recognised as a USB > 3.0. > > If write operations without TrueCrypt/VeraCrypt are equally slow, then > obviously the problem is not with encryption. > > I've read in a number of articles the erase block size on most USB flash > (NAND) is 128KB, which incurs a lot of operations on a write, when using > Linux with its 4K size sectors. Partitioning the USB drive to use 128KB > sectors and then aligning the fs on it should improve matters. > > I found this article which mentions an experiment with ext4 fs. A more > effective search should hopefully bring up examples on FAT fs. > > HTH. Apologies, I seem to have forgotten to include the link. Here's another link I came across today and which offers more detail on this topic: http://wiki.laptop.org/go/How_to_Damage_a_FLASH_Storage_Device signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?
On Wednesday, 25 November 2020 17:37:15 GMT Dr Rainer Woitok wrote: > Greetings, > > since my old 64 GB Verbatim USB sticks became too small, I bought two > new 128 GB Philips sticks. Because I need to read and write them on > both, a stand-alone Windows laptop (not connected to the internet) runn- > ing Windows Vista and Cygwin and my Gentoo laptop, I encrypted them with > old TrueCrypt on the Windows box, using them under Gentoo in TrueCrypt > compatibility mode. > > This worked well with the Verbatim USB sticks (which probably are USB > 2.0), but while reading the new USB 3.0 Philips USB sticks is signific- > antly faster than reading the old Verbatim USB sticks, writing to them > is slow as hell under Gentoo. And writing to the Philips USB sticks on > the old Vista laptop with USB 2.0 ports clearly outperforms writing to > them using the Gentoo laptop's USB 3.0 ports. > > This could be a problem with TrueCrypt/VeraCrypt or with somehow miscon- > figured USB ports. To check for the latter I provide below all kernel > configuration variables I regard USB related in the hope that some know- > ledgable people might find a glitch in there: Check dmesg to see if initialisation of the USB 3.0 drive throws up any errors. Then check 'lsusb -t' to make sure it has been recognised as a USB 3.0. If write operations without TrueCrypt/VeraCrypt are equally slow, then obviously the problem is not with encryption. I've read in a number of articles the erase block size on most USB flash (NAND) is 128KB, which incurs a lot of operations on a write, when using Linux with its 4K size sectors. Partitioning the USB drive to use 128KB sectors and then aligning the fs on it should improve matters. I found this article which mentions an experiment with ext4 fs. A more effective search should hopefully bring up examples on FAT fs. HTH. signature.asc Description: This is a digitally signed message part.
