Re: Writing large build logs to NFS extremely slow?

[Rick Macklem]

Zaphod Beeblebrox wrote:
> Is the NFS mounted filesystem NFS?
>  I've found NFS mounted ZFS has several
> pathologies like this when there is no SSD cache and/or log vdevs attached.
Just fyi, I've recently committed a change to main that changes the NFS
client so that it does append writes directly to the NFS server.
This improved the append write performance by 10x for UFS, but only
improved 30% for ZFS without any ZIL log and sync=enabled.
Append writes over NFS is still MUCH slower than a local file system.
(Partially because it has to do every write to the serve FileSync, which means
 the server must commit the changes to stable storage before replying to the
 RPC. It also must do a Getattr RPC against the server before every write,
 to get the up-to-date file size.)

This change won't by MFC'd for a couple of months and is somewhat broken
in main at this time. I am working with kib@ to fix it.

rick

On Wed, Oct 6, 2021 at 10:18 PM Felix Palmen  wrote:

> Hi all,
>
> I use a -CURRENT bhyve vm for testing port builds with poudriere. As
> this vm is only running when needed, but I want to always have access to
> the build logs, I use NFS to mount /usr/local/poudriere/data/logs from
> the host.
>
> I noticed some few ports take ridiculously long to build while barely
> using any CPU time at all. On a closer look, that's all ports producing
> a lot of compiler (warning) output, e.g. gcc, gnutls, gtk2, …
>
> So I assume appending to a large file via NFS gets slower and slower. Is
> there any mount option I could try to fix this? Right now I only have
> `nolockd`, I also tried `noncontigwr` which didn't change anything.
>
> Thinking about alternatives to NFS, are there any news for client-side
> 9p virtfs? I found  which
> still builds with a few minor adaptions, but trying to mount a 9p share
> freezes the machine.
>
> Would you suggest a different mailing list to ask?
>
> BR, Felix
>
> --
>  Dipl.-Inform. Felix Palmen ,.//..
>  {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
>  {pgp public key} http://palmen-it.de/pub.txt   //   """
>  {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A
>

Re: Writing large build logs to NFS extremely slow?

[Zaphod Beeblebrox]

Is the NFS mounted filesystem NFS?  I've found NFS mounted ZFS has several
pathologies like this when there is no SSD cache and/or log vdevs attached.

On Wed, Oct 6, 2021 at 10:18 PM Felix Palmen  wrote:

> Hi all,
>
> I use a -CURRENT bhyve vm for testing port builds with poudriere. As
> this vm is only running when needed, but I want to always have access to
> the build logs, I use NFS to mount /usr/local/poudriere/data/logs from
> the host.
>
> I noticed some few ports take ridiculously long to build while barely
> using any CPU time at all. On a closer look, that's all ports producing
> a lot of compiler (warning) output, e.g. gcc, gnutls, gtk2, …
>
> So I assume appending to a large file via NFS gets slower and slower. Is
> there any mount option I could try to fix this? Right now I only have
> `nolockd`, I also tried `noncontigwr` which didn't change anything.
>
> Thinking about alternatives to NFS, are there any news for client-side
> 9p virtfs? I found  which
> still builds with a few minor adaptions, but trying to mount a 9p share
> freezes the machine.
>
> Would you suggest a different mailing list to ask?
>
> BR, Felix
>
> --
>  Dipl.-Inform. Felix Palmen ,.//..
>  {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
>  {pgp public key} http://palmen-it.de/pub.txt   //   """
>  {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A
>

Re: Writing large build logs to NFS extremely slow?

[Rick Macklem]

Rick Macklem wrote:
?Felix Palmen wrote:
[stuff snipped]
>>Ok, thanks for the insight here! So maybe, I could have a look at
>>poudriere's code? If I understand you correctly, keeping the file open
>>and just issuing more write() calls wouldn't trigger that problem?
>Nope, I don't think keeping the file open would help much.
>However, if the file is opened with O_DIRECT, you could try setting
># sysctl vfs.nfs.nfs_directio_enable=1
>in the NFS client machine.
Oops, my bad..I for some reason was thinking you were using an NFSv3
mount, but you didn't say. For NFSv4, repeated open/close of the same
file will result in significant overhead. The official remedy for this would
be enabling delegations, but there were some significant issues with using
them that were only fixed recently, so you need a really up to date system
if you are going to enable delegations on the NFSv4 server.

rick

rick

> You could try the "nocto" mount option, which might help, if the log file
> is being re-opened many times, but I doubt it will help.

Well, I take any suggestion, so will try that as well and report back,
thanks again!

Side note, please don't CC/To me, I'm subscribed the list ;)

--
 Dipl.-Inform. Felix Palmen ,.//..
 {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
 {pgp public key} http://palmen-it.de/pub.txt   //   """
 {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A

Re: Writing large build logs to NFS extremely slow?

[Rick Macklem]

Felix Palmen wrote:
[stuff snipped]
>Ok, thanks for the insight here! So maybe, I could have a look at
>poudriere's code? If I understand you correctly, keeping the file open
>and just issuing more write() calls wouldn't trigger that problem?
Nope, I don't think keeping the file open would help much.
However, if the file is opened with O_DIRECT, you could try setting
# sysctl vfs.nfs.nfs_directio_enable=1
in the NFS client machine.

rick

> You could try the "nocto" mount option, which might help, if the log file
> is being re-opened many times, but I doubt it will help.

Well, I take any suggestion, so will try that as well and report back,
thanks again!

Side note, please don't CC/To me, I'm subscribed the list ;)

--
 Dipl.-Inform. Felix Palmen ,.//..
 {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
 {pgp public key} http://palmen-it.de/pub.txt   //   """
 {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A

Re: Writing large build logs to NFS extremely slow?

[Rick Macklem]

Felix Palmen wrote:
>* Rick Macklem  [20211009 15:35]:
>> Felix Palmen wrote:
>> Assuming your NFS performance is acceptable for other things and it
>> is only this log file that is a problem, then I doubt there is much you
>> can do to improve it.
>
>Yes, that's the only problem I found so far…
Another thing you could try is turning off synchronous writing at the NFS 
server.
# sysctl vfs.nfsd.async=1
and "sync=disabled" for ZFS if it is a ZFS file system that is exported.

However, BE FORWARNED:
Doing so violates the NFS RFCs because it can result is data loss/corrupted 
files
that are being written when the NFS server crashes/reboots.
Only do it if the data being written to the NFS server is not particularly 
important stuff.

rick

> --> Append (as in O_APPEND opens) are a poor case for NFS, since there
>   is no append write in NFS. To approximate append write, it must flush
>   all dirty data to the server, do a Getattr to find out the file's 
> current
>   size and then do the write (over and over and over again).

Ok, thanks for the insight here! So maybe, I could have a look at
poudriere's code? If I understand you correctly, keeping the file open
and just issuing more write() calls wouldn't trigger that problem?

> You could try the "nocto" mount option, which might help, if the log file
> is being re-opened many times, but I doubt it will help.

Well, I take any suggestion, so will try that as well and report back,
thanks again!

Side note, please don't CC/To me, I'm subscribed the list ;)

--
 Dipl.-Inform. Felix Palmen ,.//..
 {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
 {pgp public key} http://palmen-it.de/pub.txt   //   """
 {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A

Re: Writing large build logs to NFS extremely slow?

[Felix Palmen]

* Rick Macklem  [20211009 15:35]:
> Felix Palmen wrote:
> Assuming your NFS performance is acceptable for other things and it
> is only this log file that is a problem, then I doubt there is much you
> can do to improve it.

Yes, that's the only problem I found so far…

> --> Append (as in O_APPEND opens) are a poor case for NFS, since there
>   is no append write in NFS. To approximate append write, it must flush
>   all dirty data to the server, do a Getattr to find out the file's 
> current
>   size and then do the write (over and over and over again).

Ok, thanks for the insight here! So maybe, I could have a look at
poudriere's code? If I understand you correctly, keeping the file open
and just issuing more write() calls wouldn't trigger that problem?

> You could try the "nocto" mount option, which might help, if the log file
> is being re-opened many times, but I doubt it will help.

Well, I take any suggestion, so will try that as well and report back,
thanks again!

Side note, please don't CC/To me, I'm subscribed the list ;)

-- 
 Dipl.-Inform. Felix Palmen ,.//..
 {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
 {pgp public key} http://palmen-it.de/pub.txt   //   """
 {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A


signature.asc
Description: PGP signature

Re: Writing large build logs to NFS extremely slow?

[Rick Macklem]

Felix Palmen wrote:
>I use a -CURRENT bhyve vm for testing port builds with poudriere. As
>this vm is only running when needed, but I want to always have access to
>the build logs, I use NFS to mount /usr/local/poudriere/data/logs from
>the host.
>
>I noticed some few ports take ridiculously long to build while barely
>using any CPU time at all. On a closer look, that's all ports producing
>a lot of compiler (warning) output, e.g. gcc, gnutls, gtk2, …
>
>So I assume appending to a large file via NFS gets slower and slower. Is
>there any mount option I could try to fix this? Right now I only have
>`nolockd`, I also tried `noncontigwr` which didn't change anything.

Assuming your NFS performance is acceptable for other things and it
is only this log file that is a problem, then I doubt there is much you
can do to improve it.
--> Append (as in O_APPEND opens) are a poor case for NFS, since there
  is no append write in NFS. To approximate append write, it must flush
  all dirty data to the server, do a Getattr to find out the file's current
  size and then do the write (over and over and over again).
You can capture packets with tcpdump and then look at them in wireshark,
to see what I mean.

You could try the "nocto" mount option, which might help, if the log file
is being re-opened many times, but I doubt it will help.

rick

Thinking about alternatives to NFS, are there any news for client-side
9p virtfs? I found  which
still builds with a few minor adaptions, but trying to mount a 9p share
freezes the machine.

Re: Writing large build logs to NFS extremely slow?

[Felix Palmen]

* Mehmet Erol Sanliturk  [20211007 10:42]:
> The problem is caused mainly by  NFS definition parameters .
> 
> If you study NFS definition parameters one by one , I think you will be
> able to find which one is effective .
> 
> My opinion is the one setting is "write directly to disk" , i.e. , "do not
> use the cache" .
> 
> In the  "write directly to disk" case , without completion of a write , the
> computer in use is waiting for completion of previous write operation
> before writing a new record . This is useful in case of abrupt program
> terminations because every record is written into the disk file , by
> consuming more time .

Well this is kind of what I assume might be the problem. Could anyone
give me a hint what configuration to change on FreeBSD?

-- 
 Dipl.-Inform. Felix Palmen ,.//..
 {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
 {pgp public key} http://palmen-it.de/pub.txt   //   """
 {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A


signature.asc
Description: PGP signature

Re: Writing large build logs to NFS extremely slow?

[Mehmet Erol Sanliturk]

On Thu, Oct 7, 2021 at 5:17 AM Felix Palmen  wrote:

> Hi all,
>
> I use a -CURRENT bhyve vm for testing port builds with poudriere. As
> this vm is only running when needed, but I want to always have access to
> the build logs, I use NFS to mount /usr/local/poudriere/data/logs from
> the host.
>
> I noticed some few ports take ridiculously long to build while barely
> using any CPU time at all. On a closer look, that's all ports producing
> a lot of compiler (warning) output, e.g. gcc, gnutls, gtk2, …
>
> So I assume appending to a large file via NFS gets slower and slower. Is
> there any mount option I could try to fix this? Right now I only have
> `nolockd`, I also tried `noncontigwr` which didn't change anything.
>
> Thinking about alternatives to NFS, are there any news for client-side
> 9p virtfs? I found  which
> still builds with a few minor adaptions, but trying to mount a 9p share
> freezes the machine.
>
> Would you suggest a different mailing list to ask?
>
> BR, Felix
>
> --
>  Dipl.-Inform. Felix Palmen ,.//..
>  {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
>  {pgp public key} http://palmen-it.de/pub.txt   //   """
>  {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A
>



I have encountered such cases previously , but I am not able to remember
which parameters I have used to solve this problem ,
because I am not using the FreeBSD server now .
A similar problem occurs also in the Linux NFS server.

The problem is caused mainly by  NFS definition parameters .

If you study NFS definition parameters one by one , I think you will be
able to find which one is effective .

My opinion is the one setting is "write directly to disk" , i.e. , "do not
use the cache" .


In the  "write directly to disk" case , without completion of a write , the
computer in use is waiting for completion of previous write operation
before writing a new record . This is useful in case of abrupt program
terminations because every record is written into the disk file , by
consuming more time .

In the cache use case , time is not consumed much but the last written
records are lost in an abrupt  program termination .


My understanding from your question is this .


Mehmet Erol Sanliturk

Writing large build logs to NFS extremely slow?

[Felix Palmen]

Hi all,

I use a -CURRENT bhyve vm for testing port builds with poudriere. As
this vm is only running when needed, but I want to always have access to
the build logs, I use NFS to mount /usr/local/poudriere/data/logs from
the host.

I noticed some few ports take ridiculously long to build while barely
using any CPU time at all. On a closer look, that's all ports producing
a lot of compiler (warning) output, e.g. gcc, gnutls, gtk2, …

So I assume appending to a large file via NFS gets slower and slower. Is
there any mount option I could try to fix this? Right now I only have
`nolockd`, I also tried `noncontigwr` which didn't change anything.

Thinking about alternatives to NFS, are there any news for client-side
9p virtfs? I found  which
still builds with a few minor adaptions, but trying to mount a 9p share
freezes the machine.

Would you suggest a different mailing list to ask?

BR, Felix

-- 
 Dipl.-Inform. Felix Palmen ,.//..
 {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
 {pgp public key} http://palmen-it.de/pub.txt   //   """
 {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A


signature.asc
Description: PGP signature