Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
Thanks for the answer, it seems I'm not missing dirhashes after all.. :) -Dan Apr 12, 2024 16:25:51 Otto Moerbeek : > On Fri, Apr 12, 2024 at 12:21:43PM +0200, Dan wrote: > >> >> Really, I fear this value is due to a wrong tweak.. > > > Fear is a bad advisor. > > If you look at man 3 sysctl, you'll see what vfs.ffs.dirhash_mem > means: > > FFS_DIRHASH_MEM (vfs.ffs.dirhash_mem) > The amount of memory currently used by all directory > hashes. > > In other words a harmless mem usage metric. > > -Otto > >> >> -Dan >> >> Apr 12, 2024 09:09:06 Dan : >> >>> Yes, that fixes it for me: $ sysctl vfs.ffs vfs.ffs.dirhash_dirsize=2560 vfs.ffs.dirhash_maxmem=5242880 vfs.ffs.dirhash_mem=767359 >>> >>> >>> I have this value in 7.4 stable: >>> >>> vfs.ffs.dirhash_mem=1412837 >>> >>> is it correct? or how to fix it? >>> >>> >>> -Dan >>
Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
On Fri, Apr 12, 2024 at 12:21:43PM +0200, Dan wrote: > > Really, I fear this value is due to a wrong tweak.. Fear is a bad advisor. If you look at man 3 sysctl, you'll see what vfs.ffs.dirhash_mem means: FFS_DIRHASH_MEM (vfs.ffs.dirhash_mem) The amount of memory currently used by all directory hashes. In other words a harmless mem usage metric. -Otto > > -Dan > > Apr 12, 2024 09:09:06 Dan : > > > > >> Yes, that fixes it for me: > >> > >> $ sysctl vfs.ffs > >> vfs.ffs.dirhash_dirsize=2560 > >> vfs.ffs.dirhash_maxmem=5242880 > >> vfs.ffs.dirhash_mem=767359 > > > > > > I have this value in 7.4 stable: > > > > vfs.ffs.dirhash_mem=1412837 > > > > is it correct? or how to fix it? > > > > > > -Dan >
Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
Really, I fear this value is due to a wrong tweak.. -Dan Apr 12, 2024 09:09:06 Dan : > >> Yes, that fixes it for me: >> >> $ sysctl vfs.ffs >> vfs.ffs.dirhash_dirsize=2560 >> vfs.ffs.dirhash_maxmem=5242880 >> vfs.ffs.dirhash_mem=767359 > > > I have this value in 7.4 stable: > > vfs.ffs.dirhash_mem=1412837 > > is it correct? or how to fix it? > > > -Dan
Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
> Yes, that fixes it for me: > > $ sysctl vfs.ffs > vfs.ffs.dirhash_dirsize=2560 > vfs.ffs.dirhash_maxmem=5242880 > vfs.ffs.dirhash_mem=767359 I have this value in 7.4 stable: vfs.ffs.dirhash_mem=1412837 is it correct? or how to fix it? -Dan
Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
On Thu, Apr 11, 2024 at 06:15:14PM +0200, Otto Moerbeek wrote:
> On Thu, Apr 11, 2024 at 05:29:14PM +0200, Otto Moerbeek wrote:
>
> > On Thu, Apr 11, 2024 at 05:20:24PM +0200, Otto Moerbeek wrote:
> >
> > > On Thu, Apr 11, 2024 at 05:08:01PM +0200, Federico Giannici wrote:
> > >
> > > > On 4/11/24 16:15, Claudio Jeker wrote:
> > > > > On Thu, Apr 11, 2024 at 03:36:29PM +0200, Federico Giannici wrote:
> > > > > > On 4/11/24 14:12, Nick Holland wrote:
> > > > > > > On 4/11/24 05:47, Federico Giannici wrote:
> > > > > > > > We have a server with A LOT of files in some directories (an
> > > > > > > > email
> > > > > > > > server in maildir format).
> > > > > > > >
> > > > > > > > Since we upgraded from OpenBSD amd64 7.3 to 7.5 (passing
> > > > > > > > through 7.4) it
> > > > > > > > became very very very slow to access these large directories!
> > > > > > > ,,,
> > > > > > > You may be being bitten by the removal of softdeps (soft updates)
> > > > > > > in 7.4 more than the availability of a knob to twist. This was a
> > > > > > > huge hit for some things -- I had one backup job go from a couple
> > > > > > > hours to eight or so hours. However, it turned out that increase
> > > > > > > in time has not inconvenienced me at all, and some random lockups
> > > > > > > related to softdeps have gone away. Overall, win for me (the
> > > > > > > fscks after a lockup took hours, too, not to mention all the time
> > > > > > > and effort spent replacing part after part assuming it was a HW
> > > > > > > issue).
> > > > > > >
> > > > > > > As I understand it...there were known (known unknown?) bugs in the
> > > > > > > softdep code, the code was ugly, and it made it difficult to
> > > > > > > actually improve the code.
> > > > > >
> > > > > > No, we knew that softdeps were being deprecated and we removed from
> > > > > > everywhere some time ago. It must be something else.
> > > > > >
> > > > > > Anyway, it's strange that dirhash parameters has being changed and
> > > > > > removed
> > > > > > without any mention...
> > > > >
> > > > > It was not (this is on -current amd64):
> > > > > vfs.ffs.dirhash_dirsize=2560
> > > > > vfs.ffs.dirhash_maxmem=5242880
> > > > > vfs.ffs.dirhash_mem=4832510
> > > > >
> > > > > Are you sure your kernel and userland are in sync?
> > > >
> > > > Well, I followed the prescribed procedure (I'm using OpenBSD since...
> > > > about
> > > > 20 years).
> > > >
> > > > In EVERY machine upgraded to 7.5 now I have something like this:
> > > >
> > > > Elrond:/home/giannici$ sysctl vfs.ffs
> > > >
> > > >
> > > >
> > > > vfs.ffs.dirhash_maxmem=2560
> > > > vfs.ffs.dirhash_mem=5242880
> > > >
> > > > Elrond:/home/giannici$ uname -a
> > > >
> > > >
> > > >
> > > > OpenBSD Elrond.neomedia.it 7.5 GENERIC.MP#82 amd64
> > > >
> > > >
> > > > So, I'm the only one?
> > > >
> > > > Thanks.
> > > >
> > >
> > > I suspect
> > >
> > > In
> > > https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ffs/ffs_exatern.h.diff?r1=1.45&r2=1.46&f=h
> > >
> > > The max_softdeps entry in ffs_extern.h should have been replace by a
> > > { 0, 0 }
> > >
> > > instead of being removed.
> > >
> > > -Otto
> > >
> >
> > Yes, that fixes it for me:
> >
> > $ sysctl vfs.ffs
> > vfs.ffs.dirhash_dirsize=2560
> > vfs.ffs.dirhash_maxmem=5242880
> > vfs.ffs.dirhash_mem=767359
> > $
> >
>
> to elaborate a bit: the vfs.ffs.dirhash_maxmem enty was actually
> showing the value of the vfs.ffs.dirhash_dirsize entry. Trying to set the
> vfs.ffs.dirhash_maxmem entry would result into setting the
> vfs.ffs.dirhash_dirsize entry, which effectively disables dirhash if
> yon set it to a high value.
>
> It remains a mystery why Claudio is seeing the correct values... you'd
> almost think he uses an old sysctl binary, or his tree is out of sync
> somehow.
>
My bad I ran the command on a system that was still running 7.4 without
realizing that.
--
:wq Claudio
Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
On Thu, Apr 11, 2024 at 05:29:14PM +0200, Otto Moerbeek wrote:
> On Thu, Apr 11, 2024 at 05:20:24PM +0200, Otto Moerbeek wrote:
>
> > On Thu, Apr 11, 2024 at 05:08:01PM +0200, Federico Giannici wrote:
> >
> > > On 4/11/24 16:15, Claudio Jeker wrote:
> > > > On Thu, Apr 11, 2024 at 03:36:29PM +0200, Federico Giannici wrote:
> > > > > On 4/11/24 14:12, Nick Holland wrote:
> > > > > > On 4/11/24 05:47, Federico Giannici wrote:
> > > > > > > We have a server with A LOT of files in some directories (an email
> > > > > > > server in maildir format).
> > > > > > >
> > > > > > > Since we upgraded from OpenBSD amd64 7.3 to 7.5 (passing through
> > > > > > > 7.4) it
> > > > > > > became very very very slow to access these large directories!
> > > > > > ,,,
> > > > > > You may be being bitten by the removal of softdeps (soft updates)
> > > > > > in 7.4 more than the availability of a knob to twist. This was a
> > > > > > huge hit for some things -- I had one backup job go from a couple
> > > > > > hours to eight or so hours. However, it turned out that increase
> > > > > > in time has not inconvenienced me at all, and some random lockups
> > > > > > related to softdeps have gone away. Overall, win for me (the
> > > > > > fscks after a lockup took hours, too, not to mention all the time
> > > > > > and effort spent replacing part after part assuming it was a HW
> > > > > > issue).
> > > > > >
> > > > > > As I understand it...there were known (known unknown?) bugs in the
> > > > > > softdep code, the code was ugly, and it made it difficult to
> > > > > > actually improve the code.
> > > > >
> > > > > No, we knew that softdeps were being deprecated and we removed from
> > > > > everywhere some time ago. It must be something else.
> > > > >
> > > > > Anyway, it's strange that dirhash parameters has being changed and
> > > > > removed
> > > > > without any mention...
> > > >
> > > > It was not (this is on -current amd64):
> > > > vfs.ffs.dirhash_dirsize=2560
> > > > vfs.ffs.dirhash_maxmem=5242880
> > > > vfs.ffs.dirhash_mem=4832510
> > > >
> > > > Are you sure your kernel and userland are in sync?
> > >
> > > Well, I followed the prescribed procedure (I'm using OpenBSD since...
> > > about
> > > 20 years).
> > >
> > > In EVERY machine upgraded to 7.5 now I have something like this:
> > >
> > > Elrond:/home/giannici$ sysctl vfs.ffs
> > >
> > >
> > >
> > > vfs.ffs.dirhash_maxmem=2560
> > > vfs.ffs.dirhash_mem=5242880
> > >
> > > Elrond:/home/giannici$ uname -a
> > >
> > >
> > >
> > > OpenBSD Elrond.neomedia.it 7.5 GENERIC.MP#82 amd64
> > >
> > >
> > > So, I'm the only one?
> > >
> > > Thanks.
> > >
> >
> > I suspect
> >
> > In
> > https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ffs/ffs_exatern.h.diff?r1=1.45&r2=1.46&f=h
> >
> > The max_softdeps entry in ffs_extern.h should have been replace by a
> > { 0, 0 }
> >
> > instead of being removed.
> >
> > -Otto
> >
>
> Yes, that fixes it for me:
>
> $ sysctl vfs.ffs
> vfs.ffs.dirhash_dirsize=2560
> vfs.ffs.dirhash_maxmem=5242880
> vfs.ffs.dirhash_mem=767359
> $
>
to elaborate a bit: the vfs.ffs.dirhash_maxmem enty was actually
showing the value of the vfs.ffs.dirhash_dirsize entry. Trying to set the
vfs.ffs.dirhash_maxmem entry would result into setting the
vfs.ffs.dirhash_dirsize entry, which effectively disables dirhash if
yon set it to a high value.
It remains a mystery why Claudio is seeing the correct values... you'd
almost think he uses an old sysctl binary, or his tree is out of sync
somehow.
-Otto
Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
On Thu, Apr 11, 2024 at 05:20:24PM +0200, Otto Moerbeek wrote:
> On Thu, Apr 11, 2024 at 05:08:01PM +0200, Federico Giannici wrote:
>
> > On 4/11/24 16:15, Claudio Jeker wrote:
> > > On Thu, Apr 11, 2024 at 03:36:29PM +0200, Federico Giannici wrote:
> > > > On 4/11/24 14:12, Nick Holland wrote:
> > > > > On 4/11/24 05:47, Federico Giannici wrote:
> > > > > > We have a server with A LOT of files in some directories (an email
> > > > > > server in maildir format).
> > > > > >
> > > > > > Since we upgraded from OpenBSD amd64 7.3 to 7.5 (passing through
> > > > > > 7.4) it
> > > > > > became very very very slow to access these large directories!
> > > > > ,,,
> > > > > You may be being bitten by the removal of softdeps (soft updates)
> > > > > in 7.4 more than the availability of a knob to twist. This was a
> > > > > huge hit for some things -- I had one backup job go from a couple
> > > > > hours to eight or so hours. However, it turned out that increase
> > > > > in time has not inconvenienced me at all, and some random lockups
> > > > > related to softdeps have gone away. Overall, win for me (the
> > > > > fscks after a lockup took hours, too, not to mention all the time
> > > > > and effort spent replacing part after part assuming it was a HW
> > > > > issue).
> > > > >
> > > > > As I understand it...there were known (known unknown?) bugs in the
> > > > > softdep code, the code was ugly, and it made it difficult to
> > > > > actually improve the code.
> > > >
> > > > No, we knew that softdeps were being deprecated and we removed from
> > > > everywhere some time ago. It must be something else.
> > > >
> > > > Anyway, it's strange that dirhash parameters has being changed and
> > > > removed
> > > > without any mention...
> > >
> > > It was not (this is on -current amd64):
> > > vfs.ffs.dirhash_dirsize=2560
> > > vfs.ffs.dirhash_maxmem=5242880
> > > vfs.ffs.dirhash_mem=4832510
> > >
> > > Are you sure your kernel and userland are in sync?
> >
> > Well, I followed the prescribed procedure (I'm using OpenBSD since... about
> > 20 years).
> >
> > In EVERY machine upgraded to 7.5 now I have something like this:
> >
> > Elrond:/home/giannici$ sysctl vfs.ffs
> >
> >
> >
> > vfs.ffs.dirhash_maxmem=2560
> > vfs.ffs.dirhash_mem=5242880
> >
> > Elrond:/home/giannici$ uname -a
> >
> >
> >
> > OpenBSD Elrond.neomedia.it 7.5 GENERIC.MP#82 amd64
> >
> >
> > So, I'm the only one?
> >
> > Thanks.
> >
>
> I suspect
>
> In
> https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ffs/ffs_exatern.h.diff?r1=1.45&r2=1.46&f=h
>
> The max_softdeps entry in ffs_extern.h should have been replace by a
> { 0, 0 }
>
> instead of being removed.
>
> -Otto
>
Yes, that fixes it for me:
$ sysctl vfs.ffs
vfs.ffs.dirhash_dirsize=2560
vfs.ffs.dirhash_maxmem=5242880
vfs.ffs.dirhash_mem=767359
$
Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
On Thu, Apr 11, 2024 at 05:08:01PM +0200, Federico Giannici wrote:
> On 4/11/24 16:15, Claudio Jeker wrote:
> > On Thu, Apr 11, 2024 at 03:36:29PM +0200, Federico Giannici wrote:
> > > On 4/11/24 14:12, Nick Holland wrote:
> > > > On 4/11/24 05:47, Federico Giannici wrote:
> > > > > We have a server with A LOT of files in some directories (an email
> > > > > server in maildir format).
> > > > >
> > > > > Since we upgraded from OpenBSD amd64 7.3 to 7.5 (passing through 7.4)
> > > > > it
> > > > > became very very very slow to access these large directories!
> > > > ,,,
> > > > You may be being bitten by the removal of softdeps (soft updates)
> > > > in 7.4 more than the availability of a knob to twist. This was a
> > > > huge hit for some things -- I had one backup job go from a couple
> > > > hours to eight or so hours. However, it turned out that increase
> > > > in time has not inconvenienced me at all, and some random lockups
> > > > related to softdeps have gone away. Overall, win for me (the
> > > > fscks after a lockup took hours, too, not to mention all the time
> > > > and effort spent replacing part after part assuming it was a HW
> > > > issue).
> > > >
> > > > As I understand it...there were known (known unknown?) bugs in the
> > > > softdep code, the code was ugly, and it made it difficult to
> > > > actually improve the code.
> > >
> > > No, we knew that softdeps were being deprecated and we removed from
> > > everywhere some time ago. It must be something else.
> > >
> > > Anyway, it's strange that dirhash parameters has being changed and removed
> > > without any mention...
> >
> > It was not (this is on -current amd64):
> > vfs.ffs.dirhash_dirsize=2560
> > vfs.ffs.dirhash_maxmem=5242880
> > vfs.ffs.dirhash_mem=4832510
> >
> > Are you sure your kernel and userland are in sync?
>
> Well, I followed the prescribed procedure (I'm using OpenBSD since... about
> 20 years).
>
> In EVERY machine upgraded to 7.5 now I have something like this:
>
> Elrond:/home/giannici$ sysctl vfs.ffs
>
>
>
> vfs.ffs.dirhash_maxmem=2560
> vfs.ffs.dirhash_mem=5242880
>
> Elrond:/home/giannici$ uname -a
>
>
>
> OpenBSD Elrond.neomedia.it 7.5 GENERIC.MP#82 amd64
>
>
> So, I'm the only one?
>
> Thanks.
>
I suspect
In
https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ffs/ffs_exatern.h.diff?r1=1.45&r2=1.46&f=h
The max_softdeps entry in ffs_extern.h should have been replace by a
{ 0, 0 }
instead of being removed.
-Otto
Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
On 4/11/24 16:15, Claudio Jeker wrote: On Thu, Apr 11, 2024 at 03:36:29PM +0200, Federico Giannici wrote: On 4/11/24 14:12, Nick Holland wrote: On 4/11/24 05:47, Federico Giannici wrote: We have a server with A LOT of files in some directories (an email server in maildir format). Since we upgraded from OpenBSD amd64 7.3 to 7.5 (passing through 7.4) it became very very very slow to access these large directories! ,,, You may be being bitten by the removal of softdeps (soft updates) in 7.4 more than the availability of a knob to twist. This was a huge hit for some things -- I had one backup job go from a couple hours to eight or so hours. However, it turned out that increase in time has not inconvenienced me at all, and some random lockups related to softdeps have gone away. Overall, win for me (the fscks after a lockup took hours, too, not to mention all the time and effort spent replacing part after part assuming it was a HW issue). As I understand it...there were known (known unknown?) bugs in the softdep code, the code was ugly, and it made it difficult to actually improve the code. No, we knew that softdeps were being deprecated and we removed from everywhere some time ago. It must be something else. Anyway, it's strange that dirhash parameters has being changed and removed without any mention... It was not (this is on -current amd64): vfs.ffs.dirhash_dirsize=2560 vfs.ffs.dirhash_maxmem=5242880 vfs.ffs.dirhash_mem=4832510 Are you sure your kernel and userland are in sync? Well, I followed the prescribed procedure (I'm using OpenBSD since... about 20 years). In EVERY machine upgraded to 7.5 now I have something like this: Elrond:/home/giannici$ sysctl vfs.ffs vfs.ffs.dirhash_maxmem=2560 vfs.ffs.dirhash_mem=5242880 Elrond:/home/giannici$ uname -a OpenBSD Elrond.neomedia.it 7.5 GENERIC.MP#82 amd64 So, I'm the only one? Thanks.
Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
On Thu, Apr 11, 2024 at 04:15:19PM +0200, Claudio Jeker wrote: > On Thu, Apr 11, 2024 at 03:36:29PM +0200, Federico Giannici wrote: > > On 4/11/24 14:12, Nick Holland wrote: > > > On 4/11/24 05:47, Federico Giannici wrote: > > > > We have a server with A LOT of files in some directories (an email > > > > server in maildir format). > > > > > > > > Since we upgraded from OpenBSD amd64 7.3 to 7.5 (passing through 7.4) it > > > > became very very very slow to access these large directories! > > > ,,, > > > You may be being bitten by the removal of softdeps (soft updates) > > > in 7.4 more than the availability of a knob to twist. This was a > > > huge hit for some things -- I had one backup job go from a couple > > > hours to eight or so hours. However, it turned out that increase > > > in time has not inconvenienced me at all, and some random lockups > > > related to softdeps have gone away. Overall, win for me (the > > > fscks after a lockup took hours, too, not to mention all the time > > > and effort spent replacing part after part assuming it was a HW > > > issue). > > > > > > As I understand it...there were known (known unknown?) bugs in the > > > softdep code, the code was ugly, and it made it difficult to > > > actually improve the code. > > > > No, we knew that softdeps were being deprecated and we removed from > > everywhere some time ago. It must be something else. > > > > Anyway, it's strange that dirhash parameters has being changed and removed > > without any mention... > > It was not (this is on -current amd64): > vfs.ffs.dirhash_dirsize=2560 > vfs.ffs.dirhash_maxmem=5242880 > vfs.ffs.dirhash_mem=4832510 > > Are you sure your kernel and userland are in sync? > -- > :wq Claudio > Strange, on a very recent snap: $ sysctl | grep ffs.d vfs.ffs.dirhash_maxmem=2560 vfs.ffs.dirhash_mem=5242880 $ -Otto
Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
On Thu, Apr 11, 2024 at 03:36:29PM +0200, Federico Giannici wrote: > On 4/11/24 14:12, Nick Holland wrote: > > On 4/11/24 05:47, Federico Giannici wrote: > > > We have a server with A LOT of files in some directories (an email > > > server in maildir format). > > > > > > Since we upgraded from OpenBSD amd64 7.3 to 7.5 (passing through 7.4) it > > > became very very very slow to access these large directories! > > ,,, > > You may be being bitten by the removal of softdeps (soft updates) > > in 7.4 more than the availability of a knob to twist. This was a > > huge hit for some things -- I had one backup job go from a couple > > hours to eight or so hours. However, it turned out that increase > > in time has not inconvenienced me at all, and some random lockups > > related to softdeps have gone away. Overall, win for me (the > > fscks after a lockup took hours, too, not to mention all the time > > and effort spent replacing part after part assuming it was a HW > > issue). > > > > As I understand it...there were known (known unknown?) bugs in the > > softdep code, the code was ugly, and it made it difficult to > > actually improve the code. > > No, we knew that softdeps were being deprecated and we removed from > everywhere some time ago. It must be something else. > > Anyway, it's strange that dirhash parameters has being changed and removed > without any mention... It was not (this is on -current amd64): vfs.ffs.dirhash_dirsize=2560 vfs.ffs.dirhash_maxmem=5242880 vfs.ffs.dirhash_mem=4832510 Are you sure your kernel and userland are in sync? -- :wq Claudio
Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
On 4/11/24 14:12, Nick Holland wrote: On 4/11/24 05:47, Federico Giannici wrote: We have a server with A LOT of files in some directories (an email server in maildir format). Since we upgraded from OpenBSD amd64 7.3 to 7.5 (passing through 7.4) it became very very very slow to access these large directories! ,,, You may be being bitten by the removal of softdeps (soft updates) in 7.4 more than the availability of a knob to twist. This was a huge hit for some things -- I had one backup job go from a couple hours to eight or so hours. However, it turned out that increase in time has not inconvenienced me at all, and some random lockups related to softdeps have gone away. Overall, win for me (the fscks after a lockup took hours, too, not to mention all the time and effort spent replacing part after part assuming it was a HW issue). As I understand it...there were known (known unknown?) bugs in the softdep code, the code was ugly, and it made it difficult to actually improve the code. No, we knew that softdeps were being deprecated and we removed from everywhere some time ago. It must be something else. Anyway, it's strange that dirhash parameters has being changed and removed without any mention...
Re: Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
On 4/11/24 05:47, Federico Giannici wrote: We have a server with A LOT of files in some directories (an email server in maildir format). Since we upgraded from OpenBSD amd64 7.3 to 7.5 (passing through 7.4) it became very very very slow to access these large directories! ,,, You may be being bitten by the removal of softdeps (soft updates) in 7.4 more than the availability of a knob to twist. This was a huge hit for some things -- I had one backup job go from a couple hours to eight or so hours. However, it turned out that increase in time has not inconvenienced me at all, and some random lockups related to softdeps have gone away. Overall, win for me (the fscks after a lockup took hours, too, not to mention all the time and effort spent replacing part after part assuming it was a HW issue). As I understand it...there were known (known unknown?) bugs in the softdep code, the code was ugly, and it made it difficult to actually improve the code. Nick.
Upgraded to 7.5: vfs.ffs.dirhash_dirsize no longer exists and large directory ere veeery slow
We have a server with A LOT of files in some directories (an email server in maildir format). Since we upgraded from OpenBSD amd64 7.3 to 7.5 (passing through 7.4) it became very very very slow to access these large directories! Up to 7.3 we used the following sysctl settings and everything was ok: vfs.ffs.dirhash_maxmem=1 vfs.ffs.dirhash_dirsize=1024 But now, with 7.5, vfs.ffs.dirhash_dirsize no longer exists! Only vfs.ffs.dirhash_maxmem and vfs.ffs.dirhash_mem exist, but it's not clear how we can use them. For example, according to "man 2 sysctl" vfs.ffs.dirhash_mem is NOT Changeable, but we were able to change it! But it is not clear what it is for... Anyway, the question is: under OpenBSD 7.5, how can we improve the access to directories with A LOT of files (normal FFF2 filesystem)? We already increased the usage of cache memory with "kern.bufcachepercent=80", but it's not enough. Thanks.
