Re: ggate + gmirror write performance woes
Hi!
On Mon, Apr 09, 2007 at 12:55:43PM -0400, Sven Willenberger wrote:
> The next part of the project will be writing the freevrrp failover
> scripts to deal with I/O locking issues that will happen if the ggated
> server fails, etc.
check carp(4) and ports/net/ifstated. You can use identical advbase
configuration on both nodes and prevent service migration back from
second node to first, if first node will reboot.
also, take a look one{start,stop,restart} options for rcNG designed
initscripts.
WBR
Dmitriy
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ggate + gmirror write performance woes
On Fri, 2007-04-06 at 16:18 +0300, Nikolay Pavlov wrote:
> On Thursday, 5 April 2007 at 16:15:35 -0400, Sven Willenberger wrote:
> > On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote:
> > > Dmitriy Kirhlarov wrote:
> > > > On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:
> > > >> I am trying to set up a HA type system involving two identical boxes
> > > >> and
> > > >> have gone through the following to set up the systems:
> > > >>
> > > >> Slave server:
> > > >> ggated -R 196608 -S 196608
> > > >> (exporting /dev/amrd1 )
> > > >> net.inet.tcp.sendspace: 65536
> > > >> net.inet.tcp.recvspace: 131072
> > > >
> > > > Try
> > > > net.local.stream.recvspace=65535
> > > > net.local.stream.sendspace=65535
> > > >
> > > > Also, try increase this sysctls with
> > > > net.inet.tcp.rfc1323=1
> > > >
> > > > I use it on FreeBSD 5.x with:
> > > > net.inet.tcp.sendspace=131072
> > > > net.inet.tcp.recvspace=131072
> > > > net.local.stream.recvspace=65535
> > > > net.local.stream.sendspace=65535
> > > >
> > > > ggated -R 1048576 -S 1048576
> > > > ggatec -R 1048576 -S 1048576
> > > >
> > > > WBR.
> > > > Dmitriy
> > > > ___
> > > > freebsd-stable@freebsd.org mailing list
> > > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > > > To unsubscribe, send any mail to "[EMAIL PROTECTED]"
> > >
> > >
> > > I have seen sustained writes of 30Mb/s using the following configuration:
> > >
> > > cat /boot/loader.conf
> > > kern.ipc.nmbclusters="32768"
> > >
> > > cat /etc/sysctl.conf
> > > net.inet.tcp.sendspace=1048576
> > > net.inet.tcp.recvspace=1048576
> > >
> > > Server:
> > > /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports
> > >
> > > Client:
> > > /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18
> > > /dev/amrd0s2
> > >
> > > The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with
> > > adaptive read ahead and write back caching.
> > >
> > > Tom
> >
> > I have tried both the settings ideas suggested above but I cannot even
> > get out of the gate with those. Setting net.inet.tcp.{send,recv}space to
> > anything higher that 131072 results in ggated bailing with the error:
> > # ggated -v -a 10.10.0.19
> > info: Reading exports file (/etc/gg.exports).
> > debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list.
> > debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list.
> > info: Exporting 2 object(s).
> > error: Cannot open stream socket: No buffer space available.
> > error: Exiting.
>
> For values of net.inet.tcp.{send,recv}space more than
> 524288 you also need to adjust kern.ipc.maxsockbuf
>
> Try this configuration for example:
> kern.ipc.maxsockbuf=2049152
> net.inet.tcp.recvspace=1024576
> net.inet.tcp.sendspace=1024576
>
kern.ipc.maxsockbuf was the issue here; I increased its value and now I
no longer get the buffer space error. Furthermore, the write speed issue
was also tied to a hardware raid controller issue. After fixing that
issue, and with just the following:
kern.ipc.maxsockbuf=1048576
net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=131072
I can start ggated with -R 262144 -S 262144 as well as the ggatec and
see write speeds of 60+MB/s. I may play around with the settings more
(and see if any further speed improvements occur), but this is quite
acceptable at this point. (For the record nmbclusters is set to 32768).
The next part of the project will be writing the freevrrp failover
scripts to deal with I/O locking issues that will happen if the ggated
server fails, etc.
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ggate + gmirror write performance woes
On Thursday, 5 April 2007 at 16:15:35 -0400, Sven Willenberger wrote:
> On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote:
> > Dmitriy Kirhlarov wrote:
> > > On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:
> > >> I am trying to set up a HA type system involving two identical boxes and
> > >> have gone through the following to set up the systems:
> > >>
> > >> Slave server:
> > >> ggated -R 196608 -S 196608
> > >> (exporting /dev/amrd1 )
> > >> net.inet.tcp.sendspace: 65536
> > >> net.inet.tcp.recvspace: 131072
> > >
> > > Try
> > > net.local.stream.recvspace=65535
> > > net.local.stream.sendspace=65535
> > >
> > > Also, try increase this sysctls with
> > > net.inet.tcp.rfc1323=1
> > >
> > > I use it on FreeBSD 5.x with:
> > > net.inet.tcp.sendspace=131072
> > > net.inet.tcp.recvspace=131072
> > > net.local.stream.recvspace=65535
> > > net.local.stream.sendspace=65535
> > >
> > > ggated -R 1048576 -S 1048576
> > > ggatec -R 1048576 -S 1048576
> > >
> > > WBR.
> > > Dmitriy
> > > ___
> > > freebsd-stable@freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > > To unsubscribe, send any mail to "[EMAIL PROTECTED]"
> >
> >
> > I have seen sustained writes of 30Mb/s using the following configuration:
> >
> > cat /boot/loader.conf
> > kern.ipc.nmbclusters="32768"
> >
> > cat /etc/sysctl.conf
> > net.inet.tcp.sendspace=1048576
> > net.inet.tcp.recvspace=1048576
> >
> > Server:
> > /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports
> >
> > Client:
> > /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18
> > /dev/amrd0s2
> >
> > The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with
> > adaptive read ahead and write back caching.
> >
> > Tom
>
> I have tried both the settings ideas suggested above but I cannot even
> get out of the gate with those. Setting net.inet.tcp.{send,recv}space to
> anything higher that 131072 results in ggated bailing with the error:
> # ggated -v -a 10.10.0.19
> info: Reading exports file (/etc/gg.exports).
> debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list.
> debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list.
> info: Exporting 2 object(s).
> error: Cannot open stream socket: No buffer space available.
> error: Exiting.
For values of net.inet.tcp.{send,recv}space more than
524288 you also need to adjust kern.ipc.maxsockbuf
Try this configuration for example:
kern.ipc.maxsockbuf=2049152
net.inet.tcp.recvspace=1024576
net.inet.tcp.sendspace=1024576
>
> setting net.inet.tcp.{send,recv}space to 131072 allows me to start
> ggated with the default R and S values of 131072; anything higher
> results in "no buffer space" errors. At 131072 ggated starts but then I
> cannot even open a new connection (like ssh) to the server as the ssh
> client bails with "no buffer space available".
>
> more information:
> # netstat -m
> 514/641/1155 mbufs in use (current/cache/total)
> 512/284/796/32768 mbuf clusters in use (current/cache/total/max)
> 512/256 mbuf+clusters out of packet secondary zone in use
> (current/cache)
> 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
> 0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
> 0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
> 1152K/728K/1880K bytes allocated to network (current/cache/total)
> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
> 0/4/6656 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 0 requests for I/O initiated by sendfile
> 0 calls to protocol drain routines
>
> This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA
> Raid using LSiMegaRaid.
>
> The odd thing is that even after I set the send and recvspace down to
> values like 65536, I continue to get the no buffer error when trying to
> connect to it remotely again.
>
> Sven
>
> ___
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "[EMAIL PROTECTED]"
--
==
- Best regards, Nikolay Pavlov. <<<---
==
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ggate + gmirror write performance woes
Sven Willenberger wrote:
On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote:
Dmitriy Kirhlarov wrote:
On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:
I am trying to set up a HA type system involving two identical boxes and
have gone through the following to set up the systems:
Slave server:
ggated -R 196608 -S 196608
(exporting /dev/amrd1 )
net.inet.tcp.sendspace: 65536
net.inet.tcp.recvspace: 131072
Try
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535
Also, try increase this sysctls with
net.inet.tcp.rfc1323=1
I use it on FreeBSD 5.x with:
net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=131072
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535
ggated -R 1048576 -S 1048576
ggatec -R 1048576 -S 1048576
WBR.
Dmitriy
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
I have seen sustained writes of 30Mb/s using the following configuration:
cat /boot/loader.conf
kern.ipc.nmbclusters="32768"
cat /etc/sysctl.conf
net.inet.tcp.sendspace=1048576
net.inet.tcp.recvspace=1048576
Server:
/sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports
Client:
/sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18
/dev/amrd0s2
The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with
adaptive read ahead and write back caching.
Tom
I have tried both the settings ideas suggested above but I cannot even
get out of the gate with those. Setting net.inet.tcp.{send,recv}space to
anything higher that 131072 results in ggated bailing with the error:
# ggated -v -a 10.10.0.19
info: Reading exports file (/etc/gg.exports).
debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list.
debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list.
info: Exporting 2 object(s).
error: Cannot open stream socket: No buffer space available.
error: Exiting.
setting net.inet.tcp.{send,recv}space to 131072 allows me to start
ggated with the default R and S values of 131072; anything higher
results in "no buffer space" errors. At 131072 ggated starts but then I
cannot even open a new connection (like ssh) to the server as the ssh
client bails with "no buffer space available".
Did you also set kern.ipc.nmbclusters="32768" in /boot/loader.conf and
reboot? It sounds like you did not as this is the exact same problem I
came across before adjusting that value.
<>
This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA
Raid using LSiMegaRaid.
Do you have the cache BBU fitted (Batery Backup Unit) and the array
caching set to write back? Also have you tested writing to the array
locally without ggate to test the write speed?
The odd thing is that even after I set the send and recvspace down to
values like 65536, I continue to get the no buffer error when trying to
connect to it remotely again.
I found that the easyest way to fix this was to reboot the system with
good values for net.inet.tcp.{send,recv}space.
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ggate + gmirror write performance woes
Hi!
On Thu, Apr 05, 2007 at 04:15:35PM -0400, Sven Willenberger wrote:
> I have tried both the settings ideas suggested above but I cannot even
> get out of the gate with those. Setting net.inet.tcp.{send,recv}space to
>
> setting net.inet.tcp.{send,recv}space to 131072 allows me to start
Now you have good chance to check differences with other recommended
sysctl's. :)
WBR
Dmitriy
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ggate + gmirror write performance woes
On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote:
> Dmitriy Kirhlarov wrote:
> > On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:
> >> I am trying to set up a HA type system involving two identical boxes and
> >> have gone through the following to set up the systems:
> >>
> >> Slave server:
> >> ggated -R 196608 -S 196608
> >> (exporting /dev/amrd1 )
> >> net.inet.tcp.sendspace: 65536
> >> net.inet.tcp.recvspace: 131072
> >
> > Try
> > net.local.stream.recvspace=65535
> > net.local.stream.sendspace=65535
> >
> > Also, try increase this sysctls with
> > net.inet.tcp.rfc1323=1
> >
> > I use it on FreeBSD 5.x with:
> > net.inet.tcp.sendspace=131072
> > net.inet.tcp.recvspace=131072
> > net.local.stream.recvspace=65535
> > net.local.stream.sendspace=65535
> >
> > ggated -R 1048576 -S 1048576
> > ggatec -R 1048576 -S 1048576
> >
> > WBR.
> > Dmitriy
> > ___
> > freebsd-stable@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > To unsubscribe, send any mail to "[EMAIL PROTECTED]"
>
>
> I have seen sustained writes of 30Mb/s using the following configuration:
>
> cat /boot/loader.conf
> kern.ipc.nmbclusters="32768"
>
> cat /etc/sysctl.conf
> net.inet.tcp.sendspace=1048576
> net.inet.tcp.recvspace=1048576
>
> Server:
> /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports
>
> Client:
> /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18
> /dev/amrd0s2
>
> The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with
> adaptive read ahead and write back caching.
>
> Tom
I have tried both the settings ideas suggested above but I cannot even
get out of the gate with those. Setting net.inet.tcp.{send,recv}space to
anything higher that 131072 results in ggated bailing with the error:
# ggated -v -a 10.10.0.19
info: Reading exports file (/etc/gg.exports).
debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list.
debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list.
info: Exporting 2 object(s).
error: Cannot open stream socket: No buffer space available.
error: Exiting.
setting net.inet.tcp.{send,recv}space to 131072 allows me to start
ggated with the default R and S values of 131072; anything higher
results in "no buffer space" errors. At 131072 ggated starts but then I
cannot even open a new connection (like ssh) to the server as the ssh
client bails with "no buffer space available".
more information:
# netstat -m
514/641/1155 mbufs in use (current/cache/total)
512/284/796/32768 mbuf clusters in use (current/cache/total/max)
512/256 mbuf+clusters out of packet secondary zone in use
(current/cache)
0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
1152K/728K/1880K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/4/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA
Raid using LSiMegaRaid.
The odd thing is that even after I set the send and recvspace down to
values like 65536, I continue to get the no buffer error when trying to
connect to it remotely again.
Sven
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ggate + gmirror write performance woes
Dmitriy Kirhlarov wrote: On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote: I am trying to set up a HA type system involving two identical boxes and have gone through the following to set up the systems: Slave server: ggated -R 196608 -S 196608 (exporting /dev/amrd1 ) net.inet.tcp.sendspace: 65536 net.inet.tcp.recvspace: 131072 Try net.local.stream.recvspace=65535 net.local.stream.sendspace=65535 Also, try increase this sysctls with net.inet.tcp.rfc1323=1 I use it on FreeBSD 5.x with: net.inet.tcp.sendspace=131072 net.inet.tcp.recvspace=131072 net.local.stream.recvspace=65535 net.local.stream.sendspace=65535 ggated -R 1048576 -S 1048576 ggatec -R 1048576 -S 1048576 WBR. Dmitriy ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]" I have seen sustained writes of 30Mb/s using the following configuration: cat /boot/loader.conf kern.ipc.nmbclusters="32768" cat /etc/sysctl.conf net.inet.tcp.sendspace=1048576 net.inet.tcp.recvspace=1048576 Server: /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports Client: /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 /dev/amrd0s2 The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with adaptive read ahead and write back caching. Tom ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: ggate + gmirror write performance woes
On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote: > I am trying to set up a HA type system involving two identical boxes and > have gone through the following to set up the systems: > > Slave server: > ggated -R 196608 -S 196608 > (exporting /dev/amrd1 ) > net.inet.tcp.sendspace: 65536 > net.inet.tcp.recvspace: 131072 Try net.local.stream.recvspace=65535 net.local.stream.sendspace=65535 Also, try increase this sysctls with net.inet.tcp.rfc1323=1 I use it on FreeBSD 5.x with: net.inet.tcp.sendspace=131072 net.inet.tcp.recvspace=131072 net.local.stream.recvspace=65535 net.local.stream.sendspace=65535 ggated -R 1048576 -S 1048576 ggatec -R 1048576 -S 1048576 WBR. Dmitriy ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
