/var/log/messages Mar 23 22:21:50 sabertooth kernel: swap zone exhausted, increase kern.maxswzone Mar 23 22:21:50 sabertooth kernel: pid 86697 (mysqld), uid 88, was killed: out of swap space
how to repeat:
$ mysql -ux < file.sql (~150GB) worth
basically, it slows down continually until it dies. IF you (suspend)
the process in time it recovers some, but eventually you have to suspend
it every 1s for ~3 minutes. The load is ~10 at this point.
I've looked at top, ps, iostat, zpool iostat, vmstat -z, vmstat -m
and I don't see anything wonky. I can provide more info on request.
system description:
$ df
zmysqlD 801G 658G 142G 82% /var/db/mysql/data
zmysqlL 133G 26G 107G 20% /var/db/mysql/log
its a 600GB innodb space, mysql has
innodb_buffer_pool_size = 80GB
about 1GB of data is MyISAM the rest is InnoDB
The machine has 96GB of RAM
$ cat /etc/fstab
/dev/gpt/swap0 none swap sw 0 0
/dev/gpt/swap1 none swap sw 0 0
tmpfs /tmp tmpfs rw 2 0
swapinfo -h will show %6 and %6 usage on the swap devices
/tmp remains < 5% used
$ grep maxswzone /boot/loader.conf
kern.maxswzone="67108864" ## double the default
$ gpart show
=> 34 286749421 da3 GPT (136G)
34 128 1 freebsd-boot (64k)
162 201326592 2 freebsd-swap (96G)
201326754 85422701 3 freebsd-zfs (40G)
=> 34 286749421 da4 GPT (136G)
34 128 1 freebsd-boot (64k)
162 201326592 2 freebsd-swap (96G)
201326754 85422701 3 freebsd-zfs (40G)
da[012] are SSDs, the rest are 15krpm
$ zpool status
pool: zmysqlD
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zmysqlD ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
da7 ONLINE 0 0 0
da8 ONLINE 0 0 0
da9 ONLINE 0 0 0
da10 ONLINE 0 0 0
da11 ONLINE 0 0 0
da12 ONLINE 0 0 0
da13 ONLINE 0 0 0
da14 ONLINE 0 0 0
logs
da0 ONLINE 0 0 0
cache
da2 ONLINE 0 0 0
errors: No known data errors
pool: zmysqlL
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zmysqlL ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da5 ONLINE 0 0 0
da6 ONLINE 0 0 0
cache
da1 ONLINE 0 0 0
errors: No known data errors
pool: zroot
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da3p3 ONLINE 0 0 0
da4p3 ONLINE 0 0 0
--
------------------------------------------------------------------------
1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C
Philip M. Gollucci (pgollu...@p6m7g8.com) c: 703.336.9354
Member, Apache Software Foundation
Committer, FreeBSD Foundation
Consultant, P6M7G8 Inc.
Director Operations, Ridecharge Inc.
Work like you don't need the money,
love like you'll never get hurt,
and dance like nobody's watching.
signature.asc
Description: OpenPGP digital signature
