Re: ZFS, compression, system load, pauses (livelocks?)
2009/12/15 Ivan Voras ivo...@freebsd.org: The context of this post is file servers running FreeBSD 8 and ZFS with compressed file systems on low-end hardware, or actually high-end hardware on VMWare ESX 3.5 and 4, which kind of makes it low-end as far as storage is concerned. The servers are standby backup mirrors of production servers - thus many writes, few reads. Running this setup I notice two things: 1) load averages get very high, though the only usage these systems get is file system usage: 2) long pauses, in what looks like vfs.zfs.txg.timeout second intervals, which seemengly block everything, or at least the entire userland. These pauses are sometimes so long that file transfers fail, which must be avoided. Looking at the list of processes it looks like a large number of kernel and userland processes are woken up at once. From the kernel side there are regularily all g_* threads, but also unrelated threads like bufdaemon, softdepflush, etc. and from the userland - top, syslog, cron, etc. It is like ZFS livelocks everything else. Any ideas on the pauses issue? Hi, I've a bit striped your post. It's kind of me too message (more details here: http://lists.freebsd.org/pipermail/freebsd-geom/2009-December/003810.html). What I've figured out so far is, that lowering the kernel thread priority (as pjd@ suggested) gives quite promising results (no livelocks at all). Though my bottleneck were caused by GELI thread. The pattern there is like this: sched_prio(curthread, PRIBIO); [...] msleep(sc, sc-sc_queue_mtx, PDROP | PRIBIO, geli:w, 0); I'm running right now with changed wersion - where I have: msleep(sc, sc-sc_queue_mtx, PDROP, geli:w, 0); So I don't change initial thread priority. It doesn't give such result as using PUSER prio, but I fear, that using PUSER may cause livelocks in some other cases. This helps my case (geli encryption and periodic locks during ZFS transaction commits) with some performance penalty, but I have similar problems in other cases. When I run: # zfs scrub tank Then kernel system process/thread consumes most of CPU (95% in system) and load rises to 20+ for the period of scrubbing. During scrub my top screen looks like: last pid: 87570; load averages: 8.26, 2.84, 1.68 199 processes: 3 running, 179 sleeping, 17 waiting CPU: 2.4% user, 0.0% nice, 97.0% system, 0.6% interrupt, 0.0% idle Mem: 66M Active, 6256K Inact, 1027M Wired, 104K Cache, 240K Buf, 839M Free Swap: 4096M Total, 4096M Free PID USERNAME THR PRI NICE SIZERES STATETIME WCPU COMMAND 0 root 69 -80 0K 544K - 104:40 67.19% kernel 24 root1 -8- 0K 8K geli:w 9:56 5.66% g_eli[0] ad6 26 root1 -8- 0K 8K geli:w 9:50 5.47% g_eli[0] ad10 25 root1 -8- 0K 8K geli:w 9:53 5.37% g_eli[0] ad8 8 root 12 -8- 0K 104K vgeom: 61:35 3.27% zfskern 3 root1 -8- 0K 8K -3:22 0.68% g_up 11 root 17 -60- 0K 136K WAIT31:21 0.29% intr Intresting thing, is that I have 17 processes waiting for CPU reported (though only intr is the only process that is reported as in WAIT state - at least for top40 processes). I just wonder, whether this might be a scheduler related issue. I'm thinking about giving a SCHED_4BSD a try. Cheers, Wiktor Niesiobędzki ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: ZFS, compression, system load, pauses (livelocks?)
2009/12/15 Ivan Voras ivo...@freebsd.org: I have tried before reducing priority of ZFS taskqueues but only to PRIBIO, not below it - not much effect wrt pauses. I was testing with getting the thread as low priority as PUSER (with original pjd@ patch) and it was actually performing better than the current solution, though I found that quite risky. Cheers, Wiktor Niesiobędzki ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: fixes for ipfw and pf lock ordering issues
On Fri, Sep 24, 2004 at 10:37:54PM +, Christian S.J. Peron wrote:
Good day folks, we need some beta testers
Hi, as an author of LOR reports I feel obliged to test this patch. I was
running it for a 2 days and intended to report, that for me everything works
ok, when an panic occured. Regretably, I do not have actual panic message, but
the trace looks as follows:
pf_socket_lookup(cbb24958,cbb2495c,2,cbb24a0c,c15275a0) at
pf_socket_lookup+0x22
pf_test_tcp(cbb249c0,cbb249bc,2,c14d6200,c139e500) at pf_test_tcp+0x648
pf_test(2,c14b8014,cbb24aa8,c15275a0,c15661c0) at pf_test+0x53d
pf_check_out(0,cbb24aa8,c14b8014,2,c15275a0) at pf_check_out+0x6d
pfil_run_hooks(c066da00,cbb24b1c,c14b8014,2,c15275a0) at pfil_run_hooks+0xeb
ip_output(c139e500,0,cbb24ae8,0,0) at ip_output+0x630
tcp_twrespond(c18709a0,10,c0607304,69c,1) at tcp_twrespond+0x1ed
tcp_twstart(c186b380,0,c0606ba2,96f,0) at tcp_twstart+0x1d3
tcp_input(c139d800,14,c14b8014,1,0) at tcp_input+0x2c39
ip_input(c139d800,0,c06053ae,e7,c066d098) at ip_input+0x5b0
netisr_processqueue(c066d098,c0642940,1,c05fb4da,c10d62c0) at
netisr_processqueu
e+0x8e
swi_net(0,0,c05f9b18,269,0) at swi_net+0xe9
ithread_loop(c10de480,cbb24d48,c05f990f,31f,100) at ithread_loop+0x172
fork_exit(c04a6520,c10de480,cbb24d48) at fork_exit+0xc6
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xcbb24d7c, ebp = 0 ---
db
db show locks
exclusive sleep mutex inp (tcpinp) r = 0 (0xc1527630) locked @
/usr/src/sys/neti
net/tcp_input.c:737
exclusive sleep mutex tcp r = 0 (0xc066de6c) locked @
/usr/src/sys/netinet/tcp_i
nput.c:611
db
(gdb) l *pf_socket_lookup+0x22
0xc043a2d2 is in pf_socket_lookup (/usr/src/sys/contrib/pf/net/pf.c:2414).
2409#endif
2410struct inpcb*inp;
2411
2412#ifdef __FreeBSD__
2413if (inp_arg != NULL) {
2414*uid = inp_arg-inp_socket-so_cred-cr_uid;
2415*gid = inp_arg-inp_socket-so_cred-cr_groups[0];
2416return (1);
2417}
2418#endif
(gdb) l *pf_test_tcp+0x648
0xc043aef8 is in pf_test_tcp (/usr/src/sys/contrib/pf/net/pf.c:2781).
2776r = TAILQ_NEXT(r, entries);
2777else if (r-rule_flag PFRULE_FRAGMENT)
2778r = TAILQ_NEXT(r, entries);
2779else if ((r-flagset th-th_flags) != r-flags)
2780r = TAILQ_NEXT(r, entries);
2781else if (r-uid.op (lookup != -1 || (lookup =
2782#ifdef __FreeBSD__
2783pf_socket_lookup(uid, gid, direction, pd, inp),
1))
2784#else
2785pf_socket_lookup(uid, gid, direction, pd), 1))
If there is anything more I may provide, please tell me. I can't get my kernel
dumps on, although I have KDB_UNATTENDED option in kernel, it gaves me prompt
on panics, and when I call panic from debugger I get hangs :S If you know any
other way to get the panic message, I'd appreciate.
My comments for the patch alone:
Before the patch, I got the LOR's and rather rare panics due to this problem.
They were happening mainly when changing PF rules, sometimes on shutdown.
After the patch, I do not have any LOR messages, I tried to load PF rules in a
loop for a few minutes. After that I just left the system for it own, while
there was some activity on network (and particularly on rules with uid
matching). Till today I was quite happy with that.
If there is anything I can debug more, to help you solve the problem, please
ask.
Cheers,
Wiktor Niesiobedzki
PS. Just for the record - I tired it only with PF. I'm also planning to give
it a shot with my old IPFW rules.
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Size of arp database.
On Tue, 19 Oct 1999, Ruslan Ermilov wrote: This is because none of your ethernet interfaces matches `pc7'. 1/ What does `route -n -v get -host pc7' show? bash-2.02# route -n -v get -host pc7 u: inet 195.117.4.106; u: link ; RTM_GET: Report Metrics: len 128, pid: 0, seq 1 , errno 0, flags:UP,GATEWAY,HOST,STATIC locks: inits: sockaddrs: DST,IFP 195.117.4.106 route to: 195.117.4.106 destination: 195.117.4.106 interface: xl0 flags: UP,HOST,DONE,STATIC recvpipe sendpipe ssthresh rtt,msecrttvar hopcount mtu expire 0 0 3000 0 188 0 1500 0 locks: inits: sockaddrs: DST,GATEWAY,IFP,IFA 195.117.4.106 0.40.95.1b.63.12 xl0:0.10.4b.36.6a.fd 195.117.4.98 bash-2.02# 2/ What is the IP address of `pc7'? 195.117.4.106 3/ How your ethernet interfaces (`ifconfig -l ether') are configured? bash-2.02# ifconfig xl0 xl0: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 inet 195.117.4.98 netmask 0xffe0 broadcast 195.117.4.127 ether 00:10:4b:36:6a:fd media: 100baseTX full-duplex supported media: autoselect 100baseTX full-duplex 100baseTX half-duplex 100baseTX 10baseT/UTP full-duplex 10baseT/UTP 10baseT/UTP half-duplex Wiktor Niesiobedzki To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Size of arp database.
Hello, Is there any way to enlarge the arp database. I've got a feeling that it is limited to only 10 enteries... For me it's a bit to less. Wiktor Niesiobedzki To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Size of arp database.
On Tue, 19 Oct 1999, Dan Nelson wrote: In the last episode (Oct 19), Wiktor said: Is there any way to enlarge the arp database. I've got a feeling that it is limited to only 10 enteries... For me it's a bit to less. $ arp -a | wc -l 256 Maybe you only have 10 machines on your network? What i tested is: root@wotan:~# arp -S pc7 00:00:E8:73:FF:FD pub delete: can't locate pc7 root@wotan:~# arp -a router (195.117.4.97) at 0:a0:c5:21:14:8 wotan.2slo.waw.pl (195.117.4.98) at 0:10:4b:36:6a:fd permanent pc2 (195.117.4.101) at 0:80:48:d7:29:be pc6 (195.117.4.105) at 52:54:0:e3:9:7a pc14 (195.117.4.113) at 0:40:f6:94:b0:ca ? (195.117.4.127) at ff:ff:ff:ff:ff:ff permanent root@wotan:~# arp -d pc7 proxy delete: can't locate pc7 root@wotan:~# arp -S pc7 00:00:E8:73:FF:FD pub delete: can't locate pc7 set: proxy entry exists for non 802 device root@wotan:~# So this entry potetnialy exist, but I'm unable to delete it... What am I doing wrong? Wiktor Niesiobedzki To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Size of arp database.
On Tue, 19 Oct 1999, Dan Nelson wrote: In the last episode (Oct 19), Wiktor said: Is there any way to enlarge the arp database. I've got a feeling that it is limited to only 10 enteries... For me it's a bit to less. $ arp -a | wc -l 256 Maybe you only have 10 machines on your network? No... the problem is, that i've got recently message: arplookup 195.117.4.106 failed: could not allocate llinfo arpresolve: can't allocate llinfo for 195.117.4.106rt when the number of running and connected machines to server reach about 10. When i tried to add manualy some entry (arp -s some.ip some:hdwr:addr) they just gone. Anyone know solution of this problem? Wiktor Niesiobedzki To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
