[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 Mark Johnston changed: What|Removed |Added Status|Open|In Progress Assignee|b...@freebsd.org|ma...@freebsd.org -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #77 from commit-h...@freebsd.org --- A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=7d1ab866911a2b29e041d64bc83a93638533f957 commit 7d1ab866911a2b29e041d64bc83a93638533f957 Author: Mark Johnston AuthorDate: 2022-03-30 19:41:44 + Commit: Mark Johnston CommitDate: 2022-03-30 19:41:44 + pf: Initialize the table entry zone limit at initialization time The limit may later be updated by the "set limit" directive in pf.conf. UMA does not permit a limit to be set on a zone after any items have been allocated from a zone. Other UMA zones used by pf do not appear to be susceptible to this problem: they either set a limit at zone creation time or never set one at all. PR: 260406 Reviewed by:kp MFC after: 3 days Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D34713 sys/netpfil/pf/pf_table.c | 1 + 1 file changed, 1 insertion(+) -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #76 from Mark Johnston --- (In reply to Mark Johnston from comment #75) The problem has to do with the reimplementation of zone limits in 13.0. pf may allocate items from the table entry zone before a limit is set, and these items won't be counted. When a limit is set, and the uncounted items are freed (as can happen if there's a memory shortage), then they end up underflowing our counter. uma_zone_set_max() is missing an assertion that would have caught this. pf also needs to set a dummy limit on the table entry zone at initialization time. I'm not sure if other pf zones might have the same problem. I'll work on a patch. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #75 from Mark Johnston --- (In reply to Mark Johnston from comment #74) Never mind, I'm able to reproduce this locally. On my test system we have: vm.uma.pf_table_entries_3.limit.max_items: 200 vm.uma.pf_table_entries_3.limit.items: 17592185933560 So we have a limit accounting bug, ok. This is on a NUMA system, let's see if it happens on a single-domain system too. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 Mark Johnston changed: What|Removed |Added Status|New |Open CC||ma...@freebsd.org --- Comment #74 from Mark Johnston --- Could anyone seeing this problem please attach output from "sysctl vm.uma.pf_table_entries" at some point after the allocation failure occurs? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #73 from Kristof Provost --- (In reply to Laurent Frigault from comment #72) No, because it's not a fix but a debugging step. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #72 from Laurent Frigault --- (In reply to Kristof Provost from comment #48) Will your patch be included in the coming release 13.1 ? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #71 from tech-li...@zyxst.net --- (In reply to Kristof Provost from comment #43) The patch here also works on AMD64 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #70 from tech-li...@zyxst.net --- I'm seeing the problem on stable/13-n249464-d0199f27c06 (AMD64) now Shall I make another ticket for AMD64 specifically? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #69 from Jean-Claude MICHOT --- (In reply to tech-lists from comment #68) 13.0-RELEASE-p4 #0: Tue Aug 24 07:33:27 UTC 2021 r...@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 running on Xeon X5670 (PowerEdge R610), with 65 GB of memory JC -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #68 from tech-li...@zyxst.net --- (In reply to Jean-Claude MICHOT from comment #67) Is this on amd64 or arm64.aarch64 hardware? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 Jean-Claude MICHOT changed: What|Removed |Added CC||j...@michot.fr --- Comment #67 from Jean-Claude MICHOT --- Same problem here with 'pfctl: Cannot allocate memory.', it's reported by fail2ban (anyway same for blacklistd). 2022-02-09 19:34:30,354 fail2ban.utils [64280]: ERROR 8021b2730 -- exec: pfctl -a f2b/ssh-pf -t f2b-ssh-pf -T add 45.9.20.25 2022-02-09 19:34:30,354 fail2ban.utils [64280]: ERROR 8021b2730 -- stderr: 'pfctl: Cannot allocate memory.' 2022-02-09 19:34:30,354 fail2ban.utils [64280]: ERROR 8021b2730 -- killed with signal 127 (return code: 255) # freebsd-version -uk 13.0-RELEASE-p4 13.0-RELEASE-p4 "pfctl -T del IP" still usable, but add new rule is impossible until reboot :( # vmstat -m | grep -E 'pf|Size' Type InUse MemUse Requests Size(s) pfs_nodes20 8K 20 384 pfs_vncache 1 128K1 pfil11 1K 11 64,128 tcpfunc 1 1K1 64 pf_temp 0 0K 55 32 pf_hash 5 11524K5 2048 pf_ifnet19 7K 171 256,2048 pf_osfp 1191 123K 3573 64,128 pf_rule 269 181K 341 128,1024 pf_table1122K24200 2048 # vmstat -z | grep pf pf mtags:48, 0, 0, 84, 55, 0, 0, 0 pf tags:104, 0, 0, 0, 0, 0, 0, 0 pf states: 296, 10, 40,2703, 4287549, 0, 0,2135254 pf state keys: 88, 0, 58,6106, 4592659, 0, 0,2280096 pf source nodes:136, 1, 0, 0, 0, 0, 0, 0 pf table entry counters: 64, 0, 0, 0, 200, 0, 0, 0 pf table entries: 160, 20, 152, 48, 488,7420, 0, 0 pf frags: 248, 0, 0, 16, 15, 0, 0, 0 pf frag entries: 40, 5000, 0, 101, 40, 0, 0, 0 pf state scrubs: 40, 0, 0, 0, 0, 0, 0, 0 # sysctl -a | grep net.pf net.pf.rule_tag_hashsize: 128 net.pf.request_maxcount: 65535 net.pf.source_nodes_hashsize: 32768 net.pf.states_hashsize: 131072 # pfctl -si Status: Enabled for 93 days 22:56:25 Debug: Urgent State Table Total Rate current entries 840 searches 5992674224 738.2/s inserts 42883560.5/s removals 42875160.5/s Counters match 1169829912 144.1/s bad-offset 00.0/s fragment 00.0/s short 00.0/s normalize 870.0/s memory 00.0/s bad-timestamp 00.0/s congestion 00.0/s ip-option 20.0/s proto-cksum00.0/s state-mismatch 12060.0/s state-insert 00.0/s state-limit00.0/s src-limit 00.0/s synproxy 00.0/s map-failed 00.0/s # top -b | head -8 last pid: 20669; load averages: 0.08, 0.11, 0.09 up 93+22:59:55 18:11:49 160 processes: 1 running, 158 sleeping, 1 zombie CPU: 0.3% user, 0.0% nice, 0.1% system, 0.0% interrupt, 99.6% idle Mem: 90M Active, 829M Inact, 506M Laundry, 59G Wired, 2173M Free ARC: 52G Total, 28G MFU, 22G MRU, 3368K Anon, 281M Header, 1419M Other 48G Compressed, 61G Uncompressed, 1.27:1 Ratio Swap: 46G Total, 1939M Used, 44G Free, 4% Inuse -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #66 from tech-li...@zyxst.net --- writing to say the pf on the rpi4 is still working without error % uptime 10:50p.m. up 12 days, 7:04, 7 users, load averages: 5.11, 4.85, 4.65 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 Kajetan Staszkiewicz changed: What|Removed |Added CC||veg...@tuxpowered.net --- Comment #65 from Kajetan Staszkiewicz --- I'm observing the same thing happening on amd64. In the beginning I thought it might be due to my own patches on pf. Then I thought it only happens on systems which are high on inactive memory (my systems behave differently regarding inactive, some have lot of free memory, some bloat the inactive memory, no idea why) which gave me the idea that maybe inactive memory can't be freed when allocating memory with M_NOWAIT. However today I finally got the same issue on a system running a nonpatched 13.0-RELEASE GENERIC kernel on a system without bloated inactive memory. [17:02:34] al-router01 ~/ # vmstat -z | grep pf pf mtags:48, 0, 0,2520,12630497, 0, 0, 0 pf tags:104, 0, 0, 0, 0, 0, 0, 0 pf states: 296, 400,7038, 39177,3739030613, 0, 0, 0 pf state keys: 88, 0,7038, 85008,3739030613, 0, 0, 0 pf source nodes:136, 40, 0, 0, 0, 0, 0, 0 pf table entry counters: 64, 0, 0, 0, 0, 0, 0, 0 pf table entries: 160, 100, 320952, 310548,259049528,180913, 0, 0 pf frags: 248, 0, 0, 96,10783117, 0, 0, 0 pf frag entries: 40, 32000, 0, 606,50033469, 0, 0, 0 pf state scrubs: 40, 0, 0, 0, 0, 0, 0, 0 So there is a limit of 1M entries, 320k are used, 310k are free. When I reload pf.conf, I need double the entries, yes? So the new pf.conf won't load. But where is the missing 370k entries? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #64 from Diego Linke --- (In reply to tech-lists from comment #62) kern.maxdsiz: 34359738368 net.pf.request_maxcount: 65535 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #63 from Kristof Provost --- (In reply to tech-lists from comment #62) net.pf.request_maxcount is not relevant for this issue. The allocation fails after that check. I'm less familiar with kern.maxdsiz, but it also appears to not be relevant to this issue. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #62 from tech-li...@zyxst.net --- (In reply to Diego Linke from comment #61) Hi, What do you have for the following sysctls: kern.maxdsiz net.pf.request_maxcount ? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #61 from Diego Linke --- (In reply to Diego Linke from comment #54) Just reporting that today I faced this issue again in another machine, after the upgrade from 12 to 13.0. Yes, this machine also doesn't have a lot of memory but I never faced this issue on 12. I think something changed in either PF or how FreeBSD 13.0 deal with memory allocation. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #60 from Diego Linke --- (In reply to Diego Linke from comment #54) Just reporting that today I faced this issue again in another machine, after the upgrade from 12 to 13.0. Yes, this machine also doesn't have a lot of memory but I never faced this issue on 12. I think something changed in either PF or how FreeBSD 13.0 deal with memory allocation. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #59 from jja...@gmail.com --- (In reply to jjasen from comment #58) This is on AMD64, FreeBSD 13.0-p4. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 jja...@gmail.com changed: What|Removed |Added CC||jja...@gmail.com --- Comment #58 from jja...@gmail.com --- I am not sure if I am fighting the same problem. See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=259798 Feel free to link the tickets if relevant. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #57 from tech-li...@zyxst.net --- Hi, Reporting that the rpi4 arm64.aarch64 is still running ok. I did however encounter a similar error with an amd64-based bhyve vm. This vm is unmodified in comparison to the arm64.aarch64 device. It runs 13.0-p5 and updates via freebsd-update. The symptoms were a bit different though, in that: 1. the script ran without error 2. no services were accessible. No ping, smtp, nothing, from anywhere. (this is the best failure mode honestly. The worst nightmare would be not-to-load-leaving-everything-open). I checked the tables for blocked IPs and made sure my accessing IP wasn't listed and it wasn't. 3. on reboot, services didn't come back. I had to access via the console. I was able to bring services back by disabling pf, updating the ruleset then re-enabling it. maybe this would need a different ticket & diags -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #56 from tech-li...@zyxst.net --- still ok # doas -u _pfbadhost pf-badhost -O freebsd Password: pf-badhost 4113 - - Using experimental "aggy" aggregator... 24504 addresses added. 4155 addresses deleted. pf-badhost 4181 - - IPv4 addresses in table: 1063826457 # pfctl -f /etc/pf.conf # pfctl -f /etc/pf.conf # service sshguard status sshguard is running as pid 3001. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #55 from tech-li...@zyxst.net --- (In reply to Kristof Provost from comment #48) Looks like your patch may have fixed the issue. I'll try making pfhost load bigger tables to make sure. So far, and it does this every 3 hrs approx: [1800hrs approx] pf-badhost 2232 - - Using experimental "aggy" aggregator... 15 addresses added. pf-badhost 2300 - - IPv4 addresses in table: 618966654 [2100hrs approx] pf-badhost 2852 - - Using experimental "aggy" aggregator... 23 addresses added. 1 addresses deleted. pf-badhost 2920 - - IPv4 addresses in table: 618966677 I'll check again later, then if all ok, make pfbadhost get more IPs -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #54 from Diego Linke --- (In reply to Kristof Provost from comment #53) Yes, I does make sense. I thought that I had free memory enough to load the tables without swap, the tables are not so big. Thank you very much I will continue to monitor it :-) -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #53 from Kristof Provost --- (In reply to Diego Linke from comment #51) > My last question is I had 79 Mb in free memory plus ~476 Mb in swap space. I > wondering why PF as the last resource doesn't use swap memory to load the > tables. Kernel memory, including that allocated by pf, will not get swapped out. We wouldn't want to have to wait to process a packet until we'd read the relevant pf table back from swap space. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #52 from tech-li...@zyxst.net --- (In reply to Kristof Provost from comment #48) ok the rpi4 is back up, with a new (current, GENERIC, so debug) kernel [...] pf-badhost 1405 - - Using experimental "aggy" aggregator... 1 table created. 15680 addresses added. pf-badhost 1473 - - IPv4 addresses in table: 618966639 # pfctl -e -f /etc/pf.conf pf enabled # service sshguard status sshguard is running as pid 1119. [...] -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #51 from Diego Linke --- (In reply to Kristof Provost from comment #50) Yes it does make sense. I just reboot the machine and the PF is working fine now. I was avoiding to make it to troubleshoot the issue better. The machine HW config was always the same, it's called my attention that this started to happen just after the upgrade to 13.0. My last question is I had 79 Mb in free memory plus ~476 Mb in swap space. I wondering why PF as the last resource doesn't use swap memory to load the tables. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #50 from Kristof Provost --- (In reply to Diego Linke from comment #49) So that looks a lot more like a machine that's at the limits of its free memory, in which case what you're seeing isn't actually an error but things working as expected. The probe exists on stable/13. Apparently not in 13.0. At this point the first thing you need to try is to run this problem on a machine with more memory. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #49 from Diego Linke --- (In reply to Kristof Provost from comment #47) Yes, this is small EC2 Virtual Machine. This used to work fine with FreeBSD 12 and pf. # top -b | head -8 last pid: 22974; load averages: 0.81, 0.81, 0.76 up 16+01:51:17 12:33:44 66 processes: 2 running, 63 sleeping, 1 zombie CPU: 0.8% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.0% idle Mem: 42M Active, 179M Inact, 15M Laundry, 609M Wired, 103M Buf, 79M Free ARC: 98M Total, 4310K MFU, 32M MRU, 1824K Anon, 2034K Header, 57M Other 7726K Compressed, 31M Uncompressed, 4.12:1 Ratio Swap: 512M Total, 38M Used, 474M Free, 7% Inuse # vmstat procsmemorypage disks faults cpu r b w avm fre flt re pi po fr sr ad0 xb5 in sy cs us sy id 1 0 0 1.0G 79M 237 1 0 0 286 39 0 0 41 377 215 1 0 99 # swapinfo Device 1K-blocks UsedAvail Capacity /dev/md99 52428839088 485200 7% Please find below the output with the latest dtrace script. I think there is a syntax error on it: # dtrace -s dtrace.script -c "pfctl -f /etc/pf.conf" dtrace: failed to compile script dtrace.script: line 1: probe description fbt:kernel:zone_alloc_limit_hard:return does not match any probes /etc/pf.conf:21: cannot define table fireholL1: Cannot allocate memory /etc/pf.conf:22: cannot define table fireholL2: Cannot allocate memory /etc/pf.conf:23: cannot define table fireholL3: Cannot allocate memory /etc/pf.conf:24: cannot define table fireholWEB: Cannot allocate memory /etc/pf.conf:25: cannot define table normshield: Cannot allocate memory /etc/pf.conf:26: cannot define table ipblacklistcloud: Cannot allocate memory /etc/pf.conf:27: cannot define table Webbots: Cannot allocate memory /etc/pf.conf:28: cannot define table haley_ssh: Cannot allocate memory /etc/pf.conf:29: cannot define table bi_any_1_7d: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #48 from Kristof Provost --- (In reply to tech-lists from comment #46) You'll also want to update pfctl. cd /usr/src/sbin/pfctl && make && sudo make install should be sufficient for that. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406
--- Comment #47 from Kristof Provost ---
(In reply to Diego Linke from comment #45)
Okay, that tells us at least that I used the entry probe in a few places where
I meant to use the return probe.
But also that it looks like in cache_alloc_retry() we failed to find a bucket
with free space, and then presumably also failed zone_alloc_item(). That still
looks for all the world like you're out of memory.
Here's an updated dtrace script:
#!/usr/sbin/dtrace -s
BEGIN
{
self->in_call = 0;
}
fbt:pf:pfr_create_kentry:entry
{
self->in_call = 1;
printf("In");
}
fbt:pf:pfr_create_kentry:return
{
self->in_call = 0;
printf("=> %x (@%x)\n", arg1, arg0);
}
fbt:pf:pfr_create_kentry:return
/ arg1 == 0 /
{
stack();
}
fbt:kernel:uma_zalloc_arg:return
/ self->in_call == 1 /
{
printf("=> %x (@%x)", arg1, arg0);
}
fbt:kernel:cache_alloc_retry:return
/ self->in_call == 1 /
{
printf("=> %x (@%x)", arg1, arg0);
}
fbt:kernel:cache_alloc:return
/ self->in_call == 1 /
{
printf("=> %x (@%x)", arg1, arg0);
}
fbt:kernel:zone_alloc_item:return
/ self->in_call == 1 /
{
printf("=> %x (@%x)", arg1, arg0);
}
fbt:kernel:uma_small_alloc:return
/ self->in_call == 1 /
{
printf("=> %x (@%x)", arg1, arg0);
}
fbt:kernel:zone_alloc_limit_hard:return
/ self->in_call == 1 /
{
printf("=> %x (@%x)", arg1, arg0);
}
fbt:kernel:zone_free_limit:entry
/ self->in_call == 1 /
{
printf("In");
}
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #46 from tech-li...@zyxst.net --- (In reply to Kristof Provost from comment #43) Thanks - tried first with manually applying, and got a build underway before i saw yr update (will a kernel build/install be sufficient?) -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #45 from Diego Linke --- (In reply to Diego Linke from comment #44) Ops, I apologize, I pasted more than necesary in the comment above. Please find the correct output. This is a FreeBSD 13 RELEASE amd64 # dtrace -s dtrace.script -c "pfctl -f /etc/pf.conf" dtrace: script 'dtrace.script' matched 9 probes /etc/pf.conf:21: cannot define table fireholL1: Cannot allocate memory /etc/pf.conf:22: cannot define table fireholL2: Cannot allocate memory /etc/pf.conf:23: cannot define table fireholL3: Cannot allocate memory /etc/pf.conf:24: cannot define table fireholWEB: Cannot allocate memory /etc/pf.conf:25: cannot define table normshield: Cannot allocate memory /etc/pf.conf:26: cannot define table ipblacklistcloud: Cannot allocate memory /etc/pf.conf:27: cannot define table Webbots: Cannot allocate memory /etc/pf.conf:28: cannot define table haley_ssh: Cannot allocate memory /etc/pf.conf:29: cannot define table bi_any_1_7d: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded CPU IDFUNCTION:NAME 0 74366 pfr_create_kentry:entry In 0 21147 cache_alloc_retry:entry => f80006ac1e00 (@f80006ac1c80) 0 21145cache_alloc:entry => f80006ac1e00 (@f80006ac1c80) 0 21202zone_alloc_item:entry => 0 (@f80006ac1c80) 0 74367 pfr_create_kentry:return => 0 (@12e) 0 74367 pfr_create_kentry:return pf.ko`pfr_ina_define+0x6d6 pf.ko`pfioctl+0x49bb kernel`devfs_ioctl+0xc7 kernel`vn_ioctl+0x1a4 kernel`devfs_ioctl_f+0x1e kernel`kern_ioctl+0x26d kernel`sys_ioctl+0xf6 kernel`amd64_syscall+0x755 kernel`0x8106227e 0 74366 pfr_create_kentry:entry In 0 21147 cache_alloc_retry:entry => f80006ac1e00 (@f80006ac1c80) 0 21145cache_alloc:entry => f80006ac1e00 (@f80006ac1c80) 0 21202zone_alloc_item:entry => 0 (@f80006ac1c80) 0 74367 pfr_create_kentry:return => 0 (@12e) 0 74367 pfr_create_kentry:return pf.ko`pfr_ina_define+0x6d6 pf.ko`pfioctl+0x49bb kernel`devfs_ioctl+0xc7 kernel`vn_ioctl+0x1a4 kernel`devfs_ioctl_f+0x1e kernel`kern_ioctl+0x26d kernel`sys_ioctl+0xf6 kernel`amd64_syscall+0x755 kernel`0x8106227e 0 74366 pfr_create_kentry:entry In 0 21147 cache_alloc_retry:entry => f80006ac1e00 (@f80006ac1c80) 0 21145cache_alloc:entry => f80006ac1e00 (@f80006ac1c80) 0 21202zone_alloc_item:entry => 0 (@f80006ac1c80) 0 74367 pfr_create_kentry:return => 0 (@12e) 0 74367 pfr_create_kentry:return pf.ko`pfr_ina_define+0x6d6 pf.ko`pfioctl+0x49bb kernel`devfs_ioctl+0xc7 kernel`vn_ioctl+0x1a4 kernel`devfs_ioctl_f+0x1e kernel`kern_ioctl+0x26d kernel`sys_ioctl+0xf6 kernel`amd64_syscall+0x755 kernel`0x8106227e 0 74366 pfr_create_kentry:entry In 0 21147 cache_alloc_retry:entry => f80006ac1e00 (@f80006ac1c80) 0 21145cache_alloc:entry => f80006ac1e00 (@f80006ac1c80) 0 21202zone_alloc_item:entry => 0 (@f80006ac1c80) 0 74367 pfr_create_kentry:return => 0 (@12e) 0 74367 pfr_create_kentry:return pf.ko`pfr_ina_define+0x6d6 pf.ko`pfioctl+0x49bb kernel`devfs_ioctl+0xc7 kernel`vn_ioctl+0x1a4 kernel`devfs_ioctl_f+0x1e kernel`kern_ioctl+0x26d kernel`sys_ioctl+0xf6 kernel`amd64_syscall+0x755 kernel`0x8106227e 0 74366 pfr_create_kentry:entry In 0 21147 cache_alloc_retry:entry => f80006ac1e00 (@f80006ac1c80) 0 21145cache_alloc:entry => f80006ac1e00 (@f80006ac1c80) 0 21202zone_alloc_item:entry => 0 (@f80006ac1c80) 0 74367 pfr_create_kentry:return => 0 (@12e) 0 74367 pfr_create_kentry:return pf.ko`pfr_ina_define+0x6d6 pf.ko`pfioctl+0x49bb kernel`devfs_ioctl+0xc7 kernel`vn_ioctl+0x1a4 kernel`devfs_ioctl_f+0x1e kernel`kern_ioctl+0x26d kernel`sys_ioctl+0xf6 kernel`amd64_syscall+0x755 kernel`0x8106227e 0 74366 pfr_create_kentry:entry In 0 21147 cache_alloc_retry:entry => f80006ac1e00 (@f80006ac1c80) 0 21145cache_alloc:entry => f80006ac1e00 (@f80006ac1c
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #44 from Diego Linke --- (In reply to Kristof Provost from comment #40) Hi, please find below the dtrace output, I hope it will be useful. # dtrace -s dtrace_pf.sh -c "pfctl -f /etc/pf.conf" dtrace: script 'dtrace_pf.sh' matched 9 probes /etc/pf.conf:21: cannot define table fireholL1: Cannot allocate memory /etc/pf.conf:22: cannot define table fireholL2: Cannot allocate memory /etc/pf.conf:23: cannot define table fireholL3: Cannot allocate memory /etc/pf.conf:24: cannot define table fireholWEB: Cannot allocate memory /etc/pf.conf:25: cannot define table normshield: Cannot allocate memory /etc/pf.conf:26: cannot define table ipblacklistcloud: Cannot allocate memory /etc/pf.conf:27: cannot define table Webbots: Cannot allocate memory /etc/pf.conf:28: cannot define table haley_ssh: Cannot allocate memory /etc/pf.conf:29: cannot define table bi_any_1_7d: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded dtrace: pid 56813 exited with status 1 CPU IDFUNCTION:NAME 0 74366 pfr_create_kentry:entry In 0 21147 cache_alloc_retry:entry => f80006ac1e00 (@f80006ac1c80) 0 21145cache_alloc:entry => f80006ac1e00 (@f80006ac1c80) 0 21202zone_alloc_item:entry => 0 (@f80006ac1c80) 0 74367 pfr_create_kentry:return => 0 (@12e) 0 74367 pfr_create_kentry:return pf.ko`pfr_ina_define+0x6d6 pf.ko`pfioctl+0x49bb kernel`devfs_ioctl+0xc7 kernel`vn_ioctl+0x1a4 kernel`devfs_ioctl_f+0x1e kernel`kern_ioctl+0x26d kernel`sys_ioctl+0xf6 kernel`amd64_syscall+0x755 kernel`0x8106227e 0 74366 pfr_create_kentry:entry In 0 21147 cache_alloc_retry:entry => f80006ac1e00 (@f80006ac1c80) 0 21145cache_alloc:entry => f80006ac1e00 (@f80006ac1c80) 0 21202zone_alloc_item:entry => 0 (@f80006ac1c80) 0 74367 pfr_create_kentry:return => 0 (@12e) 0 74367 pfr_create_kentry:return pf.ko`pfr_ina_define+0x6d6 pf.ko`pfioctl+0x49bb kernel`devfs_ioctl+0xc7 kernel`vn_ioctl+0x1a4 kernel`devfs_ioctl_f+0x1e kernel`kern_ioctl+0x26d kernel`sys_ioctl+0xf6 kernel`amd64_syscall+0x755 kernel`0x8106227e 0 74366 pfr_create_kentry:entry In 0 21147 cache_alloc_retry:entry => f80006ac1e00 (@f80006ac1c80) 0 21145cache_alloc:entry => f80006ac1e00 (@f80006ac1c80) 0 21202zone_alloc_item:entry => 0 (@f80006ac1c80) 0 74367 pfr_create_kentry:return => 0 (@12e) 0 74367 pfr_create_kentry:return pf.ko`pfr_ina_define+0x6d6 pf.ko`pfioctl+0x49bb kernel`devfs_ioctl+0xc7 kernel`vn_ioctl+0x1a4 kernel`devfs_ioctl_f+0x1e kernel`kern_ioctl+0x26d kernel`sys_ioctl+0xf6 kernel`amd64_syscall+0x755 kernel`0x8106227e 0 74366 pfr_create_kentry:entry In 0 21147 cache_alloc_retry:entry => f80006ac1e00 (@f80006ac1c80) 0 21145cache_alloc:entry => f80006ac1e00 (@f80006ac1c80) 0 21202zone_alloc_item:entry => 0 (@f80006ac1c80) 0 74367 pfr_create_kentry:return => 0 (@12e) 0 74367 pfr_create_kentry:return pf.ko`pfr_ina_define+0x6d6 pf.ko`pfioctl+0x49bb kernel`devfs_ioctl+0xc7 kernel`vn_ioctl+0x1a4 kernel`devfs_ioctl_f+0x1e kernel`kern_ioctl+0x26d kernel`sys_ioctl+0xf6 kernel`amd64_syscall+0x755 kernel`0x8106227e 0 74366 pfr_create_kentry:entry In 0 21147 cache_alloc_retry:entry => f80006ac1e00 (@f80006ac1c80) 0 21145cache_alloc:entry => f80006ac1e00 (@f80006ac1c80) 0 21202zone_alloc_item:entry => 0 (@f80006ac1c80) 0 74367 pfr_create_kentry:return => 0 (@12e) 0 74367 pfr_create_kentry:return pf.ko`pfr_ina_define+0x6d6 pf.ko`pfioctl+0x49bb kernel`devfs_ioctl+0xc7 kernel`vn_ioctl+0x1a4 kernel`devfs_ioctl_f+0x1e kernel`kern_ioctl+0x26d kernel`sys_ioctl+0xf6 kernel`amd64_syscall+0x755 kernel`0x8106227e 0 74366 pfr_create_kentry:entry In 0 21147 cache_alloc_retry:entry => f80006ac1e00 (@f80006ac1c80) 0 21145cache_alloc:entry => f80006ac1e00 (@f80006ac1c80) 0 21202zon
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #43 from Kristof Provost --- Created attachment 230375 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=230375&action=edit no table limit patch This is the same patch, but not mangled by being copy/pasted into and out of the bugzilla comment field. It's also trivial to apply it manually, because it's a very simple patch. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #42 from tech-li...@zyxst.net --- i'll try just manually editing it -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #41 from tech-li...@zyxst.net --- (In reply to Kristof Provost from comment #40) The patch doesn't apply: # patch < pfpatch2.patch Hmm... Looks like a unified diff to me... The text leading up to this was: -- |diff --git a/sys/netpfil/pf/pf_table.c b/sys/netpfil/pf/pf_table.c |index 4cfe5d61e83e..859d5ad79775 100644 |--- a/sys/netpfil/pf/pf_table.c |+++ b/sys/netpfil/pf/pf_table.c -- Patching file sys/netpfil/pf/pf_table.c using Plan A... patch: malformed patch at line 12: diff --git a/sbin/pfctl/pfctl.c b/sbin/pfctl/pfctl.c Looks like it might need a newline between the first patch and the second, so tried that: # patch < pfpatch2.patch Hmm... Looks like a unified diff to me... The text leading up to this was: -- |diff --git a/sys/netpfil/pf/pf_table.c b/sys/netpfil/pf/pf_table.c |index 4cfe5d61e83e..859d5ad79775 100644 |--- a/sys/netpfil/pf/pf_table.c |+++ b/sys/netpfil/pf/pf_table.c -- Patching file sys/netpfil/pf/pf_table.c using Plan A... Hunk #1 failed at 209. 1 out of 1 hunks failed--saving rejects to sys/netpfil/pf/pf_table.c.rej Hmm... The next patch looks like a unified diff to me... The text leading up to this was: -- |diff --git a/sbin/pfctl/pfctl.c b/sbin/pfctl/pfctl.c |index a0eec1b09289..22c689934c2d 100644 |--- a/sbin/pfctl/pfctl.c |+++ b/sbin/pfctl/pfctl.c -- Patching file sbin/pfctl/pfctl.c using Plan A... Hunk #1 failed at 1918. 1 out of 1 hunks failed--saving rejects to sbin/pfctl/pfctl.c.rej Hmm... Ignoring the trailing garbage. done This is on: % gvs srctree last updated: Fri Dec 24 11:59:43 2021 + version: 251915 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406
--- Comment #40 from Kristof Provost ---
(In reply to tech-lists from comment #39)
Yes please. Let's see if that makes a difference or not.
While we're at it, it'd be useful for someone who has an affected amd64 machine
to try this dtrace script:
#!/usr/sbin/dtrace -s
BEGIN
{
self->in_call = 0;
}
fbt:pf:pfr_create_kentry:entry
{
self->in_call = 1;
printf("In");
}
fbt:pf:pfr_create_kentry:return
{
self->in_call = 0;
printf("=> %x (@%x)\n", arg1, arg0);
}
fbt:pf:pfr_create_kentry:return
/ arg1 == 0 /
{
stack();
}
fbt:kernel:uma_zalloc_arg:return
/ self->in_call == 1 /
{
printf("=> %x (@%x)", arg1, arg0);
}
fbt:kernel:cache_alloc_retry:entry
/ self->in_call == 1 /
{
printf("=> %x (@%x)", arg1, arg0);
}
fbt:kernel:cache_alloc:entry
/ self->in_call == 1 /
{
printf("=> %x (@%x)", arg1, arg0);
}
fbt:kernel:zone_alloc_item:entry
/ self->in_call == 1 /
{
printf("=> %x (@%x)", arg1, arg0);
}
fbt:kernel:uma_small_alloc:return
/ self->in_call == 1 /
{
printf("=> %x (@%x)", arg1, arg0);
}
Once the problem occurs there should be output like 'pfr_create_kentry:return
=> 0 (@120)'. Please attach the full log, because there's going to be more
information (hopefully!) in there.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #39 from tech-li...@zyxst.net --- (In reply to Kristof Provost from comment #35) Would you like me to apply the patch and reboot? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #38 from Kristof Provost --- (In reply to Andriy Gapon from comment #37) Realistically no. It's pretty deep down the code path and would massively complicate dealing with the ioctl. It would also be a largely pointless exercise, because we do a lot of uma_zalloc(M_NOWAIT) in the data path as well (e.g. for fragment handling, or to tag packets or for state creation). -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #37 from Andriy Gapon --- (In reply to Kristof Provost from comment #36) Could the allocation be done in advance, before the lock is taken? If that's technically messy to do, then I recall that UMA has some sort of a reservation mechanism. But, actually, not sure if it's suitable for the pf's use case. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #36 from Kristof Provost --- (In reply to Andriy Gapon from comment #34) The allocation is done with the PF_RULES write lock held, and that's not a sleepable lock. We need to hold the read lock for rules evaluation, so sleeping with the write lock held would just halt traffic to/from the machine. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406
--- Comment #35 from Kristof Provost ---
(In reply to Andriy Gapon from comment #31)
This should prevent the table limit from being set, so it's a little narrower
than just disabling the limit for everything:
diff --git a/sys/netpfil/pf/pf_table.c b/sys/netpfil/pf/pf_table.c
index 4cfe5d61e83e..859d5ad79775 100644
--- a/sys/netpfil/pf/pf_table.c
+++ b/sys/netpfil/pf/pf_table.c
@@ -209,7 +209,6 @@ pfr_initialize(void)
V_pfr_kentry_z = uma_zcreate("pf table entries",
sizeof(struct pfr_kentry), NULL, NULL, NULL, NULL, UMA_ALIGN_PTR,
0);
- V_pf_limits[PF_LIMIT_TABLE_ENTRIES].zone = V_pfr_kentry_z;
V_pf_limits[PF_LIMIT_TABLE_ENTRIES].limit = PFR_KENTRY_HIWAT;
}
diff --git a/sbin/pfctl/pfctl.c b/sbin/pfctl/pfctl.c
index a0eec1b09289..22c689934c2d 100644
--- a/sbin/pfctl/pfctl.c
+++ b/sbin/pfctl/pfctl.c
@@ -1918,6 +1918,9 @@ pfctl_load_limit(struct pfctl *pf, unsigned int index,
unsigned int limit)
{
struct pfioc_limit pl;
+ if (index == PF_LIMIT_TABLE_ENTRIES)
+ return (0);
+
memset(&pl, 0, sizeof(pl));
pl.index = index;
pl.limit = limit;
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #34 from Andriy Gapon --- (In reply to Kristof Provost from comment #27) Do memory allocations need to be done with M_NOWAIT if they are performed in a context of a call (ioctl) from userland? I think that it's not impossible that there could be some transient condition (plus maybe a bug) resulting in M_NOWAIT giving up too easily. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #33 from Andriy Gapon --- (In reply to tech-lists from comment #32) Comment out all calls to uma_zone_set_max() under sys/netpfil/pf. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #32 from tech-li...@zyxst.net --- (In reply to Andriy Gapon from comment #31) How could that be done? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #31 from Andriy Gapon --- I wonder whether it would help if --purely as a hack / experiment -- the limits on uma zones that pf uses were removed. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #30 from Diego Linke --- (In reply to Kristof Provost from comment #29) Yes, I did. Please see below: # vmstat -z | grep -E 'pf|ITEM' ITEM SIZE LIMIT USED FREE REQ FAILSLEEP XDOMAIN pf mtags:48, 0, 0, 84,2057, 0, 0, 0 pf tags:104, 0, 0, 0, 0, 0, 0, 0 pf states: 296, 10, 29, 153, 653447, 0, 0, 0 pf state keys: 88, 0, 29, 339, 653447, 0, 0, 0 pf source nodes:136, 1, 6, 23, 29737, 0, 0, 0 pf table entry counters: 64, 0, 0, 0, 0, 0, 0, 0 pf table entries: 160, 100, 1, 24,57544806,15982, 0, 0 pf frags: 248, 0, 0, 0, 0, 0, 0, 0 pf frag entries: 40, 5000, 0, 0, 0, 0, 0, 0 pf state scrubs: 40, 0, 0, 0, 0, 0, 0, 0 # vmstat -m | grep -E 'pf|Size' Type InUse MemUse Requests Size(s) pfs_nodes20 8K 20 384 pfs_vncache 1 8K1 8192 pfil11 1K 11 64,128 tcpfunc 1 1K1 64 pf_temp 0 0K 2323 32,64 pf_hash 5 11524K5 2048 pf_ifnet21 6K 80 128,256,2048 pf_osfp 1191 123K 2382 64,128 pf_rule4444K 44 1024 pf_table1632K28317 2048 # sysctl net.pf.request_maxcount net.pf.request_maxcount: 65535 # sysctl net.pf.request_maxcount=5080 net.pf.request_maxcount: 65535 -> 5080 # pfctl -f /etc/pf.conf /etc/pf.conf:21: cannot define table fireholL1: Cannot allocate memory /etc/pf.conf:22: cannot define table fireholL2: Cannot allocate memory /etc/pf.conf:23: cannot define table fireholL3: Cannot allocate memory /etc/pf.conf:24: cannot define table fireholWEB: Cannot allocate memory /etc/pf.conf:25: cannot define table normshield: Cannot allocate memory /etc/pf.conf:26: cannot define table ipblacklistcloud: Cannot allocate memory /etc/pf.conf:27: cannot define table Webbots: Cannot allocate memory /etc/pf.conf:28: cannot define table haley_ssh: Cannot allocate memory /etc/pf.conf:29: cannot define table bi_any_1_7d: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #29 from Kristof Provost --- (In reply to Diego Linke from comment #28) Have you checked the vmstat -z output as well? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #28 from Diego Linke --- I don't think it's an exclusive issue of arm64 platform. I am facing exactly this issue on a 13.0-RELEASE-p3 running at AWS EC2 instance in an amd64 i386 platform. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406
--- Comment #27 from Kristof Provost ---
Summarising some private debugging:
The "pfctl: Cannot allocate memory." originates in a failed uma_zalloc() in
pfr_create_kentry() on the V_pf_kentry_z ("pf table entries") zone. Confirmed
by dtrace (which also shows unexplained issues, with some probes not firing
when they should).
There's no obvious reason why this allocation should fail. There's sufficient
free memory and we're far away from the configured limit. The current suspicion
is a platform specific (this appears to only manifest on arm64) allocator
issue, but combined with the dtrace issues it's hard to say.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #26 from Kristof Provost --- I'll want more testing later, but I need to do some digging first and won't have time to do that today. Go ahead and restart it. It seems to occur sufficiently frequently that it shouldn't take very long to do further testing when I know what to check for. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #25 from tech-li...@zyxst.net --- can I restart it or would you like me to do some more testing? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #24 from tech-li...@zyxst.net --- (In reply to Kristof Provost from comment #23) it's gone up to 8 so far: pf table entries: 160, 2540, 24700,5400, 58807, 8, 0, 0 i'll run the script again: pf table entries: 160, 2540, 24700,5400, 58807, 9, 0, 0 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #23 from Kristof Provost --- (In reply to tech-lists from comment #22) Yeah, I'm aware of the consequences of this allocation failure. I'm just struggling to work out why it's failing to allocate. There's no obvious reason for it, and it's clearly failing in the memory allocator, not pf itself. Let's at least confirm expectations. Does the number of failed allocation attempts in `vmstat -z` for pf table entries (at 4 when you reported it) increase on every failed attempt to update a table? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #22 from tech-li...@zyxst.net --- I cannot add any ip to any table while it's like this. This means programs like sshguard (which monitors auth.log) cannot add ip addresses to its table on-the-fly, which means they are no longer functional: # service sshguard status sshguard is running as pid 1293. # pfctl -t sshguard -T add 1.0.0.1 pfctl: Cannot allocate memory. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #21 from tech-li...@zyxst.net --- maybe worthwhile to mention I have in /boot/loader.conf: kern.maxdsiz=4294967296 this was much smaller by default on this arch than expected when I set this value. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #20 from tech-li...@zyxst.net --- (In reply to Kristof Provost from comment #19) > The only other reason I can see for it is that we're failing to find free > memory to allocate from. So we're still at "Your system is out of memory" > rather than anything else. I've not rebooted the system yet. here's top output: last pid: 50141; load averages: 0.81, 0.35, 0.27 up 0+14:30:53 14:49:30 67 processes: 1 running, 66 sleeping CPU: 0.0% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.8% idle Mem: 9252K Active, 44M Inact, 139M Laundry, 3161M Wired, 40K Buf, 4461M Free ARC: 1352M Total, 507M MFU, 353M MRU, 132K Anon, 8512K Header, 483M Other 507M Compressed, 740M Uncompressed, 1.46:1 Ratio Swap: 12G Total, 12G Free ran the script again and it failed again. top output after this: last pid: 50441; load averages: 0.18, 0.26, 0.25 up 0+14:37:06 14:55:43 28 processes: 1 running, 27 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 12M Active, 45M Inact, 139M Laundry, 3122M Wired, 40K Buf, 4494M Free ARC: 1352M Total, 508M MFU, 353M MRU, 8516K Header, 483M Other 507M Compressed, 741M Uncompressed, 1.46:1 Ratio Swap: 12G Total, 12G Free Why would it report out-of-memory with over 4GB available? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #19 from Kristof Provost --- (In reply to tech-lists from comment #18) > pf table entries: 160, 2540, 24700,5400, 58807, 4, 0, > 0 Okay, so that's the relevant line. Limit of 2540, 24700 used, 5400 free. We also see 4 failed, which matches the failures you're seeing, but I'm not clear on why we'd get allocation failures when we're so far from the limit. That can happen if the constructor fails, but there's no constructor for pf table entries, so that's not it. The only other reason I can see for it is that we're failing to find free memory to allocate from. So we're still at "Your system is out of memory" rather than anything else. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #18 from tech-li...@zyxst.net --- # vmstat -m | grep -E 'pf|Size' Type InUse MemUse Requests Size(s) pfil11 1K 11 64,128 tcpfunc 1 1K1 64 pf_hash 5 11524K5 2048 pf_ifnet 7 4K 40 256,2048 pf_rule1938K 38 2048 pf_osfp 1191 123K 2382 64,128 pf_table 612K 27 2048 pfs_nodes20 8K 20 384 pfs_vncache 116K 46 32,16384 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #17 from tech-li...@zyxst.net --- might be of use: # pfctl -si Status: Enabled for 0 days 11:52:43 Debug: Urgent State Table Total Rate current entries 25 searches 1282643 30.0/s inserts140950.3/s removals 140700.3/s Counters match 210540.5/s bad-offset 00.0/s fragment 00.0/s short 00.0/s normalize 00.0/s memory 00.0/s bad-timestamp 00.0/s congestion 00.0/s ip-option 00.0/s proto-cksum00.0/s state-mismatch 00.0/s state-insert 00.0/s state-limit00.0/s src-limit 00.0/s synproxy 00.0/s map-failed 00.0/s -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #16 from tech-li...@zyxst.net --- (In reply to Kristof Provost from comment #14) ok it's failing now: # vmstat -z | grep pf pf mtags:48, 0, 0, 0, 0, 0, 0, 0 pf tags:104, 0, 0, 0, 0, 0, 0, 0 pf states: 312, 10, 22, 212, 13790, 0, 0, 0 pf state keys: 88, 0, 22, 760, 13790, 0, 0, 0 pf source nodes:136, 1, 0, 0, 0, 0, 0, 0 pf table entry counters: 64, 0, 0, 0, 0, 0, 0, 0 pf table entries: 160, 2540, 24700,5400, 58807, 3, 0, 0 pf frags: 248, 0, 0, 0, 0, 0, 0, 0 pf frag entries: 40, 5000, 0, 0, 0, 0, 0, 0 this is with: in /etc/sysctl.conf: # net.pf.request_maxcount: 2540 and in /etc/pf.conf: set limit table-entries 2540 so, doubling it now: sysctl net.pf.request_maxcount=5080 net.pf.request_maxcount: 2540 -> 5080 and in /etc/pf.conf set limit table-entries 5080 pfctl: Cannot allocate memory. # vmstat -z | grep pf pf mtags:48, 0, 0, 0, 0, 0, 0, 0 pf tags:104, 0, 0, 0, 0, 0, 0, 0 pf states: 312, 10, 24, 210, 14066, 0, 0, 0 pf state keys: 88, 0, 24, 758, 14066, 0, 0, 0 pf source nodes:136, 1, 0, 0, 0, 0, 0, 0 pf table entry counters: 64, 0, 0, 0, 0, 0, 0, 0 pf table entries: 160, 2540, 24700,5400, 58807, 4, 0, 0 pf frags: 248, 0, 0, 0, 0, 0, 0, 0 pf frag entries: 40, 5000, 0, 0, 0, 0, 0, 0 pf state scrubs: 40, 0, 0, 0, 0, 0, 0, 0 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #15 from tech-li...@zyxst.net --- this might be a while as the time taken for it to fail has been variable -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #14 from Kristof Provost --- (In reply to tech-lists from comment #12) Okay, so we're failing allocation in pfr_create_kentry(), which means either the V_pfr_kentry_z or V_pfr_kentry_counter_z zones. After a failure take `vmstat -z | grep pf`, and see if bumping the `set limit table-entries` number helps. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #13 from tech-li...@zyxst.net --- I'm not sure if the following information is of any use, but thought I'd mention it in case it is: 1. with more or less the same config, but configured to block many more addresses (so the table is many times larger i guess), but on amd64 (freebsd bhyve guest) running 13.0-p5, the command runs without error: pf-badhost 87438 - - Using experimental "aggy" aggregator... 2182 addresses added. 2854 addresses deleted. pf-badhost 87506 - - IPv4 addresses in table: 1044433099 but in dmesg there is the message that pf states limit has been reached. The vm still works/passes traffic though. The vm has 8GB vram. I've not tried increasing that yet. here is pfctl -si # pfctl -si Status: Enabled for 38 days 17:47:57 Debug: Urgent Interface Stats for vtnet0IPv4 IPv6 Bytes In 28726643710 Bytes Out 16163488650 Packets In Passed 55736030 Blocked 4140060 Packets Out Passed 65927350 Blocked 135590 State Table Total Rate current entries 10 searches125938973.8/s inserts 8630910.3/s removals 8630810.3/s Counters match12907490.4/s bad-offset 00.0/s fragment 00.0/s short 00.0/s normalize 60.0/s memory 134710.0/s bad-timestamp 00.0/s congestion 00.0/s ip-option 00.0/s proto-cksum00.0/s state-mismatch140.0/s state-insert 00.0/s state-limit00.0/s src-limit 00.0/s synproxy 00.0/s map-failed 00.0/s -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #12 from tech-li...@zyxst.net --- after reboot, before loading pf: # doas -u _pfbadhost pf-badhost -O freebsd Password: pf-badhost 1594 - - Using experimental "aggy" aggregator... 1 table created. 17842 addresses added. pf-badhost 1662 - - IPv4 addresses in table: 619194384 [dtrace] loads of this, maybe over a thousand lines. it fills the standard screen(8) scroll-back buffer: 1 13259 pfr_create_kentry:return a00150ffb3c0 1 13259 pfr_create_kentry:return a00150ffb320 1 13259 pfr_create_kentry:return a00150ffb280 [...] 1 13259 pfr_create_kentry:return a00158828780 1 36465 pf_find_or_create_kruleset:return 41aef0e0 1 36465 pf_find_or_create_kruleset:return 41aef0e0 1 13251 pfr_create_ktable:return 1 1 13251 pfr_create_ktable:return a00158826000 1 13251 pfr_create_ktable:return 1 1 13251 pfr_create_ktable:return a00158827800 1 13259 pfr_create_kentry:return a001588286e0 1 13259 pfr_create_kentry:return a00158828640 1 13259 pfr_create_kentry:return a001588285a0 (ends) but the rules load successfully. later on when the cronjob runs to refresh the table, it'll succeed once maybe twice then pf will bail. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #11 from tech-li...@zyxst.net --- this time, I couldn't reload pf even with the big table commented out again. it won't load any tables at all. I'll have to restart the machine. # pfctl -f /etc/pf.conf /etc/pf.conf:23: cannot define table rfc6890: Cannot allocate memory /etc/pf.conf:25: cannot define table gooDNS4: Cannot allocate memory /etc/pf.conf:26: cannot define table gooDNS6: Cannot allocate memory /etc/pf.conf:27: cannot define table friends: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded I had to kill -9 the dtrace to run it again: CPU IDFUNCTION:NAME 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 13251 pfr_create_ktable:return 1 0 13251 pfr_create_ktable:return a00116984000 0 13251 pfr_create_ktable:return 1 0 13251 pfr_create_ktable:return a0007d6b7800 0 13259 pfr_create_kentry:return 0 0 13251 pfr_create_ktable:return 1 0 13251 pfr_create_ktable:return a0007d6b7800 0 13251 pfr_create_ktable:return 1 0 13251 pfr_create_ktable:return a000957a3800 0 13259 pfr_create_kentry:return 0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 36465 pf_find_or_create_kruleset:return 41aef0e0 0 13251 pfr_create_ktable:return 1 0 13251 pfr_create_ktable:return a000957a3800 0 13251 pfr_create_ktable:return 1 0 13251 pfr_create_ktable:return a0001a8a4000 0 13259 pfr_create_kentry:return 0 0 13251 pfr_create_ktable:return 1 0 13251 pfr_create_ktable:return a000957a3800 0 13259 pfr_create_kentry:return 0 (re-ran the pfctl command again) 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 13251 pfr_create_ktable:return 1 2 13251 pfr_create_ktable:return a00080bc2000 2 13251 pfr_create_ktable:return 1 2 13251 pfr_create_ktable:return a000d24d6000 2 13259 pfr_create_kentry:return 0 2 13251 pfr_create_ktable:return 1 2 13251 pfr_create_ktable:return a000d24d6000 2 13251 pfr_create_ktable:return 1 2 13251 pfr_create_ktable:return a00116976800 2 13259 pfr_create_kentry:return 0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 36465 pf_find_or_create_kruleset:return 41aef0e0 2 13251 pfr_create_ktable:return 1 2 13251 pfr_create_ktable:return a00116976800 2 13251 pfr_create_ktable:return 1 2 13251 pfr_create_ktable:return a0001a83d000 2 13259 pfr_create_kentry:return 0 2 13251 pfr_create_ktable:return 1 2 13251 pfr_create_ktable:return a00116976800 2 13259 pfr_create_kentry:return 0 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406
--- Comment #10 from tech-li...@zyxst.net ---
OK - two tests, first one with the large table commented out (so pfctl -f
returns without error), second one with it uncommented
1.
# dtrace -n 'fbt:kernel:pfr_create_ktable:return { printf("%x", arg0); }' -n
'fbt:kernel:pf_find_or_create_kruleset:return { printf("%x", arg0); }' -n
'fbt:kernel:pfr_create_kentry:return { printf("%x", arg0); }'
dtrace: description 'fbt:kernel:pfr_create_ktable:return ' matched 1 probe
dtrace: description 'fbt:kernel:pf_find_or_create_kruleset:return ' matched 1
probe
dtrace: description 'fbt:kernel:pfr_create_kentry:return ' matched 1 probe
CPU IDFUNCTION:NAME
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 13251 pfr_create_ktable:return 1
1 13251 pfr_create_ktable:return a000133a6000
1 13251 pfr_create_ktable:return 1
1 13251 pfr_create_ktable:return a00080bc2000
1 13259 pfr_create_kentry:return a0011698b1e0
1 13259 pfr_create_kentry:return a0011698b280
1 13259 pfr_create_kentry:return a0011698b320
1 13259 pfr_create_kentry:return a0011698b3c0
1 13259 pfr_create_kentry:return a0011698b460
1 13259 pfr_create_kentry:return a0011698b500
1 13259 pfr_create_kentry:return a0011698b5a0
1 13259 pfr_create_kentry:return a0011698b640
1 13259 pfr_create_kentry:return a0011698b6e0
1 13259 pfr_create_kentry:return a0011698b780
1 13259 pfr_create_kentry:return a0011698b820
1 13259 pfr_create_kentry:return a0011698b8c0
1 13259 pfr_create_kentry:return a0011698b960
1 13259 pfr_create_kentry:return a0011698ba00
1 13259 pfr_create_kentry:return a0011698baa0
1 13259 pfr_create_kentry:return a0011698bb40
1 13251 pfr_create_ktable:return 1
1 13251 pfr_create_ktable:return a001d822a000
1 13251 pfr_create_ktable:return 1
1 13251 pfr_create_ktable:return a0004a9b1800
1 13259 pfr_create_kentry:return a0011698bbe0
1 13259 pfr_create_kentry:return a0011698bc80
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 36465 pf_find_or_create_kruleset:return 41aef0e0
1 13251 pfr_create_ktable:return 1
1 13251 pfr_create_ktable:return a001d71a2800
1 13251 pfr_create_ktable:return 1
1 13251 pfr_create_ktable:return a001d71a2000
1 13259 pfr_create_kentry:return a0011698bd20
1 13259 pfr_create_kentry:return a0011698bdc0
1 13251 pfr_create_ktable:return 1
1 13251 pfr_create_ktable:return a00108f7e800
1 13259 pfr_create_kentry:return a0011698be60
1 13259 pfr_create_kentry:return a0011698bf00
1 13259 pfr_create_kentry:return a0011698a000
2.
pfctl -f /etc/pf.conf with the large table enabled, so
# pfctl -f /etc/pf.conf
/etc/pf.conf:18: cannot define table pfbadhost: Cannot allocate memory
pfctl: Syntax error in config file: pf rules not loaded
# dtrace -n 'fbt:kernel:pfr_create_ktable:return { printf("%x", arg0); }' -n
'fbt:kernel:pf_find_or_create_kruleset:return { printf("%x", arg0); }' -n
'fbt:kernel:pfr_create_kentry:return { printf("%x", arg0); }'
dtrace: description 'fbt:kernel:pfr_create_ktable:return ' matched 1 probe
dtrace: description 'fbt:kernel:pf_find_or_create_kruleset:return ' matched 1
probe
dtrace: description 'fbt:kernel:pfr_create_kentry:return ' matched 1 probe
CPU IDFUNCTION:NAME
2 36465 pf_find_or_create_kruleset:return 41aef0e0
2 36465 pf_find_or_create_kruleset:return 41aef0e0
2 36465 pf_find_or_create_kruleset:return 41aef0e0
2 36465 pf_find_or_create_kruleset:return 41aef0e0
2 36465 pf_find_or_create_kruleset:return 41aef0e0
2 36465 pf_find_or_create_kruleset:return 41aef0e0
2 36465 pf_find_or_create_kruleset:return 41aef0e0
2 36465 pf_find_or_create_kruleset:return 41aef0e0
2 36465 pf_find_or_create_kruleset:return 0
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406
--- Comment #9 from Kristof Provost ---
> 2 36537pfr_ina_define:return c a000c1e48be0
> 2 36537pfr_ina_define:return 0 0
What? That's ... that can't be right. The first number should be the offset in
the function where we returned from. Both 0xc and 0x0 make no sense there. The
second value is the return value. That should be zero or an error number. Not a
pointer like we're clearly getting here.
Sigh, it looks like Dtrace's fbt is buggy on aarch64, and indeed if I'm reading
sys/cddl/dev/fbt/aarch64/fbt_isa.c correctly we're passing x[0] / x[1] in the
return probe, rather than the return offset and the return value. So we should
look for the return value in arg0, and indeed we see a return value of 0x0c (or
12, ENOMEM).
So we now know at least that the big allocation succeeds, but that
pfr_ina_define() or one of the functions it calls fail to allocate memory.
Let's dig a bit more.
Try:
dtrace -n 'fbt:kernel:pfr_create_ktable:return { printf("%x", arg0); }' -n
'fbt:kernel:pf_find_or_create_kruleset:return { printf("%x", arg0); }' -n
'fbt:kernel:pfr_create_kentry:return { printf("%x", arg0); }'
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #8 from tech-li...@zyxst.net --- should be "block" rather than "clock" -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406
--- Comment #7 from tech-li...@zyxst.net ---
(In reply to Kristof Provost from comment #6)
# dtrace -n 'fbt:kernel:pfr_ina_define:return { printf("%x %x", arg0, arg1); }'
dtrace: description 'fbt:kernel:pfr_ina_define:return ' matched 1 probe
CPU IDFUNCTION:NAME
0 36537pfr_ina_define:return c a000c1e55460
0 36537pfr_ina_define:return 0 0
0 36537pfr_ina_define:return 0 a001d5139d80
0 36537pfr_ina_define:return 0 0
0 36537pfr_ina_define:return 0 a001977e9000
0 36537pfr_ina_define:return 0 a0001a8a3401
0 36537pfr_ina_define:return 0 a0016fb04a80
if I try the command (pfctl -f /etc/pf.conf) repeatedly the values of the probe
change a bit:
2 36537pfr_ina_define:return c a000c1e48be0
2 36537pfr_ina_define:return 0 0
2 36537pfr_ina_define:return 0 a0017783fa80
2 36537pfr_ina_define:return 0 0
2 36537pfr_ina_define:return 0 a001df262a80
2 36537pfr_ina_define:return 0 a0001a8a3401
2 36537pfr_ina_define:return 0 a0014e050a80
each time in the pf window I get this result:
# pfctl -f /etc/pf.conf
/etc/pf.conf:18: cannot define table pfbadhost: Cannot allocate memory
pfctl: Syntax error in config file: pf rules not loaded
if I comment out the big table and its clock rules from pf.conf, it loads
without error:
# pfctl -f /etc/pf.conf
#
in the probe window:
# dtrace -n 'fbt:kernel:pfr_ina_define:return { printf("%x %x", arg0, arg1); }'
dtrace: description 'fbt:kernel:pfr_ina_define:return ' matched 1 probe
CPU IDFUNCTION:NAME
0 36537pfr_ina_define:return 0 0
0 36537pfr_ina_define:return 0 a00184267780
0 36537pfr_ina_define:return 0 0
0 36537pfr_ina_define:return 0 a001c64ef780
0 36537pfr_ina_define:return 0 a0001a8a3401
0 36537pfr_ina_define:return 0 a001c0d94c00
You asked:
> why?
maybe this is no longer the case, but I thought in-kernel was "faster" than
dynamic or via a kld particularly on low power devices
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #6 from Kristof Provost --- (In reply to tech-lists from comment #5) No, run the dtrace command in one terminal and then (after dtrace has started) the pfctl command in another. If you have pf built into the kernel (why?) you'll want to probe fbt:kernel:pfr_ina_define:return instead. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406
--- Comment #5 from tech-li...@zyxst.net ---
(In reply to Kristof Provost from comment #4)
version is FreeBSD 14.0-CURRENT #1 main-n251261-25d0ccbe101: Thu Dec 2
20:22:08 GMT 2021
kernel looks like this:
[...]
cpu ARM64
ident RPI4B
include "std.arm64"
include "std.dev"
include "std.broadcom"
include "../../conf/std.nodebug"
# enable Pf without ALTQ (HFSC)
device pf
device pflog
# forward packets without decrementing
# the time to live (TTL) counter
options IPSTEALTH
options FUSEFS
device filemon # needed in order to use WITH_META_MODE=yes in
/etc/src-env.conf
You said:
[...]
"Can you confirm what revision of main you're running, and then show the output
of `dtrace -n 'fbt:pf:pfr_ina_define:return { printf("%x %x", arg0, arg1); }'`
run during a call to the failing pfctl command."
[...]
Do you mean run it like this:
# pfctl `dtrace -n 'fbt:pf:pfr_ina_define:return { printf("%x %x", arg0, arg1);
}'` -evf /etc/pf.conf
?
The results I get from that are:
"dtrace: invalid probe specifier fbt:pf:pfr_ina_define:return { printf("%x %x",
arg0, arg1); }: probe description fbt:pf:pfr_ina_define:return does not match
any probes"
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406
--- Comment #4 from Kristof Provost ---
(In reply to tech-lists from comment #3)
That's strange. DIOCRINADEFINE does allocate memory, but the first (and
largest) allocation is done with M_WAITOK and cannot fail (or produce ENOMEM).
The alternative sources seem to mostly be very small routine allocations that
I'd only expect to fail when the system is completely out of memory.
Can you confirm what revision of main you're running, and then show the output
of `dtrace -n 'fbt:pf:pfr_ina_define:return { printf("%x %x", arg0, arg1); }'`
run during a call to the failing pfctl command.
You are running CURRENT, right? 13.0 does use M_NOWAIT for the large
allocation, but that was fixed in stable/13 back in June.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 tech-li...@zyxst.net changed: What|Removed |Added CC||tech-li...@zyxst.net --- Comment #3 from tech-li...@zyxst.net --- Created attachment 230110 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=230110&action=edit truss output of pfctl -evf /etc/pf.conf -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 Kristof Provost changed: What|Removed |Added CC||k...@freebsd.org --- Comment #2 from Kristof Provost --- Let's start by figuring out which ioctl fails to allocate memory. Run your pfctl command under truss and attach the output to the bug please. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 --- Comment #1 from tech-li...@zyxst.net --- also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=259798 some more stats 1. when it gets the error, before reboot # vmstat -m | grep -E 'pf|Size' Type InUse MemUse Requests Size(s) pfil11 1K 19 64,128 tcpfunc 1 1K1 64 pf_hash 5 11524K5 2048 pf_ifnet 7 4K 55 256,2048 pf_rule2142K 57 2048 pf_osfp 1191 123K 3573 64,128 pf_table 612K 62 2048 pfs_nodes20 8K 20 384 pfs_vncache 116K4 32,16384 2. after reboot, after running: # doas -u _pfbadhost pf-badhost -O freebsd Password: pf-badhost 1552 - - Using experimental "aggy" aggregator... 1 table created. 18912 addresses added. pf-badhost 1620 - - IPv4 addresses in table: 619200855 but before loading pf.conf: # vmstat -m | grep -E 'pf|Size' Type InUse MemUse Requests Size(s) pfil 3 1K3 128 tcpfunc 1 1K1 64 pf_hash 5 11524K5 2048 pf_ifnet 7 4K 10 256,2048 pf_table 1 2K2 2048 pfs_nodes20 8K 20 384 pfs_vncache 116K1 16384 after loading pf.conf # vmstat -m | grep -E 'pf|Size' Type InUse MemUse Requests Size(s) pfil11 1K 11 64,128 tcpfunc 1 1K1 64 pf_hash 5 11524K5 2048 pf_ifnet 7 4K 25 256,2048 pf_rule1938K 19 2048 pf_osfp 1191 123K 1191 64,128 pf_table 612K 15 2048 pfs_nodes20 8K 20 384 pfs_vncache 116K1 16384 -- You are receiving this mail because: You are the assignee for the bug.
[Bug 260406] pfctl: Cannot allocate memory (after a time)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260406 Bug ID: 260406 Summary: pfctl: Cannot allocate memory (after a time) Product: Base System Version: CURRENT Hardware: arm64 URL: https://lists.freebsd.org/archives/freebsd-pf/2021-Dec ember/000163.html OS: Any Status: New Severity: Affects Many People Priority: --- Component: bin Assignee: b...@freebsd.org Reporter: tech-li...@zyxst.net Hi, A reboot solves this problem temporarily. Please see: https://lists.freebsd.org/archives/freebsd-pf/2021-December/000163.html https://forums.freebsd.org/threads/cannot-define-table-cannot-allocate-memory-since-upgrade-to-13-0.80822/ doas -u _pfbadhost pf-badhost -O freebsd Password: pf-badhost 8462 - - Using experimental "aggy" aggregator... pfctl: Cannot allocate memory. pf-badhost 8516 - - ERROR: '/etc/pf-badhost.txt' contains invalid data! Reverting changes and bailing out... this happens no matter how much memory is given to pf kern.maxdsiz=4294967296 State Table Total Rate current entries 20 searches 5762821 4577.3/s inserts42916 34.1/s removals 42896 34.1/s Counters match 71097 56.5/s bad-offset 00.0/s fragment 00.0/s short 00.0/s normalize 00.0/s memory 00.0/s bad-timestamp 00.0/s congestion 00.0/s ip-option 00.0/s proto-cksum00.0/s state-mismatch 00.0/s state-insert 00.0/s state-limit00.0/s src-limit 00.0/s synproxy 00.0/s map-failed 00.0/s LABEL COUNTERS: TIMEOUTS: tcp.first30s tcp.opening 5s tcp.established 18000s tcp.closing 60s tcp.finwait 30s tcp.closed 30s tcp.tsdiff 10s udp.first60s udp.single 30s udp.multiple 60s icmp.first 20s icmp.error 10s other.first 60s other.single 30s other.multiple 60s frag 30s interval 10s adaptive.start6 states adaptive.end 12 states src.track 0s LIMITS: stateshard limit 10 src-nodes hard limit1 frags hard limit 5000 table-entries hard limit 2540 -- You are receiving this mail because: You are the assignee for the bug.
