Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Tue, Jan 9, 2024 at 11:28 PM Roger Pau Monné wrote: > > On Tue, Jan 09, 2024 at 12:13:04PM +0100, Niklas Hallqvist wrote: > > > On 14 Dec 2022, at 07:16, G.R. wrote: ... > > > Hi Roger, > > > Just another query of the latest status. It'll be great if you can > > > share a link to the upstream commit. > > > I'm thinking of asking for a back-port of your fix to the FreeNAS > > > community, assuming it will take a long time to roll out otherwise. > > > > > > Thanks, > > > G.R. > > > > > >> > > >> Thanks, Roger. > > > > > > > > > > Hi everyone! > > > > So did anything ever happen on this? I find myself in the same situation > > with TrueNAS core 13, and can’t see any signs of changes in the FreeBSD 13 > > branches. > > Hello, > > I don't think the change is suitable to backport, it's IMO too > intrusive and risky. It was committed late 2022, and it's in 14.0: > ... > TrueNAS/OOPNsense might consider picking it up themselves. > > Thanks, Roger. Just FYI: I was able to locate that commit in 14.0 tree and cherry-picked it into TrueNas 13. I did it last November and the build has been running stably without issue since. The fix was fairly standalone and didn't cause any trouble during the cherry-picking so it could be reasonable to raise a request in the TrueNAS forum. Thanks, G.R.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Tue, Jan 09, 2024 at 12:13:04PM +0100, Niklas Hallqvist wrote: > > On 14 Dec 2022, at 07:16, G.R. wrote: > > > > On Thu, Nov 3, 2022 at 8:37 PM Roger Pau Monné wrote: > > Roger. > Hi Roger, any news for the upstream fix? I haven't heard any news > since... > The reason I came back to this thread is that I totally forgot about > this issue and upgraded to FreeNAS 13 only to rediscover this issue > once again :-( > > Any chance the patch can apply on FreeBSD 13.1-RELEASE-p1 kernel? > > Thanks, > G.R. > > >>> > >>> Hi, > >>> > >>> I want to confirm that the patch in an official release would make quite > >>> some people very happy. E.g. among OPNsense users, there are some who > >>> suffer from the network issue [1]. FWIW, I compiled a kernel including > >>> Roger's patch, and it seems to be working without trouble in my OPNsense > >>> DomU. > >> > >> Hello to both, > >> > >> Sorry, I completely dropped the ball on that patch, didn't even > >> remember I had it pending :(. > >> > >> Will do a build test with it and commit later today, I don't think I > >> will get any feedback, and it seems to improve the situation for your > >> use-cases. > > > > Hi Roger, > > Just another query of the latest status. It'll be great if you can > > share a link to the upstream commit. > > I'm thinking of asking for a back-port of your fix to the FreeNAS > > community, assuming it will take a long time to roll out otherwise. > > > > Thanks, > > G.R. > > > >> > >> Thanks, Roger. > > > > > > Hi everyone! > > So did anything ever happen on this? I find myself in the same situation > with TrueNAS core 13, and can’t see any signs of changes in the FreeBSD 13 > branches. Hello, I don't think the change is suitable to backport, it's IMO too intrusive and risky. It was committed late 2022, and it's in 14.0: commit dabb3db7a817f003af3f89c965ba369c67fc4910 Author: Roger Pau Monné Date: Thu Nov 3 13:29:22 2022 +0100 xen/netfront: deal with mbuf data crossing a page boundary There's been a report recently of mbufs with data that crosses a page boundary. It seems those mbufs are generated by the iSCSI target system: https://lists.xenproject.org/archives/html/xen-devel/2021-12/msg01581.html In order to handle those mbufs correctly on netfront use the bus_dma interface and explicitly request that segments must not cross a page boundary. No other requirements are necessary, so it's expected that bus_dma won't need to bounce the data and hence it shouldn't introduce a too big performance penalty. Using bus_dma requires some changes to netfront, mainly in order to accommodate for the fact that now ring slots no longer have a 1:1 match with mbufs, as a single mbuf can use two ring slots if the data buffer crosses a page boundary. Store the first packet of the mbuf chain in every ring slot that's used, and use a mbuf tag in order to store the bus_dma related structures and a refcount to keep track of the pending slots before the mbuf chain can be freed. Reported by: G.R. Tested by: G.R. MFC: 1 week Differential revision: https://reviews.freebsd.org/D33876 TrueNAS/OOPNsense might consider picking it up themselves. Thanks, Roger.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
> On 14 Dec 2022, at 07:16, G.R. wrote: > > On Thu, Nov 3, 2022 at 8:37 PM Roger Pau Monné wrote: > Roger. Hi Roger, any news for the upstream fix? I haven't heard any news since... The reason I came back to this thread is that I totally forgot about this issue and upgraded to FreeNAS 13 only to rediscover this issue once again :-( Any chance the patch can apply on FreeBSD 13.1-RELEASE-p1 kernel? Thanks, G.R. >>> >>> Hi, >>> >>> I want to confirm that the patch in an official release would make quite >>> some people very happy. E.g. among OPNsense users, there are some who >>> suffer from the network issue [1]. FWIW, I compiled a kernel including >>> Roger's patch, and it seems to be working without trouble in my OPNsense >>> DomU. >> >> Hello to both, >> >> Sorry, I completely dropped the ball on that patch, didn't even >> remember I had it pending :(. >> >> Will do a build test with it and commit later today, I don't think I >> will get any feedback, and it seems to improve the situation for your >> use-cases. > > Hi Roger, > Just another query of the latest status. It'll be great if you can > share a link to the upstream commit. > I'm thinking of asking for a back-port of your fix to the FreeNAS > community, assuming it will take a long time to roll out otherwise. > > Thanks, > G.R. > >> >> Thanks, Roger. > > Hi everyone! So did anything ever happen on this? I find myself in the same situation with TrueNAS core 13, and can’t see any signs of changes in the FreeBSD 13 branches. G.R: Did you ever build a kernel for TrueNAS 13? Care to share it? I don’t have a build environment setup, so I thought I’d just try to ask :-) /Niklas
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Thu, Nov 3, 2022 at 8:37 PM Roger Pau Monné wrote: > > > > Roger. > > > Hi Roger, any news for the upstream fix? I haven't heard any news since... > > > The reason I came back to this thread is that I totally forgot about > > > this issue and upgraded to FreeNAS 13 only to rediscover this issue > > > once again :-( > > > > > > Any chance the patch can apply on FreeBSD 13.1-RELEASE-p1 kernel? > > > > > > Thanks, > > > G.R. > > > > > > > Hi, > > > > I want to confirm that the patch in an official release would make quite > > some people very happy. E.g. among OPNsense users, there are some who > > suffer from the network issue [1]. FWIW, I compiled a kernel including > > Roger's patch, and it seems to be working without trouble in my OPNsense > > DomU. > > Hello to both, > > Sorry, I completely dropped the ball on that patch, didn't even > remember I had it pending :(. > > Will do a build test with it and commit later today, I don't think I > will get any feedback, and it seems to improve the situation for your > use-cases. Hi Roger, Just another query of the latest status. It'll be great if you can share a link to the upstream commit. I'm thinking of asking for a back-port of your fix to the FreeNAS community, assuming it will take a long time to roll out otherwise. Thanks, G.R. > > Thanks, Roger.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Thu, Nov 03, 2022 at 07:58:52AM +0100, Paul Leiber wrote: > > > Am 30.10.2022 um 17:36 schrieb G.R.: > > On Mon, Jan 10, 2022 at 10:54 PM Roger Pau Monné > > wrote: > > > > So looks like at least the imbalance between two directions are not > > > > related to your patch. > > > > Likely the debug build is a bigger contributor to the perf difference > > > > in both directions. > > > > > > > > I also tried your patch on a release build, and didn't observe any > > > > major difference in iperf3 numbers. > > > > Roughly match the 30Gbps and 1.xGbps number on the stock release kernel. > > > Thanks a lot, will try to get this upstream then. > > > > > > Roger. > > Hi Roger, any news for the upstream fix? I haven't heard any news since... > > The reason I came back to this thread is that I totally forgot about > > this issue and upgraded to FreeNAS 13 only to rediscover this issue > > once again :-( > > > > Any chance the patch can apply on FreeBSD 13.1-RELEASE-p1 kernel? > > > > Thanks, > > G.R. > > > > Hi, > > I want to confirm that the patch in an official release would make quite some > people very happy. E.g. among OPNsense users, there are some who > suffer from the network issue [1]. FWIW, I compiled a kernel including > Roger's patch, and it seems to be working without trouble in my OPNsense DomU. Hello to both, Sorry, I completely dropped the ball on that patch, didn't even remember I had it pending :(. Will do a build test with it and commit later today, I don't think I will get any feedback, and it seems to improve the situation for your use-cases. Thanks, Roger.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
Am 30.10.2022 um 17:36 schrieb G.R.: On Mon, Jan 10, 2022 at 10:54 PM Roger Pau Monné wrote: So looks like at least the imbalance between two directions are not related to your patch. Likely the debug build is a bigger contributor to the perf difference in both directions. I also tried your patch on a release build, and didn't observe any major difference in iperf3 numbers. Roughly match the 30Gbps and 1.xGbps number on the stock release kernel. Thanks a lot, will try to get this upstream then. Roger. Hi Roger, any news for the upstream fix? I haven't heard any news since... The reason I came back to this thread is that I totally forgot about this issue and upgraded to FreeNAS 13 only to rediscover this issue once again :-( Any chance the patch can apply on FreeBSD 13.1-RELEASE-p1 kernel? Thanks, G.R. Hi, I want to confirm that the patch in an official release would make quite some people very happy. E.g. among OPNsense users, there are some who suffer from the network issue [1]. FWIW, I compiled a kernel including Roger's patch, and it seems to be working without trouble in my OPNsense DomU. Best regards, Paul [1] https://forum.opnsense.org/index.php?topic=28708.15
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Mon, Jan 10, 2022 at 10:54 PM Roger Pau Monné wrote: > > So looks like at least the imbalance between two directions are not > > related to your patch. > > Likely the debug build is a bigger contributor to the perf difference > > in both directions. > > > > I also tried your patch on a release build, and didn't observe any > > major difference in iperf3 numbers. > > Roughly match the 30Gbps and 1.xGbps number on the stock release kernel. > > Thanks a lot, will try to get this upstream then. > > Roger. Hi Roger, any news for the upstream fix? I haven't heard any news since... The reason I came back to this thread is that I totally forgot about this issue and upgraded to FreeNAS 13 only to rediscover this issue once again :-( Any chance the patch can apply on FreeBSD 13.1-RELEASE-p1 kernel? Thanks, G.R.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Mon, Jan 10, 2022 at 10:54 PM Roger Pau Monné wrote: > > On Sat, Jan 08, 2022 at 01:14:26AM +0800, G.R. wrote: > > On Wed, Jan 5, 2022 at 10:33 PM Roger Pau Monné > > wrote: > > > > > > On Wed, Jan 05, 2022 at 12:05:39AM +0800, G.R. wrote: > > > > > > > > But seems like this patch is not stable enough yet and has its > > > > > > > > own > > > > > > > > issue -- memory is not properly released? > > > > > > > > > > > > > > I know. I've been working on improving it this morning and I'm > > > > > > > attaching an updated version below. > > > > > > > > > > > > > Good news. > > > > > > With this new patch, the NAS domU can serve iSCSI disk without OOM > > > > > > panic, at least for a little while. > > > > > > I'm going to keep it up and running for a while to see if it's > > > > > > stable over time. > > > > > > > > > > Thanks again for all the testing. Do you see any difference > > > > > performance wise? > > > > I'm still on a *debug* kernel build to capture any potential panic -- > > > > none so far -- no performance testing yet. > > > > Since I'm a home user with a relatively lightweight workload, so far I > > > > didn't observe any difference in daily usage. > > > > > > > > I did some quick iperf3 testing just now. > > > > > > Thanks for doing this. > > > > > > > 1. between nas domU <=> Linux dom0 running on an old i7-3770 based box. > > > > The peak is roughly 12 Gbits/s when domU is the server. > > > > But I do see regression down to ~8.5 Gbits/s when I repeat the test in > > > > a short burst. > > > > The regression can recover when I leave the system idle for a while. > > > > > > > > When dom0 is the iperf3 server, the transfer rate is much lower, down > > > > all the way to 1.x Gbits/s. > > > > Sometimes, I can see the following kernel log repeats during the > > > > testing, likely contributing to the slowdown. > > > > interrupt storm detected on "irq2328:"; throttling > > > > interrupt source > > > > > > I assume the message is in the domU, not the dom0? > > Yes, in the TrueNAS domU. > > BTW, I rebooted back to the stock kernel and the message is no longer > > observed. > > > > With the stock kernel, the transfer rate from dom0 to nas domU can be > > as high as 30Gbps. > > The variation is still observed, sometimes down to ~19Gbps. There is > > no retransmission in this direction. > > > > For the reverse direction, the observed low transfer rate still exists. > > It's still within the range of 1.x Gbps, but should still be better > > than the previous test. > > The huge number of re-transmission is still observed. > > The same behavior can be observed on a stock FreeBSD 12.2 image, so > > this is not specific to TrueNAS. > > So that's domU sending the data, and dom0 receiving it. Correct. > > > > > According to the packet capture, the re-transmission appears to be > > caused by packet reorder. > > Here is one example incident: > > 1. dom0 sees a sequence jump in the incoming stream and begins to send out > > SACKs > > 2. When SACK shows up at domU, it begins to re-transmit lost frames > >(the re-transmit looks weird since it show up as a mixed stream of > > 1448 bytes and 12 bytes packets, instead of always 1448 bytes) > > 3. Suddenly the packets that are believed to have lost show up, dom0 > > accept them as if they are re-transmission > > Hm, so there seems to be some kind of issue with ordering I would say. Agree. > > > 4. The actual re-transmission finally shows up in dom0... > > Should we expect packet reorder on a direct virtual link? Sounds fishy to > > me. > > Any chance we can get this re-transmission fixed? > > Does this still happen with all the extra features disabled? (-rxcsum > -txcsum -lro -tso) No obvious impact I would say. After disabling all extra features: xn0: flags=8843 metric 0 mtu 1500 ether 00:18:3c:51:6e:4c inet 192.168.1.9 netmask 0xff00 broadcast 192.168.1.255 media: Ethernet manual status: active nd6 options=9 The iperf3 result: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 2.04 GBytes 1.75 Gbits/sec 12674 sender [ 5] 0.00-10.14 sec 2.04 GBytes 1.73 Gbits/sec receiver BTW, those extra features have huge impact on the dom0 => domU direction. It goes all the way down from ~30 / 18 Gbps to 3.5 / 1.8 Gbps (variation range) without those. But there is no retransmission at all in both configs for this direction. I wonder why such a huge difference since the nic is purely virtual without any HW acceleration? Any further suggestions on this retransmission issue? > > > So looks like at least the imbalance between two directions are not > > related to your patch. > > Likely the debug build is a bigger contributor to the perf difference > > in both directions. > > > > I also tried your patch on a release build, and didn't observe any > > major difference in iperf3 numbers. > > Roughly match the 30Gbps and 1.xGbps number on the stock release kernel. > >
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Sat, Jan 08, 2022 at 01:14:26AM +0800, G.R. wrote: > On Wed, Jan 5, 2022 at 10:33 PM Roger Pau Monné wrote: > > > > On Wed, Jan 05, 2022 at 12:05:39AM +0800, G.R. wrote: > > > > > > > But seems like this patch is not stable enough yet and has its own > > > > > > > issue -- memory is not properly released? > > > > > > > > > > > > I know. I've been working on improving it this morning and I'm > > > > > > attaching an updated version below. > > > > > > > > > > > Good news. > > > > > With this new patch, the NAS domU can serve iSCSI disk without OOM > > > > > panic, at least for a little while. > > > > > I'm going to keep it up and running for a while to see if it's stable > > > > > over time. > > > > > > > > Thanks again for all the testing. Do you see any difference > > > > performance wise? > > > I'm still on a *debug* kernel build to capture any potential panic -- > > > none so far -- no performance testing yet. > > > Since I'm a home user with a relatively lightweight workload, so far I > > > didn't observe any difference in daily usage. > > > > > > I did some quick iperf3 testing just now. > > > > Thanks for doing this. > > > > > 1. between nas domU <=> Linux dom0 running on an old i7-3770 based box. > > > The peak is roughly 12 Gbits/s when domU is the server. > > > But I do see regression down to ~8.5 Gbits/s when I repeat the test in > > > a short burst. > > > The regression can recover when I leave the system idle for a while. > > > > > > When dom0 is the iperf3 server, the transfer rate is much lower, down > > > all the way to 1.x Gbits/s. > > > Sometimes, I can see the following kernel log repeats during the > > > testing, likely contributing to the slowdown. > > > interrupt storm detected on "irq2328:"; throttling interrupt > > > source > > > > I assume the message is in the domU, not the dom0? > Yes, in the TrueNAS domU. > BTW, I rebooted back to the stock kernel and the message is no longer > observed. > > With the stock kernel, the transfer rate from dom0 to nas domU can be > as high as 30Gbps. > The variation is still observed, sometimes down to ~19Gbps. There is > no retransmission in this direction. > > For the reverse direction, the observed low transfer rate still exists. > It's still within the range of 1.x Gbps, but should still be better > than the previous test. > The huge number of re-transmission is still observed. > The same behavior can be observed on a stock FreeBSD 12.2 image, so > this is not specific to TrueNAS. So that's domU sending the data, and dom0 receiving it. > > According to the packet capture, the re-transmission appears to be > caused by packet reorder. > Here is one example incident: > 1. dom0 sees a sequence jump in the incoming stream and begins to send out > SACKs > 2. When SACK shows up at domU, it begins to re-transmit lost frames >(the re-transmit looks weird since it show up as a mixed stream of > 1448 bytes and 12 bytes packets, instead of always 1448 bytes) > 3. Suddenly the packets that are believed to have lost show up, dom0 > accept them as if they are re-transmission Hm, so there seems to be some kind of issue with ordering I would say. > 4. The actual re-transmission finally shows up in dom0... > Should we expect packet reorder on a direct virtual link? Sounds fishy to me. > Any chance we can get this re-transmission fixed? Does this still happen with all the extra features disabled? (-rxcsum -txcsum -lro -tso) > So looks like at least the imbalance between two directions are not > related to your patch. > Likely the debug build is a bigger contributor to the perf difference > in both directions. > > I also tried your patch on a release build, and didn't observe any > major difference in iperf3 numbers. > Roughly match the 30Gbps and 1.xGbps number on the stock release kernel. Thanks a lot, will try to get this upstream then. Roger.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Wed, Jan 5, 2022 at 10:33 PM Roger Pau Monné wrote: > > On Wed, Jan 05, 2022 at 12:05:39AM +0800, G.R. wrote: > > > > > > But seems like this patch is not stable enough yet and has its own > > > > > > issue -- memory is not properly released? > > > > > > > > > > I know. I've been working on improving it this morning and I'm > > > > > attaching an updated version below. > > > > > > > > > Good news. > > > > With this new patch, the NAS domU can serve iSCSI disk without OOM > > > > panic, at least for a little while. > > > > I'm going to keep it up and running for a while to see if it's stable > > > > over time. > > > > > > Thanks again for all the testing. Do you see any difference > > > performance wise? > > I'm still on a *debug* kernel build to capture any potential panic -- > > none so far -- no performance testing yet. > > Since I'm a home user with a relatively lightweight workload, so far I > > didn't observe any difference in daily usage. > > > > I did some quick iperf3 testing just now. > > Thanks for doing this. > > > 1. between nas domU <=> Linux dom0 running on an old i7-3770 based box. > > The peak is roughly 12 Gbits/s when domU is the server. > > But I do see regression down to ~8.5 Gbits/s when I repeat the test in > > a short burst. > > The regression can recover when I leave the system idle for a while. > > > > When dom0 is the iperf3 server, the transfer rate is much lower, down > > all the way to 1.x Gbits/s. > > Sometimes, I can see the following kernel log repeats during the > > testing, likely contributing to the slowdown. > > interrupt storm detected on "irq2328:"; throttling interrupt > > source > > I assume the message is in the domU, not the dom0? Yes, in the TrueNAS domU. BTW, I rebooted back to the stock kernel and the message is no longer observed. With the stock kernel, the transfer rate from dom0 to nas domU can be as high as 30Gbps. The variation is still observed, sometimes down to ~19Gbps. There is no retransmission in this direction. For the reverse direction, the observed low transfer rate still exists. It's still within the range of 1.x Gbps, but should still be better than the previous test. The huge number of re-transmission is still observed. The same behavior can be observed on a stock FreeBSD 12.2 image, so this is not specific to TrueNAS. According to the packet capture, the re-transmission appears to be caused by packet reorder. Here is one example incident: 1. dom0 sees a sequence jump in the incoming stream and begins to send out SACKs 2. When SACK shows up at domU, it begins to re-transmit lost frames (the re-transmit looks weird since it show up as a mixed stream of 1448 bytes and 12 bytes packets, instead of always 1448 bytes) 3. Suddenly the packets that are believed to have lost show up, dom0 accept them as if they are re-transmission 4. The actual re-transmission finally shows up in dom0... Should we expect packet reorder on a direct virtual link? Sounds fishy to me. Any chance we can get this re-transmission fixed? So looks like at least the imbalance between two directions are not related to your patch. Likely the debug build is a bigger contributor to the perf difference in both directions. I also tried your patch on a release build, and didn't observe any major difference in iperf3 numbers. Roughly match the 30Gbps and 1.xGbps number on the stock release kernel. > > > Another thing that looks alarming is the retransmission is high: > > [ ID] Interval Transfer Bitrate Retr Cwnd > > [ 5] 0.00-1.00 sec 212 MBytes 1.78 Gbits/sec 110231 KBytes > > [ 5] 1.00-2.00 sec 230 MBytes 1.92 Gbits/sec1439 KBytes > > [ 5] 2.00-3.00 sec 228 MBytes 1.92 Gbits/sec3335 KBytes > > [ 5] 3.00-4.00 sec 204 MBytes 1.71 Gbits/sec1486 KBytes > > [ 5] 4.00-5.00 sec 201 MBytes 1.69 Gbits/sec 812258 KBytes > > [ 5] 5.00-6.00 sec 179 MBytes 1.51 Gbits/sec1372 KBytes > > [ 5] 6.00-7.00 sec 50.5 MBytes 423 Mbits/sec2154 KBytes > > [ 5] 7.00-8.00 sec 194 MBytes 1.63 Gbits/sec 339172 KBytes > > [ 5] 8.00-9.00 sec 156 MBytes 1.30 Gbits/sec 854215 KBytes > > [ 5] 9.00-10.00 sec 143 MBytes 1.20 Gbits/sec 997 93.8 KBytes > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-10.00 sec 1.76 GBytes 1.51 Gbits/sec 3120 > > sender > > [ 5] 0.00-10.45 sec 1.76 GBytes 1.44 Gbits/sec > > receiver > > Do you see the same when running the same tests on a debug kernel > without my patch applied? (ie: a kernel build yourself from the same > baseline but just without my patch applied) > > I'm mostly interested in knowing whether the patch itself causes any > regressions from the current state (which might not be great already). > > > > > 2. between a remote box <=> nas domU, through a 1Gbps eth
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Wed, Jan 05, 2022 at 12:05:39AM +0800, G.R. wrote: > > > > > But seems like this patch is not stable enough yet and has its own > > > > > issue -- memory is not properly released? > > > > > > > > I know. I've been working on improving it this morning and I'm > > > > attaching an updated version below. > > > > > > > Good news. > > > With this new patch, the NAS domU can serve iSCSI disk without OOM > > > panic, at least for a little while. > > > I'm going to keep it up and running for a while to see if it's stable > > > over time. > > > > Thanks again for all the testing. Do you see any difference > > performance wise? > I'm still on a *debug* kernel build to capture any potential panic -- > none so far -- no performance testing yet. > Since I'm a home user with a relatively lightweight workload, so far I > didn't observe any difference in daily usage. > > I did some quick iperf3 testing just now. Thanks for doing this. > 1. between nas domU <=> Linux dom0 running on an old i7-3770 based box. > The peak is roughly 12 Gbits/s when domU is the server. > But I do see regression down to ~8.5 Gbits/s when I repeat the test in > a short burst. > The regression can recover when I leave the system idle for a while. > > When dom0 is the iperf3 server, the transfer rate is much lower, down > all the way to 1.x Gbits/s. > Sometimes, I can see the following kernel log repeats during the > testing, likely contributing to the slowdown. > interrupt storm detected on "irq2328:"; throttling interrupt > source I assume the message is in the domU, not the dom0? > Another thing that looks alarming is the retransmission is high: > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 212 MBytes 1.78 Gbits/sec 110231 KBytes > [ 5] 1.00-2.00 sec 230 MBytes 1.92 Gbits/sec1439 KBytes > [ 5] 2.00-3.00 sec 228 MBytes 1.92 Gbits/sec3335 KBytes > [ 5] 3.00-4.00 sec 204 MBytes 1.71 Gbits/sec1486 KBytes > [ 5] 4.00-5.00 sec 201 MBytes 1.69 Gbits/sec 812258 KBytes > [ 5] 5.00-6.00 sec 179 MBytes 1.51 Gbits/sec1372 KBytes > [ 5] 6.00-7.00 sec 50.5 MBytes 423 Mbits/sec2154 KBytes > [ 5] 7.00-8.00 sec 194 MBytes 1.63 Gbits/sec 339172 KBytes > [ 5] 8.00-9.00 sec 156 MBytes 1.30 Gbits/sec 854215 KBytes > [ 5] 9.00-10.00 sec 143 MBytes 1.20 Gbits/sec 997 93.8 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 1.76 GBytes 1.51 Gbits/sec 3120 sender > [ 5] 0.00-10.45 sec 1.76 GBytes 1.44 Gbits/sec receiver Do you see the same when running the same tests on a debug kernel without my patch applied? (ie: a kernel build yourself from the same baseline but just without my patch applied) I'm mostly interested in knowing whether the patch itself causes any regressions from the current state (which might not be great already). > > 2. between a remote box <=> nas domU, through a 1Gbps ethernet cable. > Roughly saturate the link when domU is the server, without obvious perf drop > When domU running as a client, the achieved BW is ~30Mbps lower than the peak. > Retransmission sometimes also shows up in this scenario, more > seriously when domU is the client. > > I cannot test with the stock kernel nor with your patch in release > mode immediately. > > But according to the observed imbalance between inbounding and > outgoing path, non-trivial penalty applies I guess? We should get a baseline using the same sources without my path applied. Thanks, Roger.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
> > > > But seems like this patch is not stable enough yet and has its own > > > > issue -- memory is not properly released? > > > > > > I know. I've been working on improving it this morning and I'm > > > attaching an updated version below. > > > > > Good news. > > With this new patch, the NAS domU can serve iSCSI disk without OOM > > panic, at least for a little while. > > I'm going to keep it up and running for a while to see if it's stable over > > time. > > Thanks again for all the testing. Do you see any difference > performance wise? I'm still on a *debug* kernel build to capture any potential panic -- none so far -- no performance testing yet. Since I'm a home user with a relatively lightweight workload, so far I didn't observe any difference in daily usage. I did some quick iperf3 testing just now. 1. between nas domU <=> Linux dom0 running on an old i7-3770 based box. The peak is roughly 12 Gbits/s when domU is the server. But I do see regression down to ~8.5 Gbits/s when I repeat the test in a short burst. The regression can recover when I leave the system idle for a while. When dom0 is the iperf3 server, the transfer rate is much lower, down all the way to 1.x Gbits/s. Sometimes, I can see the following kernel log repeats during the testing, likely contributing to the slowdown. interrupt storm detected on "irq2328:"; throttling interrupt source Another thing that looks alarming is the retransmission is high: [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 212 MBytes 1.78 Gbits/sec 110231 KBytes [ 5] 1.00-2.00 sec 230 MBytes 1.92 Gbits/sec1439 KBytes [ 5] 2.00-3.00 sec 228 MBytes 1.92 Gbits/sec3335 KBytes [ 5] 3.00-4.00 sec 204 MBytes 1.71 Gbits/sec1486 KBytes [ 5] 4.00-5.00 sec 201 MBytes 1.69 Gbits/sec 812258 KBytes [ 5] 5.00-6.00 sec 179 MBytes 1.51 Gbits/sec1372 KBytes [ 5] 6.00-7.00 sec 50.5 MBytes 423 Mbits/sec2154 KBytes [ 5] 7.00-8.00 sec 194 MBytes 1.63 Gbits/sec 339172 KBytes [ 5] 8.00-9.00 sec 156 MBytes 1.30 Gbits/sec 854215 KBytes [ 5] 9.00-10.00 sec 143 MBytes 1.20 Gbits/sec 997 93.8 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.76 GBytes 1.51 Gbits/sec 3120 sender [ 5] 0.00-10.45 sec 1.76 GBytes 1.44 Gbits/sec receiver 2. between a remote box <=> nas domU, through a 1Gbps ethernet cable. Roughly saturate the link when domU is the server, without obvious perf drop When domU running as a client, the achieved BW is ~30Mbps lower than the peak. Retransmission sometimes also shows up in this scenario, more seriously when domU is the client. I cannot test with the stock kernel nor with your patch in release mode immediately. But according to the observed imbalance between inbounding and outgoing path, non-trivial penalty applies I guess? > > BTW, an irrelevant question: > > What's the current status of HVM domU on top of storage driver domain? > > About 7 years ago, one user on the list was able to get this setup up > > and running with your help (patch).[1] > > When I attempted to reproduce a similar setup two years later, I > > discovered that the patch was not submitted. > > And even with that patch the setup cannot be reproduced successfully. > > We spent some time debugging on the problem together[2], but didn't > > bottom out the root cause at that time. > > In case it's still broken and you still have the interest and time, I > > can launch a separate thread on this topic and provide required > > testing environment. > > Yes, better as a new thread please. > > FWIW, I haven't looked at this since a long time, but I recall some > fixes in order to be able to use driver domains with HVM guests, which > require attaching the disk to dom0 in order for the device model > (QEMU) to access it. > > I would give it a try without using stubdomains and see what you get. > You will need to run `xl devd` inside of the driver domain, so you > will need to install xen-tools on the domU. There's an init script to > launch `xl devd` at boot, it's called 'xendriverdomain'. Looks like I'm unlucky once again. Let's follow up in a separate thread. > Thanks, Roger.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Fri, Dec 31, 2021 at 10:47:57PM +0800, G.R. wrote: > On Fri, Dec 31, 2021 at 2:52 AM Roger Pau Monné wrote: > > > > On Thu, Dec 30, 2021 at 11:12:57PM +0800, G.R. wrote: > > > On Thu, Dec 30, 2021 at 3:07 AM Roger Pau Monné > > > wrote: > > > > > > > > On Wed, Dec 29, 2021 at 11:27:50AM +0100, Roger Pau Monné wrote: > > > > > On Wed, Dec 29, 2021 at 05:13:00PM +0800, G.R. wrote: > > > > > > > > > > > > > > I think this is hitting a KASSERT, could you paste the text > > > > > > > printed as > > > > > > > part of the panic (not just he backtrace)? > > > > > > > > > > > > > > Sorry this is taking a bit of time to solve. > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > Sorry that I didn't make it clear in the first place. > > > > > > It is the same cross boundary assertion. > > > > > > > > > > I see. After looking at the code it seems like sglist will coalesce > > > > > contiguous physical ranges without taking page boundaries into > > > > > account, which is not suitable for our purpose here. I guess I will > > > > > either have to modify sglist, or switch to using bus_dma. The main > > > > > problem with using bus_dma is that it will require bigger changes to > > > > > netfront I think. > > > > > > > > I have a crappy patch to use bus_dma. It's not yet ready for upstream > > > > but you might want to give it a try to see if it solves the cross page > > > > boundary issues. > > > > > > > I think this version is better. > > > > Thanks for all the testing. > > > > > It fixed the mbuf cross boundary issue and allowed me to boot from one > > > disk image successfully. > > > > It's good to know it seems to handle splitting mbufs fragments at page > > boundaries correctly. > > > > > But seems like this patch is not stable enough yet and has its own > > > issue -- memory is not properly released? > > > > I know. I've been working on improving it this morning and I'm > > attaching an updated version below. > > > Good news. > With this new patch, the NAS domU can serve iSCSI disk without OOM > panic, at least for a little while. > I'm going to keep it up and running for a while to see if it's stable over > time. Thanks again for all the testing. Do you see any difference performance wise? > BTW, an irrelevant question: > What's the current status of HVM domU on top of storage driver domain? > About 7 years ago, one user on the list was able to get this setup up > and running with your help (patch).[1] > When I attempted to reproduce a similar setup two years later, I > discovered that the patch was not submitted. > And even with that patch the setup cannot be reproduced successfully. > We spent some time debugging on the problem together[2], but didn't > bottom out the root cause at that time. > In case it's still broken and you still have the interest and time, I > can launch a separate thread on this topic and provide required > testing environment. Yes, better as a new thread please. FWIW, I haven't looked at this since a long time, but I recall some fixes in order to be able to use driver domains with HVM guests, which require attaching the disk to dom0 in order for the device model (QEMU) to access it. I would give it a try without using stubdomains and see what you get. You will need to run `xl devd` inside of the driver domain, so you will need to install xen-tools on the domU. There's an init script to launch `xl devd` at boot, it's called 'xendriverdomain'. Thanks, Roger.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Fri, Dec 31, 2021 at 2:52 AM Roger Pau Monné wrote: > > On Thu, Dec 30, 2021 at 11:12:57PM +0800, G.R. wrote: > > On Thu, Dec 30, 2021 at 3:07 AM Roger Pau Monné > > wrote: > > > > > > On Wed, Dec 29, 2021 at 11:27:50AM +0100, Roger Pau Monné wrote: > > > > On Wed, Dec 29, 2021 at 05:13:00PM +0800, G.R. wrote: > > > > > > > > > > > > I think this is hitting a KASSERT, could you paste the text printed > > > > > > as > > > > > > part of the panic (not just he backtrace)? > > > > > > > > > > > > Sorry this is taking a bit of time to solve. > > > > > > > > > > > > Thanks! > > > > > > > > > > > Sorry that I didn't make it clear in the first place. > > > > > It is the same cross boundary assertion. > > > > > > > > I see. After looking at the code it seems like sglist will coalesce > > > > contiguous physical ranges without taking page boundaries into > > > > account, which is not suitable for our purpose here. I guess I will > > > > either have to modify sglist, or switch to using bus_dma. The main > > > > problem with using bus_dma is that it will require bigger changes to > > > > netfront I think. > > > > > > I have a crappy patch to use bus_dma. It's not yet ready for upstream > > > but you might want to give it a try to see if it solves the cross page > > > boundary issues. > > > > > I think this version is better. > > Thanks for all the testing. > > > It fixed the mbuf cross boundary issue and allowed me to boot from one > > disk image successfully. > > It's good to know it seems to handle splitting mbufs fragments at page > boundaries correctly. > > > But seems like this patch is not stable enough yet and has its own > > issue -- memory is not properly released? > > I know. I've been working on improving it this morning and I'm > attaching an updated version below. > Good news. With this new patch, the NAS domU can serve iSCSI disk without OOM panic, at least for a little while. I'm going to keep it up and running for a while to see if it's stable over time. BTW, an irrelevant question: What's the current status of HVM domU on top of storage driver domain? About 7 years ago, one user on the list was able to get this setup up and running with your help (patch).[1] When I attempted to reproduce a similar setup two years later, I discovered that the patch was not submitted. And even with that patch the setup cannot be reproduced successfully. We spent some time debugging on the problem together[2], but didn't bottom out the root cause at that time. In case it's still broken and you still have the interest and time, I can launch a separate thread on this topic and provide required testing environment. [1] https://lists.xenproject.org/archives/html/xen-users/2014-08/msg3.html [2] https://xen-users.narkive.com/9ihP0QG4/hvm-domu-on-storage-driver-domain Thanks, G.R. > Thanks, Roger.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Thu, Dec 30, 2021 at 11:12:57PM +0800, G.R. wrote:
> On Thu, Dec 30, 2021 at 3:07 AM Roger Pau Monné wrote:
> >
> > On Wed, Dec 29, 2021 at 11:27:50AM +0100, Roger Pau Monné wrote:
> > > On Wed, Dec 29, 2021 at 05:13:00PM +0800, G.R. wrote:
> > > > >
> > > > > I think this is hitting a KASSERT, could you paste the text printed as
> > > > > part of the panic (not just he backtrace)?
> > > > >
> > > > > Sorry this is taking a bit of time to solve.
> > > > >
> > > > > Thanks!
> > > > >
> > > > Sorry that I didn't make it clear in the first place.
> > > > It is the same cross boundary assertion.
> > >
> > > I see. After looking at the code it seems like sglist will coalesce
> > > contiguous physical ranges without taking page boundaries into
> > > account, which is not suitable for our purpose here. I guess I will
> > > either have to modify sglist, or switch to using bus_dma. The main
> > > problem with using bus_dma is that it will require bigger changes to
> > > netfront I think.
> >
> > I have a crappy patch to use bus_dma. It's not yet ready for upstream
> > but you might want to give it a try to see if it solves the cross page
> > boundary issues.
> >
> I think this version is better.
Thanks for all the testing.
> It fixed the mbuf cross boundary issue and allowed me to boot from one
> disk image successfully.
It's good to know it seems to handle splitting mbufs fragments at page
boundaries correctly.
> But seems like this patch is not stable enough yet and has its own
> issue -- memory is not properly released?
I know. I've been working on improving it this morning and I'm
attaching an updated version below.
Thanks, Roger.
---
diff --git a/sys/dev/xen/netfront/netfront.c b/sys/dev/xen/netfront/netfront.c
index 8dba5a8dc6d5..69528cc39b94 100644
--- a/sys/dev/xen/netfront/netfront.c
+++ b/sys/dev/xen/netfront/netfront.c
@@ -71,6 +71,8 @@ __FBSDID("$FreeBSD$");
#include
#include
+#include
+
#include "xenbus_if.h"
/* Features supported by all backends. TSO and LRO can be negotiated */
@@ -199,6 +201,17 @@ struct netfront_txq {
struct taskqueue*tq;
struct task defrtask;
+ bus_dma_segment_t segs[MAX_TX_REQ_FRAGS];
+ struct mbuf_xennet {
+ struct m_tagtag;
+ bus_dma_tag_t dma_tag;
+ bus_dmamap_tdma_map;
+ struct netfront_txq *txq;
+ SLIST_ENTRY(mbuf_xennet) next;
+ u_int count;
+ } xennet_tag[NET_TX_RING_SIZE + 1];
+ SLIST_HEAD(, mbuf_xennet) tags;
+
boolfull;
};
@@ -221,6 +234,8 @@ struct netfront_info {
struct ifmedia sc_media;
+ bus_dma_tag_t dma_tag;
+
boolxn_reset;
};
@@ -301,6 +316,42 @@ xn_get_rx_ref(struct netfront_rxq *rxq, RING_IDX ri)
return (ref);
}
+#define MTAG_COOKIE 1218492000
+#define MTAG_XENNET 0
+
+static void mbuf_grab(struct mbuf *m)
+{
+ struct mbuf_xennet *ref;
+
+ ref = (struct mbuf_xennet *)m_tag_locate(m, MTAG_COOKIE,
+ MTAG_XENNET, NULL);
+ KASSERT(ref != NULL, ("Cannot find refcount"));
+ ref->count++;
+}
+
+static void mbuf_release(struct mbuf *m)
+{
+ struct mbuf_xennet *ref;
+
+ ref = (struct mbuf_xennet *)m_tag_locate(m, MTAG_COOKIE,
+ MTAG_XENNET, NULL);
+ KASSERT(ref != NULL, ("Cannot find refcount"));
+ KASSERT(ref->count > 0, ("Invalid reference count"));
+
+ if (--ref->count == 0)
+ m_freem(m);
+}
+
+static void tag_free(struct m_tag *t)
+{
+ struct mbuf_xennet *ref = (struct mbuf_xennet *)t;
+
+ KASSERT(ref->count == 0, ("Free mbuf tag with pending refcnt"));
+ bus_dmamap_sync(ref->dma_tag, ref->dma_map, BUS_DMASYNC_POSTWRITE);
+ bus_dmamap_destroy(ref->dma_tag, ref->dma_map);
+ SLIST_INSERT_HEAD(&ref->txq->tags, ref, next);
+}
+
#define IPRINTK(fmt, args...) \
printf("[XEN] " fmt, ##args)
#ifdef INVARIANTS
@@ -778,11 +829,18 @@ disconnect_txq(struct netfront_txq *txq)
static void
destroy_txq(struct netfront_txq *txq)
{
+ unsigned int i;
free(txq->ring.sring, M_DEVBUF);
buf_ring_free(txq->br, M_DEVBUF);
taskqueue_drain_all(txq->tq);
taskqueue_free(txq->tq);
+
+ for (i = 0; i <= NET_TX_RING_SIZE; i++) {
+ bus_dmamap_destroy(txq->info->dma_tag,
+ txq->xennet_tag[i].dma_map);
+ txq->xennet_tag[i].dma_map = NULL;
+ }
}
static void
@@ -822,10 +880,27 @@ setup_txqs(device_t dev, struct netfront_info *info,
mtx_init(&txq->lock, txq->name, "netfront transmit lock",
MTX_DEF);
+ SLIST_INIT(&txq->tags);
for (i = 0; i <= NET_TX_RING_SIZE; i++) {
txq->mbufs[i] = (void *) ((u_long) i+1);
txq->grant_ref[i] = GRANT_REF_INVALID
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Thu, Dec 30, 2021 at 3:07 AM Roger Pau Monné wrote: > > On Wed, Dec 29, 2021 at 11:27:50AM +0100, Roger Pau Monné wrote: > > On Wed, Dec 29, 2021 at 05:13:00PM +0800, G.R. wrote: > > > > > > > > I think this is hitting a KASSERT, could you paste the text printed as > > > > part of the panic (not just he backtrace)? > > > > > > > > Sorry this is taking a bit of time to solve. > > > > > > > > Thanks! > > > > > > > Sorry that I didn't make it clear in the first place. > > > It is the same cross boundary assertion. > > > > I see. After looking at the code it seems like sglist will coalesce > > contiguous physical ranges without taking page boundaries into > > account, which is not suitable for our purpose here. I guess I will > > either have to modify sglist, or switch to using bus_dma. The main > > problem with using bus_dma is that it will require bigger changes to > > netfront I think. > > I have a crappy patch to use bus_dma. It's not yet ready for upstream > but you might want to give it a try to see if it solves the cross page > boundary issues. > I think this version is better. It fixed the mbuf cross boundary issue and allowed me to boot from one disk image successfully. But seems like this patch is not stable enough yet and has its own issue -- memory is not properly released? The stack trace is likely not useful, but anyway... Context: pmap_growkernel: no memory to grow kernel <118>Dec 30 22:55:47 nas kernel[2164]: Last message 'pid 1066 (python3.9)' repeated 1 times, suppressed by syslog-ng on nas.rglab.us <118>Dec 30 22:55:47 nas kernel: pid 2086 (python3.9), jid 0, uid 0, was killed: out of swap space panic: pmap_growkernel: no memory to grow kernel cpuid = 1 time = 1640876153 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe009b971210 vpanic() at vpanic+0x17b/frame 0xfe009b971260 panic() at panic+0x43/frame 0xfe009b9712c0 pmap_growkernel() at pmap_growkernel+0x2f1/frame 0xfe009b971300 vm_map_insert() at vm_map_insert+0x27b/frame 0xfe009b971390 vm_map_find() at vm_map_find+0x5ed/frame 0xfe009b971470 kva_import() at kva_import+0x3c/frame 0xfe009b9714b0 vmem_try_fetch() at vmem_try_fetch+0xde/frame 0xfe009b971500 vmem_xalloc() at vmem_xalloc+0x4db/frame 0xfe009b971580 kva_import_domain() at kva_import_domain+0x36/frame 0xfe009b9715b0 vmem_try_fetch() at vmem_try_fetch+0xde/frame 0xfe009b971600 vmem_xalloc() at vmem_xalloc+0x4db/frame 0xfe009b971680 vmem_alloc() at vmem_alloc+0x8a/frame 0xfe009b9716d0 kmem_malloc_domainset() at kmem_malloc_domainset+0x92/frame 0xfe009b971740 keg_alloc_slab() at keg_alloc_slab+0xfa/frame 0xfe009b9717a0 keg_fetch_slab() at keg_fetch_slab+0xfe/frame 0xfe009b971830 zone_fetch_slab() at zone_fetch_slab+0x61/frame 0xfe009b971870 zone_import() at zone_import+0x75/frame 0xfe009b9718f0 zone_alloc_item() at zone_alloc_item+0x56/frame 0xfe009b971930 abd_borrow_buf() at abd_borrow_buf+0x1f/frame 0xfe009b971950 vdev_geom_io_start() at vdev_geom_io_start+0x189/frame 0xfe009b971980 zio_vdev_io_start() at zio_vdev_io_start+0x1e4/frame 0xfe009b9719d0 zio_nowait() at zio_nowait+0x11a/frame 0xfe009b971a30 vdev_queue_io_done() at vdev_queue_io_done+0x1b8/frame 0xfe009b971a90 zio_vdev_io_done() at zio_vdev_io_done+0xe3/frame 0xfe009b971ad0 zio_execute() at zio_execute+0x6a/frame 0xfe009b971b20 taskqueue_run_locked() at taskqueue_run_locked+0x168/frame 0xfe009b971b80 taskqueue_thread_loop() at taskqueue_thread_loop+0x94/frame 0xfe009b971bb0 fork_exit() at fork_exit+0x80/frame 0xfe009b971bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe009b971bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Wed, Dec 29, 2021 at 11:27:50AM +0100, Roger Pau Monné wrote:
> On Wed, Dec 29, 2021 at 05:13:00PM +0800, G.R. wrote:
> > >
> > > I think this is hitting a KASSERT, could you paste the text printed as
> > > part of the panic (not just he backtrace)?
> > >
> > > Sorry this is taking a bit of time to solve.
> > >
> > > Thanks!
> > >
> > Sorry that I didn't make it clear in the first place.
> > It is the same cross boundary assertion.
>
> I see. After looking at the code it seems like sglist will coalesce
> contiguous physical ranges without taking page boundaries into
> account, which is not suitable for our purpose here. I guess I will
> either have to modify sglist, or switch to using bus_dma. The main
> problem with using bus_dma is that it will require bigger changes to
> netfront I think.
I have a crappy patch to use bus_dma. It's not yet ready for upstream
but you might want to give it a try to see if it solves the cross page
boundary issues.
Thanks, Roger.
---
diff --git a/sys/dev/xen/netfront/netfront.c b/sys/dev/xen/netfront/netfront.c
index 8dba5a8dc6d5..693ef25c8783 100644
--- a/sys/dev/xen/netfront/netfront.c
+++ b/sys/dev/xen/netfront/netfront.c
@@ -71,6 +71,8 @@ __FBSDID("$FreeBSD$");
#include
#include
+#include
+
#include "xenbus_if.h"
/* Features supported by all backends. TSO and LRO can be negotiated */
@@ -199,6 +201,12 @@ struct netfront_txq {
struct taskqueue*tq;
struct task defrtask;
+ bus_dmamap_tdma_map;
+ struct mbuf_refcount {
+ struct m_tagtag;
+ u_int count;
+ } refcount_tag[NET_TX_RING_SIZE + 1];
+
boolfull;
};
@@ -221,6 +229,8 @@ struct netfront_info {
struct ifmedia sc_media;
+ bus_dma_tag_t dma_tag;
+
boolxn_reset;
};
@@ -301,6 +311,39 @@ xn_get_rx_ref(struct netfront_rxq *rxq, RING_IDX ri)
return (ref);
}
+#define MTAG_COOKIE 1218492000
+#define MTAG_REFCOUNT 0
+
+static void mbuf_grab(struct mbuf *m)
+{
+ struct mbuf_refcount *ref;
+
+ ref = (struct mbuf_refcount *)m_tag_locate(m, MTAG_COOKIE,
+ MTAG_REFCOUNT, NULL);
+ KASSERT(ref != NULL, ("Cannot find refcount"));
+ ref->count++;
+}
+
+static void mbuf_release(struct mbuf *m)
+{
+ struct mbuf_refcount *ref;
+
+ ref = (struct mbuf_refcount *)m_tag_locate(m, MTAG_COOKIE,
+ MTAG_REFCOUNT, NULL);
+ KASSERT(ref != NULL, ("Cannot find refcount"));
+ KASSERT(ref->count > 0, ("Invalid reference count"));
+
+ if (--ref->count == 0)
+ m_freem(m);
+}
+
+static void tag_free(struct m_tag *t)
+{
+ struct mbuf_refcount *ref = (struct mbuf_refcount *)t;
+
+ KASSERT(ref->count == 0, ("Free mbuf tag with pending refcnt"));
+}
+
#define IPRINTK(fmt, args...) \
printf("[XEN] " fmt, ##args)
#ifdef INVARIANTS
@@ -783,6 +826,7 @@ destroy_txq(struct netfront_txq *txq)
buf_ring_free(txq->br, M_DEVBUF);
taskqueue_drain_all(txq->tq);
taskqueue_free(txq->tq);
+ bus_dmamap_destroy(txq->info->dma_tag, txq->dma_map);
}
static void
@@ -826,6 +870,11 @@ setup_txqs(device_t dev, struct netfront_info *info,
for (i = 0; i <= NET_TX_RING_SIZE; i++) {
txq->mbufs[i] = (void *) ((u_long) i+1);
txq->grant_ref[i] = GRANT_REF_INVALID;
+ m_tag_setup(&txq->refcount_tag[i].tag,
+ MTAG_COOKIE, MTAG_REFCOUNT,
+ sizeof(txq->refcount_tag[i]) -
+ sizeof(txq->refcount_tag[i].tag));
+ txq->refcount_tag[i].tag.m_tag_free = &tag_free;
}
txq->mbufs[NET_TX_RING_SIZE] = (void *)0;
@@ -874,10 +923,18 @@ setup_txqs(device_t dev, struct netfront_info *info,
device_printf(dev, "xen_intr_alloc_and_bind_local_port
failed\n");
goto fail_bind_port;
}
+
+ error = bus_dmamap_create(info->dma_tag, 0, &txq->dma_map);
+ if (error != 0) {
+ device_printf(dev, "failed to create dma map\n");
+ goto fail_dma_map;
+ }
}
return (0);
+fail_dma_map:
+ xen_intr_unbind(&txq->xen_intr_handle);
fail_bind_port:
taskqueue_drain_all(txq->tq);
fail_start_thread:
@@ -1041,7 +1098,7 @@ xn_release_tx_bufs(struct netfront_txq *txq)
if (txq->mbufs_cnt < 0) {
panic("%s: tx_chain_cnt must be >= 0", __func__);
}
- m_free(m);
+ mbuf_release(m);
}
}
@@ -1311,7 +1368,7 @@ xn_txeof(struct netfront_txq *txq)
txq->mbufs[id] = NULL;
add_id_to_freelist(txq->mbufs, id);
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Wed, Dec 29, 2021 at 05:13:00PM +0800, G.R. wrote: > > > > I think this is hitting a KASSERT, could you paste the text printed as > > part of the panic (not just he backtrace)? > > > > Sorry this is taking a bit of time to solve. > > > > Thanks! > > > Sorry that I didn't make it clear in the first place. > It is the same cross boundary assertion. I see. After looking at the code it seems like sglist will coalesce contiguous physical ranges without taking page boundaries into account, which is not suitable for our purpose here. I guess I will either have to modify sglist, or switch to using bus_dma. The main problem with using bus_dma is that it will require bigger changes to netfront I think. Thanks, Roger.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
> > I think this is hitting a KASSERT, could you paste the text printed as > part of the panic (not just he backtrace)? > > Sorry this is taking a bit of time to solve. > > Thanks! > Sorry that I didn't make it clear in the first place. It is the same cross boundary assertion. Also sorry about the email format if it mess up in your side. I am typing in the Gmail app and don't find a way to switch to plain text. >
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
Adding xen-devel back. On Wed, Dec 29, 2021 at 01:44:18AM +0800, G.R. wrote: > On Tue, Dec 28, 2021 at 3:05 AM Roger Pau Monné wrote: > > > > On Sun, Dec 26, 2021 at 02:06:55AM +0800, G.R. wrote: > > > > > Thanks. I've raised this on freensd-net for advice [0]. IMO netfront > > > > > shouldn't receive an mbuf that crosses a page boundary, but if that's > > > > > indeed a legit mbuf I will figure out the best way to handle it. > > > > > > > > > > I have a clumsy patch (below) that might solve this, if you want to > > > > > give it a try. > > > > > > > > Applied the patch and it worked like a charm! > > > > Thank you so much for your quick help! > > > > Wish you a wonderful holiday! > > > > > > I may have said too quickly... > > > With the patch I can attach the iscsi disk and neither the dom0 nor > > > the NAS domU complains this time. > > > But when I attempt to mount the attached disk it reports I/O errors > > > randomly. > > > By randomly I mean different disks behave differently... > > > I don't see any error logs from kernels this time. > > > (most of the iscsi disks are NTFS FS and mounted through the user mode > > > fuse library) > > > But since I have a local backup copy of the image, I can confirm that > > > mounting that backup image does not result in any I/O error. > > > Looks like something is still broken here... > > > > Indeed. That patch was likely too simple, and didn't properly handle > > the split of mbuf data buffers. > > > > I have another version based on using sglist, which I think it's also > > a worthwhile change for netfront. Can you please give it a try? I've > > done a very simple test and seems fine, but you certainly have more > > interesting cases. > > > > You will have to apply it on top of a clean tree, without any of the > > other patches applied. > > Unfortunately this new version is even worse. > It not only does not fix the known issue on iSCSI, but also creating > regression on NFS. > The regression on NFS is kind of random that it takes a > non-deterministic time to show up. > Here is a stack trace for reference: > db:0:kdb.enter.default> bt > Tracing pid 1696 tid 100622 td 0xf800883d5740 > kdb_enter() at kdb_enter+0x37/frame 0xfe009f80d900 > vpanic() at vpanic+0x197/frame 0xfe009f80d950 > panic() at panic+0x43/frame 0xfe009f80d9b0 > xn_txq_mq_start_locked() at xn_txq_mq_start_locked+0x5bc/frame > 0xfe009f80da50 I think this is hitting a KASSERT, could you paste the text printed as part of the panic (not just he backtrace)? Sorry this is taking a bit of time to solve. Thanks!
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Sun, Dec 26, 2021 at 02:06:55AM +0800, G.R. wrote:
> > > Thanks. I've raised this on freensd-net for advice [0]. IMO netfront
> > > shouldn't receive an mbuf that crosses a page boundary, but if that's
> > > indeed a legit mbuf I will figure out the best way to handle it.
> > >
> > > I have a clumsy patch (below) that might solve this, if you want to
> > > give it a try.
> >
> > Applied the patch and it worked like a charm!
> > Thank you so much for your quick help!
> > Wish you a wonderful holiday!
>
> I may have said too quickly...
> With the patch I can attach the iscsi disk and neither the dom0 nor
> the NAS domU complains this time.
> But when I attempt to mount the attached disk it reports I/O errors randomly.
> By randomly I mean different disks behave differently...
> I don't see any error logs from kernels this time.
> (most of the iscsi disks are NTFS FS and mounted through the user mode
> fuse library)
> But since I have a local backup copy of the image, I can confirm that
> mounting that backup image does not result in any I/O error.
> Looks like something is still broken here...
Indeed. That patch was likely too simple, and didn't properly handle
the split of mbuf data buffers.
I have another version based on using sglist, which I think it's also
a worthwhile change for netfront. Can you please give it a try? I've
done a very simple test and seems fine, but you certainly have more
interesting cases.
You will have to apply it on top of a clean tree, without any of the
other patches applied.
Thanks, Roger.
---
diff --git a/sys/dev/xen/netfront/netfront.c b/sys/dev/xen/netfront/netfront.c
index 8dba5a8dc6d5..37ea7b1fa059 100644
--- a/sys/dev/xen/netfront/netfront.c
+++ b/sys/dev/xen/netfront/netfront.c
@@ -33,6 +33,8 @@ __FBSDID("$FreeBSD$");
#include "opt_inet.h"
#include "opt_inet6.h"
+#include
+
#include
#include
#include
@@ -40,6 +42,7 @@ __FBSDID("$FreeBSD$");
#include
#include
#include
+#include
#include
#include
#include
@@ -199,6 +202,12 @@ struct netfront_txq {
struct taskqueue*tq;
struct task defrtask;
+ struct sglist *segments;
+ struct mbuf_refcount {
+ struct m_tagtag;
+ u_int count;
+ } refcount_tag[NET_TX_RING_SIZE + 1];
+
boolfull;
};
@@ -301,6 +310,38 @@ xn_get_rx_ref(struct netfront_rxq *rxq, RING_IDX ri)
return (ref);
}
+#define MTAG_REFCOUNT 0
+
+static void mbuf_grab(uint32_t cookie, struct mbuf *m)
+{
+ struct mbuf_refcount *ref;
+
+ ref = (struct mbuf_refcount *)m_tag_locate(m, cookie, MTAG_REFCOUNT,
+ NULL);
+ KASSERT(ref != NULL, ("Cannot find refcount"));
+ ref->count++;
+}
+
+static void mbuf_release(uint32_t cookie, struct mbuf *m)
+{
+ struct mbuf_refcount *ref;
+
+ ref = (struct mbuf_refcount *)m_tag_locate(m, cookie, MTAG_REFCOUNT,
+ NULL);
+ KASSERT(ref != NULL, ("Cannot find refcount"));
+ KASSERT(ref->count > 0, ("Invalid reference count"));
+
+ if(--ref->count == 0)
+ m_freem(m);
+}
+
+static void tag_free(struct m_tag *t)
+{
+ struct mbuf_refcount *ref = (struct mbuf_refcount *)t;
+
+ KASSERT(ref->count == 0, ("Free mbuf tag with pending refcnt"));
+}
+
#define IPRINTK(fmt, args...) \
printf("[XEN] " fmt, ##args)
#ifdef INVARIANTS
@@ -778,7 +819,7 @@ disconnect_txq(struct netfront_txq *txq)
static void
destroy_txq(struct netfront_txq *txq)
{
-
+ sglist_free(txq->segments);
free(txq->ring.sring, M_DEVBUF);
buf_ring_free(txq->br, M_DEVBUF);
taskqueue_drain_all(txq->tq);
@@ -826,6 +867,11 @@ setup_txqs(device_t dev, struct netfront_info *info,
for (i = 0; i <= NET_TX_RING_SIZE; i++) {
txq->mbufs[i] = (void *) ((u_long) i+1);
txq->grant_ref[i] = GRANT_REF_INVALID;
+ m_tag_setup(&txq->refcount_tag[i].tag,
+ (unsigned long)txq, MTAG_REFCOUNT,
+ sizeof(txq->refcount_tag[i]) -
+ sizeof(txq->refcount_tag[i].tag));
+ txq->refcount_tag[i].tag.m_tag_free = &tag_free;
}
txq->mbufs[NET_TX_RING_SIZE] = (void *)0;
@@ -874,10 +920,18 @@ setup_txqs(device_t dev, struct netfront_info *info,
device_printf(dev, "xen_intr_alloc_and_bind_local_port
failed\n");
goto fail_bind_port;
}
+
+ txq->segments = sglist_alloc(MAX_TX_REQ_FRAGS, M_WAITOK);
+ if (txq->segments == NULL) {
+ device_printf(dev, "failed to allocate sglist\n");
+ goto fail_sglist;
+ }
}
return (0);
+fail_sglist:
+ xen_intr_unbind(&txq->xen_intr_handle);
fail_bind_port:
taskqueue_drai
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
> > Thanks. I've raised this on freensd-net for advice [0]. IMO netfront > > shouldn't receive an mbuf that crosses a page boundary, but if that's > > indeed a legit mbuf I will figure out the best way to handle it. > > > > I have a clumsy patch (below) that might solve this, if you want to > > give it a try. > > Applied the patch and it worked like a charm! > Thank you so much for your quick help! > Wish you a wonderful holiday! I may have said too quickly... With the patch I can attach the iscsi disk and neither the dom0 nor the NAS domU complains this time. But when I attempt to mount the attached disk it reports I/O errors randomly. By randomly I mean different disks behave differently... I don't see any error logs from kernels this time. (most of the iscsi disks are NTFS FS and mounted through the user mode fuse library) But since I have a local backup copy of the image, I can confirm that mounting that backup image does not result in any I/O error. Looks like something is still broken here...
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
> > Please find the trace and the kernel CL below. > > Note, the domU get stuck into a bootloop with this assertion as the > > situation will come back after domU restart and only dom0 reboot can > > get the situation back to normal. > > The trace I captured below is within the boot loop. I suspect the > > initial trigger may look different. Will give it another try soon. > > I think I figured out the cause of the boot loop. It was not due to some mystery offender packet from the FreeBSD domU surviving across reboot, but because the windows iSCSI initiator keeps retrying :-). That said, I did pay some prices figuring this out. The boot-loop seems to have brought my box into a weird state that those disks behind the controller keep detaching soon after NAS domU reboot. Rebooting the physical host does not help this time. I was almost desperate but thankfully running the NAS on the physical host directly still works. And switching it back to domU together with config reloading fixed this problem :-) Not sure if it was the domU config being corrupted or something left sticky in the PCI pass-through? > Thanks. I've raised this on freensd-net for advice [0]. IMO netfront > shouldn't receive an mbuf that crosses a page boundary, but if that's > indeed a legit mbuf I will figure out the best way to handle it. > > I have a clumsy patch (below) that might solve this, if you want to > give it a try. Applied the patch and it worked like a charm! Thank you so much for your quick help! Wish you a wonderful holiday! Thanks, G.R.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Thu, Dec 23, 2021 at 11:49:08PM +0800, G.R. wrote:
> On Wed, Dec 22, 2021 at 3:13 AM Roger Pau Monné wrote:
>
> > Could you build a debug kernel with the following patch applied and
> > give me the trace when it explodes?
>
> Please find the trace and the kernel CL below.
> Note, the domU get stuck into a bootloop with this assertion as the
> situation will come back after domU restart and only dom0 reboot can
> get the situation back to normal.
> The trace I captured below is within the boot loop. I suspect the
> initial trigger may look different. Will give it another try soon.
>
> FreeBSD 12.2-RELEASE-p11 #0 c8625d629c3(truenas/12.0-stable)-dirty:
> Wed Dec 22 20:26:46 UTC 2021
> The repo is here: https://github.com/freenas/os.git
>
> db:0:kdb.enter.default> bt
> Tracing pid 0 tid 101637 td 0xf80069cc4000
> kdb_enter() at kdb_enter+0x37/frame 0xfe009f121460
> vpanic() at vpanic+0x197/frame 0xfe009f1214b0
> panic() at panic+0x43/frame 0xfe009f121510
> xn_txq_mq_start_locked() at xn_txq_mq_start_locked+0x4c6/frame
> 0xfe009f121580
> xn_txq_mq_start() at xn_txq_mq_start+0x84/frame 0xfe009f1215b0
> ether_output_frame() at ether_output_frame+0xb4/frame 0xfe009f1215e0
> ether_output() at ether_output+0x6a5/frame 0xfe009f121680
> ip_output() at ip_output+0x1319/frame 0xfe009f1217e0
> tcp_output() at tcp_output+0x1dbf/frame 0xfe009f121980
> tcp_usr_send() at tcp_usr_send+0x3c9/frame 0xfe009f121a40
> sosend_generic() at sosend_generic+0x440/frame 0xfe009f121af0
> sosend() at sosend+0x66/frame 0xfe009f121b20
> icl_send_thread() at icl_send_thread+0x44e/frame 0xfe009f121bb0
> fork_exit() at fork_exit+0x80/frame 0xfe009f121bf0
> fork_trampoline() at fork_trampoline+0xe/frame 0xfe009f121bf0
Thanks. I've raised this on freensd-net for advice [0]. IMO netfront
shouldn't receive an mbuf that crosses a page boundary, but if that's
indeed a legit mbuf I will figure out the best way to handle it.
I have a clumsy patch (below) that might solve this, if you want to
give it a try.
Regards, Roger.
[0] https://lists.freebsd.org/archives/freebsd-net/2021-December/001179.html
---
diff --git a/sys/dev/xen/netfront/netfront.c b/sys/dev/xen/netfront/netfront.c
index 87bc3ecfc4dd..c8f807778b75 100644
--- a/sys/dev/xen/netfront/netfront.c
+++ b/sys/dev/xen/netfront/netfront.c
@@ -1529,6 +1529,35 @@ xn_count_frags(struct mbuf *m)
return (nfrags);
}
+static inline int fragment(struct mbuf *m)
+{
+ while (m != NULL) {
+ vm_offset_t offset = mtod(m, vm_offset_t) & PAGE_MASK;
+
+ if (offset + m->m_len > PAGE_SIZE) {
+ /* Split mbuf because it crosses a page boundary. */
+ struct mbuf *m_new = m_getcl(M_NOWAIT, MT_DATA, 0);
+
+ if (m_new == NULL)
+ return (ENOMEM);
+
+ m_copydata(m, 0, m->m_len - (PAGE_SIZE - offset),
+ mtod(m_new, caddr_t));
+
+ /* Set adjusted mbuf sizes. */
+ m_new->m_len = m->m_len - (PAGE_SIZE - offset);
+ m->m_len = PAGE_SIZE - offset;
+
+ /* Insert new mbuf into chain. */
+ m_new->m_next = m->m_next;
+ m->m_next = m_new;
+ }
+ m = m->m_next;
+ }
+
+ return (0);
+}
+
/**
* Given an mbuf chain, make sure we have enough room and then push
* it onto the transmit ring.
@@ -1541,6 +1570,12 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct
mbuf *m_head)
struct ifnet *ifp = np->xn_ifp;
u_int nfrags;
int otherend_id;
+ int rc;
+
+ /* Fragment if mbuf crosses a page boundary. */
+ rc = fragment(m_head);
+ if (rc != 0)
+ return (rc);
/**
* Defragment the mbuf if necessary.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Wed, Dec 22, 2021 at 3:13 AM Roger Pau Monné wrote: > Could you build a debug kernel with the following patch applied and > give me the trace when it explodes? Please find the trace and the kernel CL below. Note, the domU get stuck into a bootloop with this assertion as the situation will come back after domU restart and only dom0 reboot can get the situation back to normal. The trace I captured below is within the boot loop. I suspect the initial trigger may look different. Will give it another try soon. FreeBSD 12.2-RELEASE-p11 #0 c8625d629c3(truenas/12.0-stable)-dirty: Wed Dec 22 20:26:46 UTC 2021 The repo is here: https://github.com/freenas/os.git db:0:kdb.enter.default> bt Tracing pid 0 tid 101637 td 0xf80069cc4000 kdb_enter() at kdb_enter+0x37/frame 0xfe009f121460 vpanic() at vpanic+0x197/frame 0xfe009f1214b0 panic() at panic+0x43/frame 0xfe009f121510 xn_txq_mq_start_locked() at xn_txq_mq_start_locked+0x4c6/frame 0xfe009f121580 xn_txq_mq_start() at xn_txq_mq_start+0x84/frame 0xfe009f1215b0 ether_output_frame() at ether_output_frame+0xb4/frame 0xfe009f1215e0 ether_output() at ether_output+0x6a5/frame 0xfe009f121680 ip_output() at ip_output+0x1319/frame 0xfe009f1217e0 tcp_output() at tcp_output+0x1dbf/frame 0xfe009f121980 tcp_usr_send() at tcp_usr_send+0x3c9/frame 0xfe009f121a40 sosend_generic() at sosend_generic+0x440/frame 0xfe009f121af0 sosend() at sosend+0x66/frame 0xfe009f121b20 icl_send_thread() at icl_send_thread+0x44e/frame 0xfe009f121bb0 fork_exit() at fork_exit+0x80/frame 0xfe009f121bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe009f121bf0
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Wed, Dec 22, 2021 at 02:19:03AM +0800, G.R. wrote:
> > > I omitted all operational details with the assumption that you are
> > > familiar
> > > with TrueNAS and iSCSI setup.
> >
> > Not really. Ideally I would like a way to reproduce that can be done
> > using iperf, nc or similar simple command line tool, without requiring
> > to setup iSCSI.
> I think it would be tricky then. The problem hide itself well enough
> that I wasn't
> aware soon after upgrading since everything else works flawlessly --
> nfs, ssh, web etc.
>
> > Can you also paste the output of `ifconfig xn0`?
> Here it is:
> xn0: flags=8843 metric 0 mtu 1500
> options=503
> ether 00:18:3c:51:6e:4c
> inet 192.168.1.9 netmask 0xff00 broadcast 192.168.1.255
> media: Ethernet manual
> status: active
> nd6 options=1
>
> >
> > If I provided a patch for the FreeBSD kernel, would you be able to
> > apply and test it?
> Probably. I did this before when your XEN support for freeBSD was not
> available out-of-box.
> Just need to recreate all the required environments to apply the patch.
Could you build a debug kernel with the following patch applied and
give me the trace when it explodes?
Thanks, Roger.
---
diff --git a/sys/dev/xen/netfront/netfront.c b/sys/dev/xen/netfront/netfront.c
index fd2d97a7c70c..87bc3ecfc4dd 100644
--- a/sys/dev/xen/netfront/netfront.c
+++ b/sys/dev/xen/netfront/netfront.c
@@ -1519,8 +1519,12 @@ xn_count_frags(struct mbuf *m)
{
int nfrags;
- for (nfrags = 0; m != NULL; m = m->m_next)
+ for (nfrags = 0; m != NULL; m = m->m_next) {
+ KASSERT(
+ (mtod(m, vm_offset_t) & PAGE_MASK) + m->m_len <= PAGE_SIZE,
+ ("mbuf fragment crosses a page boundary"));
nfrags++;
+ }
return (nfrags);
}
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
> > I omitted all operational details with the assumption that you are familiar > > with TrueNAS and iSCSI setup. > > Not really. Ideally I would like a way to reproduce that can be done > using iperf, nc or similar simple command line tool, without requiring > to setup iSCSI. I think it would be tricky then. The problem hide itself well enough that I wasn't aware soon after upgrading since everything else works flawlessly -- nfs, ssh, web etc. > Can you also paste the output of `ifconfig xn0`? Here it is: xn0: flags=8843 metric 0 mtu 1500 options=503 ether 00:18:3c:51:6e:4c inet 192.168.1.9 netmask 0xff00 broadcast 192.168.1.255 media: Ethernet manual status: active nd6 options=1 > > If I provided a patch for the FreeBSD kernel, would you be able to > apply and test it? Probably. I did this before when your XEN support for freeBSD was not available out-of-box. Just need to recreate all the required environments to apply the patch. BTW, uname -a gives me the following; >12.2-RELEASE-p11 FreeBSD 12.2-RELEASE-p11 75566f060d4(HEAD) TRUENAS amd64 Thanks, Timothy
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Tue, Dec 21, 2021 at 01:13:43AM +0800, G.R. wrote: > First of all, thank you for your quick response, Juergen and Roger. > I just realized that I run into mail forwarding issue from sourceforge > mail alias service, and only found the responses when I checked the > list archive. As a result, I have to manually merge Roger's response > to reply... > > > > I have to admit that this trial process is blind as I have no idea > > > which component in the combo is to be blamed. Is it a bug in the > > > backend-driver, frontend-driver or the hypervisor itself? Or due to > > > incompatible versions? Any suggestion on other diagnose ideas (e.g. > > > debug logs) will be welcome, while I work on the planned experiments. > > > > This is a bug in FreeBSD netfront, so no matter which Linux or Xen > > version you use. > > > > Does it make a difference if you disable TSO and LRO from netfront? > > > > $ ifconfig xn0 -tso -lro > It does not, the fatal error still show up after this command. Thanks for testing. > > > > Do you have instructions I can follow in order to try to reproduce the > > issue? > I don't know if there are any special details in my setup. > Hopefully I don't miss anything useful: > 1. Build a TrueNAS 12.0U7 DOM-U by flushing the OS image into a vdisk > 2. Create / import a zfs pool to the DOM-U > 3. Create and share some file based iSCSI extents on the pool > 4. Mount the iSCSI extent through some initiator clients. > The domU xn0 should be disabled immediately after step #4. > > I omitted all operational details with the assumption that you are familiar > with TrueNAS and iSCSI setup. Not really. Ideally I would like a way to reproduce that can be done using iperf, nc or similar simple command line tool, without requiring to setup iSCSI. > For step #4, I can reproduce it with both ipxe initiator and the win7 > built-in client. > As a result, I assume the client version does not matter. > For #2, I actually have a physical disk and controller assigned to DOM-U. > But I suspect this is probably irrelevant. > For #3, I'm not sure if the content in the extent matters. > So far I have been testing the same extent, which is formatted as an NTFS > disk. Can you also paste the output of `ifconfig xn0`? If I provided a patch for the FreeBSD kernel, would you be able to apply and test it? Thanks, Roger.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
First of all, thank you for your quick response, Juergen and Roger. I just realized that I run into mail forwarding issue from sourceforge mail alias service, and only found the responses when I checked the list archive. As a result, I have to manually merge Roger's response to reply... > > I have to admit that this trial process is blind as I have no idea > > which component in the combo is to be blamed. Is it a bug in the > > backend-driver, frontend-driver or the hypervisor itself? Or due to > > incompatible versions? Any suggestion on other diagnose ideas (e.g. > > debug logs) will be welcome, while I work on the planned experiments. > > This is a bug in FreeBSD netfront, so no matter which Linux or Xen > version you use. > > Does it make a difference if you disable TSO and LRO from netfront? > > $ ifconfig xn0 -tso -lro It does not, the fatal error still show up after this command. > > Do you have instructions I can follow in order to try to reproduce the > issue? I don't know if there are any special details in my setup. Hopefully I don't miss anything useful: 1. Build a TrueNAS 12.0U7 DOM-U by flushing the OS image into a vdisk 2. Create / import a zfs pool to the DOM-U 3. Create and share some file based iSCSI extents on the pool 4. Mount the iSCSI extent through some initiator clients. The domU xn0 should be disabled immediately after step #4. I omitted all operational details with the assumption that you are familiar with TrueNAS and iSCSI setup. For step #4, I can reproduce it with both ipxe initiator and the win7 built-in client. As a result, I assume the client version does not matter. For #2, I actually have a physical disk and controller assigned to DOM-U. But I suspect this is probably irrelevant. For #3, I'm not sure if the content in the extent matters. So far I have been testing the same extent, which is formatted as an NTFS disk. > > Thanks, Roger. > > > > Thanks, > > G.R.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Sun, Dec 19, 2021 at 02:35:56AM +0800, G.R. wrote: > Hi all, > > I ran into the following error report in the DOM0 kernel after a recent > upgrade: > [ 501.840816] vif vif-1-0 vif1.0: Cross page boundary, txp->offset: > 2872, size: 1460 > [ 501.840828] vif vif-1-0 vif1.0: fatal error; disabling device > [ 501.841076] xenbr0: port 2(vif1.0) entered disabled state > Once this error happens, the DOM-U behind this vif is no-longer > accessible. And recreating the same DOM-U does not fix the problem. > Only a reboot on the physical host machine helps. > > The problem showed up after a recent upgrade on the DOM-U OS from > FreeNAS 11.3 to TrueNAS 12.0U7 and breaks the iSCSI service while > leaving other services like NFS intact. > The underlying OS for the NAS is FreeBSD, version 11.3 and 12.2 respectively. > So far I have tried the following combos: > - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 11.3 DOM-U: Good > - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed > - Linux 5.10 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed > - Linux 5.10 DOM0 + XEN 4.11 + FreeBSD 12.2 DOM-U: Regressed > > I plan to try out the XEN 4.14 version which is the latest I can get > from the distro (Debian). > If that still does not fix the problem, I would build the 4.16 version > from source as my last resort. > > I have to admit that this trial process is blind as I have no idea > which component in the combo is to be blamed. Is it a bug in the > backend-driver, frontend-driver or the hypervisor itself? Or due to > incompatible versions? Any suggestion on other diagnose ideas (e.g. > debug logs) will be welcome, while I work on the planned experiments. This is a bug in FreeBSD netfront, so no matter which Linux or Xen version you use. Does it make a difference if you disable TSO and LRO from netfront? $ ifconfig xn0 -tso -lro Do you have instructions I can follow in order to try to reproduce the issue? Thanks, Roger.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On Sun, Dec 19, 2021 at 2:35 AM G.R. wrote: > > Hi all, > > I ran into the following error report in the DOM0 kernel after a recent > upgrade: > [ 501.840816] vif vif-1-0 vif1.0: Cross page boundary, txp->offset: > 2872, size: 1460 > [ 501.840828] vif vif-1-0 vif1.0: fatal error; disabling device > [ 501.841076] xenbr0: port 2(vif1.0) entered disabled state > Once this error happens, the DOM-U behind this vif is no-longer > accessible. And recreating the same DOM-U does not fix the problem. > Only a reboot on the physical host machine helps. > > The problem showed up after a recent upgrade on the DOM-U OS from > FreeNAS 11.3 to TrueNAS 12.0U7 and breaks the iSCSI service while > leaving other services like NFS intact. To clarify -- mounting iSCSI disk will cause the problem immediately. > The underlying OS for the NAS is FreeBSD, version 11.3 and 12.2 respectively. > So far I have tried the following combos: > - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 11.3 DOM-U: Good > - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed > - Linux 5.10 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed > - Linux 5.10 DOM0 + XEN 4.11 + FreeBSD 12.2 DOM-U: Regressed - Linux 5.10 DOM0 + XEN 4.14 + FreeBSD 12.2 DOM-U: Regressed > > I plan to try out the XEN 4.14 version which is the latest I can get > from the distro (Debian). I just upgraded to Debian bullseye (11) from buster (10) and migrated to XEN4.14 as a result. The syndrome persists, unfortunately. BTW, my Dom0 kernel is a custom built version. Does any kernel config could contribute to this problem? > If that still does not fix the problem, I would build the 4.16 version > from source as my last resort. > > I have to admit that this trial process is blind as I have no idea > which component in the combo is to be blamed. Is it a bug in the > backend-driver, frontend-driver or the hypervisor itself? Or due to > incompatible versions? Any suggestion on other diagnose ideas (e.g. > debug logs) will be welcome, while I work on the planned experiments. > > Thanks, > G.R.
Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0
On 18.12.21 19:35, G.R. wrote: Hi all, I ran into the following error report in the DOM0 kernel after a recent upgrade: [ 501.840816] vif vif-1-0 vif1.0: Cross page boundary, txp->offset: 2872, size: 1460 [ 501.840828] vif vif-1-0 vif1.0: fatal error; disabling device The dom0 network backend has detected an inconsistency in the data received from the domU's frontend. In this case a request's memory buffer crossed a page boundary, which is not allowed. There has been a recent change in the xen netback driver to stop the interface in such conditions, as such invalid requests are regarded to be malicious and might lead to crashes in dom0. So this issue should be reported to FreeBSD maintainers in order to have the Xen netfornt driver fixed there. [ 501.841076] xenbr0: port 2(vif1.0) entered disabled state Once this error happens, the DOM-U behind this vif is no-longer accessible. And recreating the same DOM-U does not fix the problem. Only a reboot on the physical host machine helps. The problem showed up after a recent upgrade on the DOM-U OS from FreeNAS 11.3 to TrueNAS 12.0U7 and breaks the iSCSI service while leaving other services like NFS intact. The underlying OS for the NAS is FreeBSD, version 11.3 and 12.2 respectively. So far I have tried the following combos: - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 11.3 DOM-U: Good - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed - Linux 5.10 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed - Linux 5.10 DOM0 + XEN 4.11 + FreeBSD 12.2 DOM-U: Regressed This information (especially the FreeBSD version affected) is probably important for the FreeBSD maintainers. I plan to try out the XEN 4.14 version which is the latest I can get from the distro (Debian). If that still does not fix the problem, I would build the 4.16 version from source as my last resort. Xen is NOT to blame here. Juergen OpenPGP_0xB0DE9DD628BF132F.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Possible bug? DOM-U network stopped working after fatal error reported in DOM0
Hi all, I ran into the following error report in the DOM0 kernel after a recent upgrade: [ 501.840816] vif vif-1-0 vif1.0: Cross page boundary, txp->offset: 2872, size: 1460 [ 501.840828] vif vif-1-0 vif1.0: fatal error; disabling device [ 501.841076] xenbr0: port 2(vif1.0) entered disabled state Once this error happens, the DOM-U behind this vif is no-longer accessible. And recreating the same DOM-U does not fix the problem. Only a reboot on the physical host machine helps. The problem showed up after a recent upgrade on the DOM-U OS from FreeNAS 11.3 to TrueNAS 12.0U7 and breaks the iSCSI service while leaving other services like NFS intact. The underlying OS for the NAS is FreeBSD, version 11.3 and 12.2 respectively. So far I have tried the following combos: - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 11.3 DOM-U: Good - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed - Linux 5.10 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed - Linux 5.10 DOM0 + XEN 4.11 + FreeBSD 12.2 DOM-U: Regressed I plan to try out the XEN 4.14 version which is the latest I can get from the distro (Debian). If that still does not fix the problem, I would build the 4.16 version from source as my last resort. I have to admit that this trial process is blind as I have no idea which component in the combo is to be blamed. Is it a bug in the backend-driver, frontend-driver or the hypervisor itself? Or due to incompatible versions? Any suggestion on other diagnose ideas (e.g. debug logs) will be welcome, while I work on the planned experiments. Thanks, G.R.
