[ovirt-users] Re: Dropped RX Packets
Den 16 maj 2019 19:17 skrev Magnus Isaksson :Hello@strahil The packet drops are frequent, every time i run "ip -s link" on the guest there is new dropped packets, on the hosts it says "0" and in oVirt it says "0".I can run tcpdump on hosts and guests, but i don't know how to capture the dropped packets with tcpdump.There are no RX or TX errors anywhere, not on hosts, guests or switches.The connection drops are completely random, sometimes after a few minutes and sometimes after a couple of hours, really hard to narrow down, but this may be some errors in our customers network, they are investigating it now, so i will come back with that issue is it still persists.@OliverI tried this, unfortunately still same result, still dropping packets.@DarellI tried increasing the RX and TX buffer on the hosts, but the guests still drop packets.I am using dual 10G, setup in Active-Backup going to two switches, but the second switch is now turned off during the testing to narrow this down.I have experienced these types of issues, you know, in general on any platform, when frame sizes don't quite match up. From NIC to switch to router- the whole chain from you to your customer- is that investigated at all?Another thing to check might be the VM NIC "hardware", whether the issue goes away or changes, if you switch to an "e1000" instead of "virtio", or vice versa?/KRegards Magnus___Users mailing list -- users@ovirt.orgTo unsubscribe send an email to users-le...@ovirt.orgPrivacy Statement: https://www.ovirt.org/site/privacy-policy/oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TISY75JZ34AJAVV3B233WLCJ5PFBZRRL/___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3K7FGH5U557O6743Y5NIVJLLTLRYBNJ/
[ovirt-users] Re: Dropped RX Packets
Hello @strahil The packet drops are frequent, every time i run "ip -s link" on the guest there is new dropped packets, on the hosts it says "0" and in oVirt it says "0". I can run tcpdump on hosts and guests, but i don't know how to capture the dropped packets with tcpdump. There are no RX or TX errors anywhere, not on hosts, guests or switches. The connection drops are completely random, sometimes after a few minutes and sometimes after a couple of hours, really hard to narrow down, but this may be some errors in our customers network, they are investigating it now, so i will come back with that issue is it still persists. @Oliver I tried this, unfortunately still same result, still dropping packets. @Darell I tried increasing the RX and TX buffer on the hosts, but the guests still drop packets. I am using dual 10G, setup in Active-Backup going to two switches, but the second switch is now turned off during the testing to narrow this down. Regards Magnus ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TISY75JZ34AJAVV3B233WLCJ5PFBZRRL/
[ovirt-users] Re: Dropped RX Packets
Check your host for dropped packets as well. I had found that some of my older 10G cards were setting smaller buffers than they could, and using ethtool to set tx and rx buffers to their max values significantly improved things for those cards. And look at your switch to be sure it/they are not dropping packets for some reason. If you’re using dual 10g links, how do you have them configured on the host? > On May 16, 2019, at 9:38 AM, Oliver Riesener > wrote: > > Hi Magnus, > > I've had a bad **virtual** network card three times in the last five years. > Yes it' possible. > > I my case, NFS services didn't work as expected, but other services were ok. > > Today if this would happen again, i unplug and replug the VM nic. Like: > > GUI::Compute::VirtualMachines::VMname::Network Interfaces::nicN > -> Edit CardStatus -> Unplugged :: OK > -> Edit CardStatus -> Plugged :: OK > > HTH > > Oliver > > On 16.05.19 15:17, Magnus Isaksson wrote: >> Hello all! >> >> I'm having quite some trouble with VMs that have a large amount of dropped >> packets on RX. >> This, plus customers complain about short dropped connections, for example >> one customer has a SQL server and an other serevr connecting to it, and it >> is randomly dropping connections. Before they moved their VM:s to us they >> did not have any of these issues. >> >> Does anyone have an idea of what this can be due to? And how can i fix it? >> It is starting to be a deal breaker for our customers on whether they will >> stay with us or not. >> >> I was thinking of reinstalling the nodes with oVirt Node, instead of the >> full CentOS, would this perhaps fix the issue? >> >> The enviroment is: >> Huawei x6000 with 4 nodes >> Each node having Intel X722 network card and connecting with 10G (fiber) to >> a Juniper EX 4600. Storage via FC to a IBM FS900. >> Each node is running a full CentOS 7.6 connecting to a Engine 4.2.8.2 >> >> Regards >> Magnus >> ___ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-le...@ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/QXGQSKYBUCFPDCBIQVAAZAWFQX54A2BD/ > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/FQXYN3P2QD727ZKGCNDZCCOOJVJU52DU/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WDQLQVXY3YIPFB4NGK4QVKQU7XXWOV7Z/
[ovirt-users] Re: Dropped RX Packets
Hi Magnus, I've had a bad **virtual** network card three times in the last five years. Yes it' possible. I my case, NFS services didn't work as expected, but other services were ok. Today if this would happen again, i unplug and replug the VM nic. Like: GUI::Compute::VirtualMachines::VMname::Network Interfaces::nicN -> Edit CardStatus -> Unplugged :: OK -> Edit CardStatus -> Plugged :: OK HTH Oliver On 16.05.19 15:17, Magnus Isaksson wrote: Hello all! I'm having quite some trouble with VMs that have a large amount of dropped packets on RX. This, plus customers complain about short dropped connections, for example one customer has a SQL server and an other serevr connecting to it, and it is randomly dropping connections. Before they moved their VM:s to us they did not have any of these issues. Does anyone have an idea of what this can be due to? And how can i fix it? It is starting to be a deal breaker for our customers on whether they will stay with us or not. I was thinking of reinstalling the nodes with oVirt Node, instead of the full CentOS, would this perhaps fix the issue? The enviroment is: Huawei x6000 with 4 nodes Each node having Intel X722 network card and connecting with 10G (fiber) to a Juniper EX 4600. Storage via FC to a IBM FS900. Each node is running a full CentOS 7.6 connecting to a Engine 4.2.8.2 Regards Magnus ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QXGQSKYBUCFPDCBIQVAAZAWFQX54A2BD/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FQXYN3P2QD727ZKGCNDZCCOOJVJU52DU/
[ovirt-users] Re: Dropped RX Packets
Hi Magnus, do you notice any repetition there ? Does it happen completely random ? Usually to debug network issues you will need tcpdump from Guest, Host and the other side if possible.Is that an option ? Do you see in the host's tab those RX errors ? What is the output of "ip -s link" on the Guest ? Best Regards,Strahil Nikolov В четвъртък, 16 май 2019 г., 9:19:57 ч. Гринуич-4, Magnus Isaksson написа: Hello all! I'm having quite some trouble with VMs that have a large amount of dropped packets on RX. This, plus customers complain about short dropped connections, for example one customer has a SQL server and an other serevr connecting to it, and it is randomly dropping connections. Before they moved their VM:s to us they did not have any of these issues. Does anyone have an idea of what this can be due to? And how can i fix it? It is starting to be a deal breaker for our customers on whether they will stay with us or not. I was thinking of reinstalling the nodes with oVirt Node, instead of the full CentOS, would this perhaps fix the issue? The enviroment is: Huawei x6000 with 4 nodes Each node having Intel X722 network card and connecting with 10G (fiber) to a Juniper EX 4600. Storage via FC to a IBM FS900. Each node is running a full CentOS 7.6 connecting to a Engine 4.2.8.2 Regards Magnus ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QXGQSKYBUCFPDCBIQVAAZAWFQX54A2BD/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/SCZNYVKLR54USPJW3EYA2NX5IH7BZDR6/
