Re: pci passthrough - VF reset at boot is dropping assigned MAC
On Mon, 2011-04-25 at 10:28 -0600, David Ahern wrote: Running qemu-kvm.git as of today (ffce28f, April 18, 2011) the virtual function passed to the VM is losing its assigned mac address. That is, prior to launching qemu-kvm, the following command is run to set the MAC address: ip link set dev eth2 vf 0 mac 02:12:34:56:79:20 Yet, when the VM boots the MAC address is random which is what happens when the VF is reset. Looking through the commit logs between 0.13.0 -- the version in Fedora 14 -- and latest git I found the following: commit d9488459ff2ab113293586c1c36b1679bb15deee Author: Alex Williamson alex.william...@redhat.com Date: Thu Mar 17 15:24:31 2011 -0600 device-assignment: Reset device on system reset On system reset, we currently try to quiesce DMA by clearing the command register. This assumes that nothing re-enables bus master support without first de-programming the device. Use a bigger hammer to help the guest not shoot itself by issuing a function reset via sysfs on each system reset. Signed-off-by: Alex Williamson alex.william...@redhat.com Acked-by: Chris Wright chr...@redhat.com Signed-off-by: Marcelo Tosatti mtosa...@redhat.com Is this the cause of the MAC address reset and is this behavior intended? Ugh, I hope not, it's certainly not an intended side effect. Can you see if the problem still happens if you revert this patch? If it does, we might need more device specific reset functions to save and restore that extra bit of state. I assume this is still the 82576 VF you were asking about before? Thanks, Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: pci passthrough - VF reset at boot is dropping assigned MAC
On 04/25/11 10:37, Alex Williamson wrote: On Mon, 2011-04-25 at 10:28 -0600, David Ahern wrote: Running qemu-kvm.git as of today (ffce28f, April 18, 2011) the virtual function passed to the VM is losing its assigned mac address. That is, prior to launching qemu-kvm, the following command is run to set the MAC address: ip link set dev eth2 vf 0 mac 02:12:34:56:79:20 Yet, when the VM boots the MAC address is random which is what happens when the VF is reset. Looking through the commit logs between 0.13.0 -- the version in Fedora 14 -- and latest git I found the following: commit d9488459ff2ab113293586c1c36b1679bb15deee Author: Alex Williamson alex.william...@redhat.com Date: Thu Mar 17 15:24:31 2011 -0600 device-assignment: Reset device on system reset On system reset, we currently try to quiesce DMA by clearing the command register. This assumes that nothing re-enables bus master support without first de-programming the device. Use a bigger hammer to help the guest not shoot itself by issuing a function reset via sysfs on each system reset. Signed-off-by: Alex Williamson alex.william...@redhat.com Acked-by: Chris Wright chr...@redhat.com Signed-off-by: Marcelo Tosatti mtosa...@redhat.com Is this the cause of the MAC address reset and is this behavior intended? Ugh, I hope not, it's certainly not an intended side effect. Can you see if the problem still happens if you revert this patch? If it does, I commented out the write() in the reset function and indeed the mac address was not reset on VM boot. we might need more device specific reset functions to save and restore that extra bit of state. I assume this is still the 82576 VF you were asking about before? Thanks, Yes. I got distracted end of last week. Response to that thread coming soon. David Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: pci passthrough - VF reset at boot is dropping assigned MAC
On Mon, 2011-04-25 at 10:41 -0600, David Ahern wrote: On 04/25/11 10:37, Alex Williamson wrote: On Mon, 2011-04-25 at 10:28 -0600, David Ahern wrote: Running qemu-kvm.git as of today (ffce28f, April 18, 2011) the virtual function passed to the VM is losing its assigned mac address. That is, prior to launching qemu-kvm, the following command is run to set the MAC address: ip link set dev eth2 vf 0 mac 02:12:34:56:79:20 Yet, when the VM boots the MAC address is random which is what happens when the VF is reset. Looking through the commit logs between 0.13.0 -- the version in Fedora 14 -- and latest git I found the following: commit d9488459ff2ab113293586c1c36b1679bb15deee Author: Alex Williamson alex.william...@redhat.com Date: Thu Mar 17 15:24:31 2011 -0600 device-assignment: Reset device on system reset On system reset, we currently try to quiesce DMA by clearing the command register. This assumes that nothing re-enables bus master support without first de-programming the device. Use a bigger hammer to help the guest not shoot itself by issuing a function reset via sysfs on each system reset. Signed-off-by: Alex Williamson alex.william...@redhat.com Acked-by: Chris Wright chr...@redhat.com Signed-off-by: Marcelo Tosatti mtosa...@redhat.com Is this the cause of the MAC address reset and is this behavior intended? Ugh, I hope not, it's certainly not an intended side effect. Can you see if the problem still happens if you revert this patch? If it does, I commented out the write() in the reset function and indeed the mac address was not reset on VM boot. Ok, here's what I see on my system: # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 # modprobe -r igbvf # echo 1 /sys/bus/pci/devices/:01:11.5/reset # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c So yes, it does change. However, if I set the VF mac instead of using a randomly generated one, I get: # modprobe -r igbvf # ip link set eth2 vf 6 mac 02:00:10:91:73:01 # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c igbvf :01:11.5: Address: 02:00:10:91:73:01 # modprobe -r igbvf # echo 1 /sys/bus/pci/devices/:01:11.5/reset # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c igbvf :01:11.5: Address: 02:00:10:91:73:01 igbvf :01:11.5: Address: 02:00:10:91:73:01 So now it sticks. You're going to get random mac addresses on the VFs every time you reload the igb driver (ie. ever boot) anyway (at least with these sr-iov cards), so if you need consistent macs, they probably need to be set before launching the VM anyway. Thanks, Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: pci passthrough - VF reset at boot is dropping assigned MAC
On 04/25/11 11:30, Alex Williamson wrote: So yes, it does change. However, if I set the VF mac instead of using a randomly generated one, I get: # modprobe -r igbvf # ip link set eth2 vf 6 mac 02:00:10:91:73:01 # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c igbvf :01:11.5: Address: 02:00:10:91:73:01 # modprobe -r igbvf # echo 1 /sys/bus/pci/devices/:01:11.5/reset # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c igbvf :01:11.5: Address: 02:00:10:91:73:01 igbvf :01:11.5: Address: 02:00:10:91:73:01 So now it sticks. You're going to get random mac addresses on the VFs every time you reload the igb driver (ie. ever boot) anyway (at least with these sr-iov cards), so if you need consistent macs, they probably need to be set before launching the VM anyway. Thanks, You lost me on this. I do not have the igbvf driver loaded in the host, only the guest. I am setting the MAC address for the VF in the host before launching the VM. The host's igb driver gets loaded at boot only. David Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: pci passthrough - VF reset at boot is dropping assigned MAC
On 04/25/11 11:30, Alex Williamson wrote: # modprobe -r igbvf # ip link set eth2 vf 6 mac 02:00:10:91:73:01 # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c igbvf :01:11.5: Address: 02:00:10:91:73:01 # modprobe -r igbvf # echo 1 /sys/bus/pci/devices/:01:11.5/reset # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c igbvf :01:11.5: Address: 02:00:10:91:73:01 igbvf :01:11.5: Address: 02:00:10:91:73:01 So now it sticks. You're going to get random mac addresses on the VFs every time you reload the igb driver (ie. ever boot) anyway (at least with these sr-iov cards), so if you need consistent macs, they probably need to be set before launching the VM anyway. Thanks, Alex Ok, I was able to repeat the above commands from the host command line. However, when qemu-kvm starts the MAC is reset. # ip link show | less 2: eth2: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:12:34:56:80:20 -- that's the MAC address I set I start qemu-kvm (unpatched version) and the host side sees the address changed: # ip link show | less 2: eth2: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff vf 0 MAC 7a:17:3f:98:0f:db Can you try that aspect on your end - seeing if the MAC address maintains after starting qemu-kvm? David -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: pci passthrough - VF reset at boot is dropping assigned MAC
On Mon, 2011-04-25 at 12:04 -0600, David Ahern wrote: On 04/25/11 11:30, Alex Williamson wrote: # modprobe -r igbvf # ip link set eth2 vf 6 mac 02:00:10:91:73:01 # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c igbvf :01:11.5: Address: 02:00:10:91:73:01 # modprobe -r igbvf # echo 1 /sys/bus/pci/devices/:01:11.5/reset # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c igbvf :01:11.5: Address: 02:00:10:91:73:01 igbvf :01:11.5: Address: 02:00:10:91:73:01 So now it sticks. You're going to get random mac addresses on the VFs every time you reload the igb driver (ie. ever boot) anyway (at least with these sr-iov cards), so if you need consistent macs, they probably need to be set before launching the VM anyway. Thanks, Alex Ok, I was able to repeat the above commands from the host command line. However, when qemu-kvm starts the MAC is reset. # ip link show | less 2: eth2: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:12:34:56:80:20 -- that's the MAC address I set I start qemu-kvm (unpatched version) and the host side sees the address changed: # ip link show | less 2: eth2: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff vf 0 MAC 7a:17:3f:98:0f:db Can you try that aspect on your end - seeing if the MAC address maintains after starting qemu-kvm? I don't see this happening on my system, once manually set the mac never changes. I can restart and reset the VM and the host and guest both continue seeing the set mac address. I tested it with both a recent rhel6.1 host kernel as well as upstream 2.6.39-rc4. If I switch to a VF with an unset mac, those will change on each VM reset or restart. Thanks, Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: pci passthrough - VF reset at boot is dropping assigned MAC
On 04/25/11 12:36, Alex Williamson wrote: On Mon, 2011-04-25 at 12:04 -0600, David Ahern wrote: On 04/25/11 11:30, Alex Williamson wrote: # modprobe -r igbvf # ip link set eth2 vf 6 mac 02:00:10:91:73:01 # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c igbvf :01:11.5: Address: 02:00:10:91:73:01 # modprobe -r igbvf # echo 1 /sys/bus/pci/devices/:01:11.5/reset # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c igbvf :01:11.5: Address: 02:00:10:91:73:01 igbvf :01:11.5: Address: 02:00:10:91:73:01 So now it sticks. You're going to get random mac addresses on the VFs every time you reload the igb driver (ie. ever boot) anyway (at least with these sr-iov cards), so if you need consistent macs, they probably need to be set before launching the VM anyway. Thanks, Alex Ok, I was able to repeat the above commands from the host command line. However, when qemu-kvm starts the MAC is reset. # ip link show | less 2: eth2: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:12:34:56:80:20 -- that's the MAC address I set I start qemu-kvm (unpatched version) and the host side sees the address changed: # ip link show | less 2: eth2: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff vf 0 MAC 7a:17:3f:98:0f:db Can you try that aspect on your end - seeing if the MAC address maintains after starting qemu-kvm? I don't see this happening on my system, once manually set the mac never changes. I can restart and reset the VM and the host and guest both continue seeing the set mac address. I tested it with both a recent rhel6.1 host kernel as well as upstream 2.6.39-rc4. If I switch to a VF with an unset mac, those will change on each VM reset or restart. Blacklist igbvf in the host and you will. That must be the difference: I was preventing the vf driver from loading in the host -- it's not needed there, so why load it? I rebooted for a fresh run. Loaded the igbvf driver before starting the VM using my tools. With the igbvf driver loaded in the host the MAC address for the VF was not reset. As for why I blacklisted it -- udev. What a PITA with VFs. I saw the feature for Fedora 15 which should address this. David -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: pci passthrough - VF reset at boot is dropping assigned MAC
On Mon, 2011-04-25 at 13:12 -0600, David Ahern wrote: On 04/25/11 12:36, Alex Williamson wrote: On Mon, 2011-04-25 at 12:04 -0600, David Ahern wrote: On 04/25/11 11:30, Alex Williamson wrote: # modprobe -r igbvf # ip link set eth2 vf 6 mac 02:00:10:91:73:01 # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c igbvf :01:11.5: Address: 02:00:10:91:73:01 # modprobe -r igbvf # echo 1 /sys/bus/pci/devices/:01:11.5/reset # modprobe igbvf # dmesg | grep igbvf \:01\:11.5\: Address\: igbvf :01:11.5: Address: d2:c8:17:d6:97:f7 igbvf :01:11.5: Address: 4e:ee:2a:d8:12:7c igbvf :01:11.5: Address: 02:00:10:91:73:01 igbvf :01:11.5: Address: 02:00:10:91:73:01 So now it sticks. You're going to get random mac addresses on the VFs every time you reload the igb driver (ie. ever boot) anyway (at least with these sr-iov cards), so if you need consistent macs, they probably need to be set before launching the VM anyway. Thanks, Alex Ok, I was able to repeat the above commands from the host command line. However, when qemu-kvm starts the MAC is reset. # ip link show | less 2: eth2: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff vf 0 MAC 02:12:34:56:80:20 -- that's the MAC address I set I start qemu-kvm (unpatched version) and the host side sees the address changed: # ip link show | less 2: eth2: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff vf 0 MAC 7a:17:3f:98:0f:db Can you try that aspect on your end - seeing if the MAC address maintains after starting qemu-kvm? I don't see this happening on my system, once manually set the mac never changes. I can restart and reset the VM and the host and guest both continue seeing the set mac address. I tested it with both a recent rhel6.1 host kernel as well as upstream 2.6.39-rc4. If I switch to a VF with an unset mac, those will change on each VM reset or restart. Blacklist igbvf in the host and you will. That must be the difference: I was preventing the vf driver from loading in the host -- it's not needed there, so why load it? I already have it blacklisted. It's not needed if you're using the VFs they way we are, but there are other uses. I rebooted for a fresh run. Loaded the igbvf driver before starting the VM using my tools. With the igbvf driver loaded in the host the MAC address for the VF was not reset. As for why I blacklisted it -- udev. What a PITA with VFs. I saw the feature for Fedora 15 which should address this. Yes, my VM is up to renaming the VFs eth1340 since the mac changes every boot. I'm still confused though as I did a whole round of testing after a reboot where igbvf was never loaded and the set mac address stuck across VM restarts and resets. Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: pci passthrough - VF reset at boot is dropping assigned MAC
On 04/25/11 13:18, Alex Williamson wrote: I don't see this happening on my system, once manually set the mac never changes. I can restart and reset the VM and the host and guest both continue seeing the set mac address. I tested it with both a recent rhel6.1 host kernel as well as upstream 2.6.39-rc4. If I switch to a VF with an unset mac, those will change on each VM reset or restart. Blacklist igbvf in the host and you will. That must be the difference: I was preventing the vf driver from loading in the host -- it's not needed there, so why load it? I already have it blacklisted. It's not needed if you're using the VFs they way we are, but there are other uses. I rebooted for a fresh run. Loaded the igbvf driver before starting the VM using my tools. With the igbvf driver loaded in the host the MAC address for the VF was not reset. As for why I blacklisted it -- udev. What a PITA with VFs. I saw the feature for Fedora 15 which should address this. Yes, my VM is up to renaming the VFs eth1340 since the mac changes every boot. I'm still confused though as I did a whole round of testing after a reboot where igbvf was never loaded and the set mac address stuck across VM restarts and resets. Alex The resetting of the VM MAC address is fixed in 2.6.39-rc4, so it's a Fedora 14, 2.6.35.12 problem. David -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: pci passthrough - VF reset at boot is dropping assigned MAC
On 04/25/11 14:29, David Ahern wrote: On 04/25/11 13:18, Alex Williamson wrote: I don't see this happening on my system, once manually set the mac never changes. I can restart and reset the VM and the host and guest both continue seeing the set mac address. I tested it with both a recent rhel6.1 host kernel as well as upstream 2.6.39-rc4. If I switch to a VF with an unset mac, those will change on each VM reset or restart. Blacklist igbvf in the host and you will. That must be the difference: I was preventing the vf driver from loading in the host -- it's not needed there, so why load it? I already have it blacklisted. It's not needed if you're using the VFs they way we are, but there are other uses. I rebooted for a fresh run. Loaded the igbvf driver before starting the VM using my tools. With the igbvf driver loaded in the host the MAC address for the VF was not reset. As for why I blacklisted it -- udev. What a PITA with VFs. I saw the feature for Fedora 15 which should address this. Yes, my VM is up to renaming the VFs eth1340 since the mac changes every boot. I'm still confused though as I did a whole round of testing after a reboot where igbvf was never loaded and the set mac address stuck across VM restarts and resets. Alex The resetting of the VM MAC address is fixed in 2.6.39-rc4, so it's a Fedora 14, 2.6.35.12 problem. Just to finish this off. This is the patch that fixed the MAC address reset problem: commit a6b5ea353845b3f3d9ac4317c0b3be9cc37c259b Author: Greg Rose gregory.v.r...@intel.com Date: Sat Nov 6 05:42:59 2010 + igb: Warn on attempt to override administratively set MAC/VLAN Print a warning message to the system log when the VF attempts to override administratively set MAC/VLAN configuration. Signed-off-by: Greg Rose gregory.v.r...@intel.com Signed-off-by: Jeff Kirsher jeffrey.t.kirs...@intel.com David -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
