Do we need "amd_iommu=on" for AMD system anymore?

[Yalan Zhang]

Hi,

Since long time ago, to enable the SR-IOV VF pci passthrough function, I'm
always adding "amd_iommu=on" into kernel cmdline on AMD system.
But recently I found even I do not do this action, IOMMU is still enabled
by kernel on AMD system.
After searching, I found there is no such setting any more, refer to
https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/kernel-parameters.txt#L286

There are only 3 possible value as below:
amd_iommu= fullflush
off
force_isolation

Could anyone can help to confirm the changes?  Thank you!
And another question, it is said that the "iommu=pt" option is to improves
IO performance for devices in the host, it is not a must for VF PCI
passthrough, right?
I'm not sure about the user cases.

[Reference]
1. Reference about adding "amd_iommu=on", and it may be outdated:
http://dpdk-guide.gitlab.io/dpdk-guide/setup/iommu.html
2. On AMD system without adding "amd_iommu=on"  in the kernel cmdline, the
iommu is enabled:
# cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.el8.x86_64
root=/dev/mapper/rhel_hp--dl385g10--16-root ro crashkernel=auto
resume=/dev/mapper/rhel_hp--dl385g10--16-swap
rd.lvm.lv=rhel_hp-dl385g10-16/root
rd.lvm.lv=rhel_hp-dl385g10-16/swap console=ttyS0,115200n81

# dmesg | grep -i iommu
[3.712029] iommu: Default domain type: Passthrough
[6.736019] pci :00:00.2: IOMMU performance counters supported
...
[6.780040] pci :e0:00.2: IOMMU performance counters supported
[6.786740] pci :00:01.0: Adding to iommu group 0
[6.791876] pci :00:01.1: Adding to iommu group 1
[6.797015] pci :00:01.2: Adding to iommu group 2
[6.802150] pci :00:01.4: Adding to iommu group 3
...
[7.866463] pci :e0:00.2: Found IOMMU cap 0x40
[7.920222] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4
counters/bank).
[7.927428] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4
counters/bank)
...


---
Best Regards,
Yalan Zhang
IRC: yalzhang

Re: Virtual Bridge "Network" for Sandbox

[Paul O'Rorke]

Hi Peter,

I had not anticipated this much response.  I do have VLANs on my 
switch(s).  We are a modest operation and I configure without the use of 
any config tool as such.


I would love to see your configuration, even if for Puppet, I expect it 
will be informative/helpful!


Thanks for the offer.

*Paul O'Rorke*/ 




On 2020-06-29 1:01 p.m., Peter Crowther wrote:
Paul, if you can set up a VLAN on your network infrastructure between 
the two hosts, I'll share the recipe I use with Open VSwitch.  We 
trunk a VLAN between our hosts for sandboxed guests, setting up a OVS 
bridge on each host that handles guests but also has a connection onto 
the VLAN.  Are you using any configuration tool?  I've got this in 
Puppet but should be able to convert to a shell script.


Cheers,

- Peter

On Mon, 29 Jun 2020 at 17:45, Paul O'Rorke > wrote:


Thanks Laine,

I will take a look at Open vSwitch, it looks interesting.

I am a generalist, I need to know enough about a lot of things to
get many different tasks done, but do not have the in depth
knowledge required to "patch" anything.  If I manage to wrangle a
working solution should I post it?

Needless to say I would be supportive of said feature being
implemented by those more competent than I...

Jocularity aside, thanks for the heads up on Open vSwitch.

*Paul O'Rorke*


On 2020-06-29 9:13 a.m., Laine Stump wrote:

On 6/29/20 11:01 AM, Paul O'Rorke wrote:

Hi all,

I couldn't find any documentation on this, hopefully someone can
point me in the right direction.

I recently set up a sand-boxed environment for our developers.
There are domain controller(s), workstations and servers in
there.  The whole thing is running on a single host using a
"Virtual Network" defined in virt-manager on that host.

Now I find I want to add more guests and there are not enough
resources on this one host.  Can I somehow make this Virtual
Network available to two hosts?  I do not want to move to a
bridged network and have to physically join the two hosts with a
discrete link when they are already on the same subnet at the
host level.

Is that possible?


You might be able to this using OpenvSwitch (iow "probably can,
but I don't know the details" :-)) but libvirt doesn't have
anything to set it up for you; you would need to create and
configure the OVS switch outside of libvirt, then attach the
libvirt guests to that switch (using "
...  ...")

I've idly thought about having this as a libvirt feature over the
years, but as I never have that many guests, it was never a
personal priority, and it wasn't immediately clear what was the
best way to handle, e.g. DHCP, and routing to the outside.
Definitely "patches are welcome" though :-)

Re: Virtual Bridge "Network" for Sandbox

[Paul O'Rorke]

Thanks Laine,

I got another suggestion, off list, about using vxlans that I will also 
look into.


I would be more than happy to post what I eventually decide to use for 
production.


regards

*Paul O'Rorke*


On 2020-06-29 12:46 p.m., Laine Stump wrote:

On 6/29/20 12:43 PM, Paul O'Rorke wrote:

Thanks Laine,

I will take a look at Open vSwitch, it looks interesting.

I am a generalist, I need to know enough about a lot of things to get 
many different tasks done, but do not have the in depth knowledge 
required to "patch" anything.  If I manage to wrangle a working 
solution should I post it?


Even a list of the steps you took to implement it manuall external to 
libvirt would be useful. Maybe that would inspire someone else to add 
support in libvirt virtual networks. We used to put stuff like that in 
the wiki, but I think the preferred location has changed / is changing 
and I'm not sure at the moment what the new norm is.




Needless to say I would be supportive of said feature being 
implemented by those more competent than I...


Jocularity aside, thanks for the heads up on Open vSwitch.

*Paul O'Rorke*


On 2020-06-29 9:13 a.m., Laine Stump wrote:

On 6/29/20 11:01 AM, Paul O'Rorke wrote:

Hi all,

I couldn't find any documentation on this, hopefully someone can 
point me in the right direction.


I recently set up a sand-boxed environment for our developers. 
There are domain controller(s), workstations and servers in there.  
The whole thing is running on a single host using a "Virtual 
Network" defined in virt-manager on that host.


Now I find I want to add more guests and there are not enough 
resources on this one host.  Can I somehow make this Virtual 
Network available to two hosts?  I do not want to move to a bridged 
network and have to physically join the two hosts with a discrete 
link when they are already on the same subnet at the host level.


Is that possible?


You might be able to this using OpenvSwitch (iow "probably can, but 
I don't know the details" :-)) but libvirt doesn't have anything to 
set it up for you; you would need to create and configure the OVS 
switch outside of libvirt, then attach the libvirt guests to that 
switch (using " ... type='openvswitch'> ...")


I've idly thought about having this as a libvirt feature over the 
years, but as I never have that many guests, it was never a personal 
priority, and it wasn't immediately clear what was the best way to 
handle, e.g. DHCP, and routing to the outside. Definitely "patches 
are welcome" though :-)

Re: Virtual Bridge "Network" for Sandbox

[Peter Crowther]

Paul, if you can set up a VLAN on your network infrastructure between the
two hosts, I'll share the recipe I use with Open VSwitch.  We trunk a VLAN
between our hosts for sandboxed guests, setting up a OVS bridge on each
host that handles guests but also has a connection onto the VLAN.  Are you
using any configuration tool?  I've got this in Puppet but should be able
to convert to a shell script.

Cheers,

- Peter

On Mon, 29 Jun 2020 at 17:45, Paul O'Rorke 
wrote:

> Thanks Laine,
>
> I will take a look at Open vSwitch, it looks interesting.
>
> I am a generalist, I need to know enough about a lot of things to get many
> different tasks done, but do not have the in depth knowledge required to
> "patch" anything.  If I manage to wrangle a working solution should I post
> it?
>
> Needless to say I would be supportive of said feature being implemented by
> those more competent than I...
>
> Jocularity aside, thanks for the heads up on Open vSwitch.
>
> *Paul O'Rorke*
>
>
> On 2020-06-29 9:13 a.m., Laine Stump wrote:
>
> On 6/29/20 11:01 AM, Paul O'Rorke wrote:
>
> Hi all,
>
> I couldn't find any documentation on this, hopefully someone can point me
> in the right direction.
>
> I recently set up a sand-boxed environment for our developers. There are
> domain controller(s), workstations and servers in there.  The whole thing
> is running on a single host using a "Virtual Network" defined in
> virt-manager on that host.
>
> Now I find I want to add more guests and there are not enough resources on
> this one host.  Can I somehow make this Virtual Network available to two
> hosts?  I do not want to move to a bridged network and have to physically
> join the two hosts with a discrete link when they are already on the same
> subnet at the host level.
>
> Is that possible?
>
>
> You might be able to this using OpenvSwitch (iow "probably can, but I
> don't know the details" :-)) but libvirt doesn't have anything to set it up
> for you; you would need to create and configure the OVS switch outside of
> libvirt, then attach the libvirt guests to that switch (using " type='bridge'> ...  ...")
>
> I've idly thought about having this as a libvirt feature over the years,
> but as I never have that many guests, it was never a personal priority, and
> it wasn't immediately clear what was the best way to handle, e.g. DHCP, and
> routing to the outside. Definitely "patches are welcome" though :-)
>
>

Re: Virtual Bridge "Network" for Sandbox

[Laine Stump]

On 6/29/20 12:43 PM, Paul O'Rorke wrote:

Thanks Laine,

I will take a look at Open vSwitch, it looks interesting.

I am a generalist, I need to know enough about a lot of things to get 
many different tasks done, but do not have the in depth knowledge 
required to "patch" anything.  If I manage to wrangle a working solution 
should I post it?


Even a list of the steps you took to implement it manuall external to 
libvirt would be useful. Maybe that would inspire someone else to add 
support in libvirt virtual networks. We used to put stuff like that in 
the wiki, but I think the preferred location has changed / is changing 
and I'm not sure at the moment what the new norm is.




Needless to say I would be supportive of said feature being implemented 
by those more competent than I...


Jocularity aside, thanks for the heads up on Open vSwitch.

*Paul O'Rorke*


On 2020-06-29 9:13 a.m., Laine Stump wrote:

On 6/29/20 11:01 AM, Paul O'Rorke wrote:

Hi all,

I couldn't find any documentation on this, hopefully someone can 
point me in the right direction.


I recently set up a sand-boxed environment for our developers. There 
are domain controller(s), workstations and servers in there.  The 
whole thing is running on a single host using a "Virtual Network" 
defined in virt-manager on that host.


Now I find I want to add more guests and there are not enough 
resources on this one host.  Can I somehow make this Virtual Network 
available to two hosts?  I do not want to move to a bridged network 
and have to physically join the two hosts with a discrete link when 
they are already on the same subnet at the host level.


Is that possible?


You might be able to this using OpenvSwitch (iow "probably can, but I 
don't know the details" :-)) but libvirt doesn't have anything to set 
it up for you; you would need to create and configure the OVS switch 
outside of libvirt, then attach the libvirt guests to that switch 
(using " ...  
...")


I've idly thought about having this as a libvirt feature over the 
years, but as I never have that many guests, it was never a personal 
priority, and it wasn't immediately clear what was the best way to 
handle, e.g. DHCP, and routing to the outside. Definitely "patches are 
welcome" though :-)

Re: Virtual Bridge "Network" for Sandbox

[Paul O'Rorke]

Thanks Laine,

I will take a look at Open vSwitch, it looks interesting.

I am a generalist, I need to know enough about a lot of things to get 
many different tasks done, but do not have the in depth knowledge 
required to "patch" anything.  If I manage to wrangle a working solution 
should I post it?


Needless to say I would be supportive of said feature being implemented 
by those more competent than I...


Jocularity aside, thanks for the heads up on Open vSwitch.

*Paul O'Rorke*


On 2020-06-29 9:13 a.m., Laine Stump wrote:

On 6/29/20 11:01 AM, Paul O'Rorke wrote:

Hi all,

I couldn't find any documentation on this, hopefully someone can 
point me in the right direction.


I recently set up a sand-boxed environment for our developers. There 
are domain controller(s), workstations and servers in there.  The 
whole thing is running on a single host using a "Virtual Network" 
defined in virt-manager on that host.


Now I find I want to add more guests and there are not enough 
resources on this one host.  Can I somehow make this Virtual Network 
available to two hosts?  I do not want to move to a bridged network 
and have to physically join the two hosts with a discrete link when 
they are already on the same subnet at the host level.


Is that possible?


You might be able to this using OpenvSwitch (iow "probably can, but I 
don't know the details" :-)) but libvirt doesn't have anything to set 
it up for you; you would need to create and configure the OVS switch 
outside of libvirt, then attach the libvirt guests to that switch 
(using " ...  
...")


I've idly thought about having this as a libvirt feature over the 
years, but as I never have that many guests, it was never a personal 
priority, and it wasn't immediately clear what was the best way to 
handle, e.g. DHCP, and routing to the outside. Definitely "patches are 
welcome" though :-)

Re: Virtual Bridge "Network" for Sandbox

[Laine Stump]

On 6/29/20 11:01 AM, Paul O'Rorke wrote:

Hi all,

I couldn't find any documentation on this, hopefully someone can point 
me in the right direction.


I recently set up a sand-boxed environment for our developers. There are 
domain controller(s), workstations and servers in there.  The whole 
thing is running on a single host using a "Virtual Network" defined in 
virt-manager on that host.


Now I find I want to add more guests and there are not enough resources 
on this one host.  Can I somehow make this Virtual Network available to 
two hosts?  I do not want to move to a bridged network and have to 
physically join the two hosts with a discrete link when they are already 
on the same subnet at the host level.


Is that possible?


You might be able to this using OpenvSwitch (iow "probably can, but I 
don't know the details" :-)) but libvirt doesn't have anything to set it 
up for you; you would need to create and configure the OVS switch 
outside of libvirt, then attach the libvirt guests to that switch (using 
" ...  ...")


I've idly thought about having this as a libvirt feature over the years, 
but as I never have that many guests, it was never a personal priority, 
and it wasn't immediately clear what was the best way to handle, e.g. 
DHCP, and routing to the outside. Definitely "patches are welcome" 
though :-)

Virtual Bridge "Network" for Sandbox

[Paul O'Rorke]

Hi all,

I couldn't find any documentation on this, hopefully someone can point 
me in the right direction.


I recently set up a sand-boxed environment for our developers. There are 
domain controller(s), workstations and servers in there.  The whole 
thing is running on a single host using a "Virtual Network" defined in 
virt-manager on that host.


Now I find I want to add more guests and there are not enough resources 
on this one host.  Can I somehow make this Virtual Network available to 
two hosts?  I do not want to move to a bridged network and have to 
physically join the two hosts with a discrete link when they are already 
on the same subnet at the host level.


Is that possible?

--

*Paul O'Rorke*

Re: get contents of a storage volume in bytes

[Daniel P. Berrange]

On Sun, Jun 28, 2020 at 07:51:01PM +0530, Shashwat shagun wrote:
> Hi, i have two machines with libvirt installed. I want to move a storage
> volume from the 1st machine to 2nd machine with code.
> 
> I'm thinking of reading a storage volume into a byte array/stream then
> uploading through my own/custom handler (written in go) to the 2nd
> machine's custom handler which will then be writing the byte stream to
> libvirt storage volume.
> 
> Is there a libvirt function to read storage volume into byte stream?

You can use the Download & Upload methods on the StorageVol object

  https://godoc.org/libvirt.org/libvirt-go#StorageVol.Download
  https://godoc.org/libvirt.org/libvirt-go#StorageVol.Upload

Together with the stream object

  https://godoc.org/libvirt.org/libvirt-go#Stream

Regards,
Daniel
--
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: get contents of a storage volume in bytes

[Peter Krempa]

On Sun, Jun 28, 2020 at 19:51:01 +0530, Shashwat shagun wrote:
> Hi, i have two machines with libvirt installed. I want to move a storage
> volume from the 1st machine to 2nd machine with code.
> 
> I'm thinking of reading a storage volume into a byte array/stream then
> uploading through my own/custom handler (written in go) to the 2nd
> machine's custom handler which will then be writing the byte stream to
> libvirt storage volume.
> 
> Is there a libvirt function to read storage volume into byte stream?

https://libvirt.org/html/libvirt-libvirt-storage.html#virStorageVolDownload

and

https://libvirt.org/html/libvirt-libvirt-storage.html#virStorageVolUpload

for the counterpart of writing a storage volume.