[ovirt-devel] Re: virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)

[Richard W.M. Jones]

On Mon, Nov 09, 2020 at 09:16:33AM +0200, Yedidyah Bar David wrote:
> On Wed, Oct 14, 2020 at 8:54 AM Yedidyah Bar David  wrote:
> >
> > On Tue, Oct 13, 2020 at 8:40 PM Richard W.M. Jones  
> > wrote:
> > >
> > > On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote:
> > > > On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones  
> > > > wrote:
> > > > >
> > > > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > > > > > I think this is the right solution - when virt-something tool fails,
> > > > > > it should log the reason for the failure - the error that caused the
> > > > > > tool to fail. I'm not sure this is easy to do as the failing code
> > > > > > run inside a special VM. Maybe the code running in the VM should log
> > > > > > the output in a machine readable way, so once an error is detected
> > > > > > virt-something can report the error as the reason, without running
> > > > > > in debug mode.
> > > > >
> > > > > All the virt-* tools that I've written have a non-zero exit code and
> > > > > print an error message on stderr when they fail.  Errors from inside
> > > > > the appliance are propagated to the library and thence to the tool
> > > > > correctly.
> > > > >
> > > > > I think the best thing to do is:
> > > > >
> > > > >  - spool up stdout + stderr from the tool
> > > > >
> > > > >  - if the exit code != 0, save the spooled output for analysis
> > > > >
> > > > >  - if the exit code == 0, discard it (or keep it if you like)
> > > >
> > > > This is what we already do, and the result is not helpful. If you look
> > > > at the log message in the previous message, basically the only
> > > > info about the error is:
> > > >
> > > > libguestfs error: guestfs_launch failed
> > > >
> > > > I don't see what we can do with this error message.
> > >
> > > Right, so in this particular instance the error message would tell us
> > > that you should run libguestfs-test-tool because your qemu/kernel/etc
> > > is broken in some way :-/
> > >
> > > There's not a particularly good answer here if you don't want to ever
> > > use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
> > > libguestfs-test-tool if you see any error which matches the substring
> > > /guestfs_launch/ ?
> >
> > Another (orthogonal?) option:
> >
> > Make LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE log elsewhere, not to stdout/err
> > (e.g. some other file descriptor, or to a file passed via env or whatever).
> > This way, it might make sense for vdsm to always pass these vars, continue
> > logging all stdout/err, and log/keep debug/trace logs only on errors.
> 
> This now happened again:
> 
> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/565/
> 
> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/565/artifact/exported-artifacts/test_logs/basic-suite-master/lago-basic-suite-master-host-1/_var_log/vdsm/vdsm.log
> 
> 2020-11-09 01:05:42,031-0500 INFO  (jsonrpc/4) [api.host] FINISH
> getAllVmIoTunePolicies return={'status': {'code': 0, 'message':
> 'Done'}, 'io_tune_policies_dict':
> {'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
> 'current_values': [{'name': 'vda', 'path':
> '/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
> 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
> 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
> 'read_iops_sec': 0}}]}}} from=::1,34002 (api:54)
> 2020-11-09 01:05:42,038-0500 DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer]
> Return 'Host.getAllVmIoTunePolicies' in bridge with
> {'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
> 'current_values': [{'name': 'vda', 'path':
> '/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
> 'ioTune': {'total_bytes_sec': 0, &#x

[ovirt-devel] Re: virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)

[Richard W.M. Jones]

On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote:
> On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones  wrote:
> >
> > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > > I think this is the right solution - when virt-something tool fails,
> > > it should log the reason for the failure - the error that caused the
> > > tool to fail. I'm not sure this is easy to do as the failing code
> > > run inside a special VM. Maybe the code running in the VM should log
> > > the output in a machine readable way, so once an error is detected
> > > virt-something can report the error as the reason, without running
> > > in debug mode.
> >
> > All the virt-* tools that I've written have a non-zero exit code and
> > print an error message on stderr when they fail.  Errors from inside
> > the appliance are propagated to the library and thence to the tool
> > correctly.
> >
> > I think the best thing to do is:
> >
> >  - spool up stdout + stderr from the tool
> >
> >  - if the exit code != 0, save the spooled output for analysis
> >
> >  - if the exit code == 0, discard it (or keep it if you like)
> 
> This is what we already do, and the result is not helpful. If you look
> at the log message in the previous message, basically the only
> info about the error is:
> 
> libguestfs error: guestfs_launch failed
> 
> I don't see what we can do with this error message.

Right, so in this particular instance the error message would tell us
that you should run libguestfs-test-tool because your qemu/kernel/etc
is broken in some way :-/

There's not a particularly good answer here if you don't want to ever
use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
libguestfs-test-tool if you see any error which matches the substring
/guestfs_launch/ ?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/TPSPSMRWA5ZFV4SFVKIW7CCIKD5OAW6P/

[ovirt-devel] Re: virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)

[Richard W.M. Jones]

On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> I think this is the right solution - when virt-something tool fails,
> it should log the reason for the failure - the error that caused the
> tool to fail. I'm not sure this is easy to do as the failing code
> run inside a special VM. Maybe the code running in the VM should log
> the output in a machine readable way, so once an error is detected
> virt-something can report the error as the reason, without running
> in debug mode.

All the virt-* tools that I've written have a non-zero exit code and
print an error message on stderr when they fail.  Errors from inside
the appliance are propagated to the library and thence to the tool
correctly.

I think the best thing to do is:

 - spool up stdout + stderr from the tool

 - if the exit code != 0, save the spooled output for analysis

 - if the exit code == 0, discard it (or keep it if you like)

Without LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE stdout + stderr should be
quite minimal for all tools.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PF4WEIEIQFA7FA2ISYQHFFFG6J5JD2K4/

[ovirt-devel] Re: Disk sizes not updated on unmap/discard

[Richard W.M. Jones]

On Fri, Oct 02, 2020 at 01:57:04AM +0300, Nir Soffer wrote:
> On Wed, Sep 30, 2020 at 1:49 PM Tomáš Golembiovský  
> wrote:
> > Hi,
> >
> > currently, when we run virt-sparsify on VM or user runs VM with discard
> > enabled and when the disk is on block storage in qcow, the results are
> > not reflected in oVirt. The blocks get discarded, storage can reuse them
> > and reports correct allocation statistics, but oVirt does not. In oVirt
> > one can still see the original allocation for disk and storage domain as
> > it was before blocks were discarded. This is super-confusing to the
> > users because when they check after running virt-sparsify and see the
> > same values they think sparsification is not working. Which is not true.
> 
> This may be documentation issue. This is a known limitation of oVirt thin
> provisioned storage. We allocate space as needed, but we release the
> space only when a volume is deleted.
> 
> > It all seems to be because of our LVM layout that we have on storage
> > domain. The feature page for discard [1] suggests it could be solved by
> > running lvreduce. But this does not seem to be true. When blocks are
> > discarded the QCOW does not necessarily change its apparent size, the
> > blocks don't have to be removed from the end of the disk. So running
> > lvreduce is likely to remove valuable data.
> 
> We have an API to (safely) reduce a volume to optimal size:
> http://ovirt.github.io/ovirt-engine-api-model/master/#services/disk/methods/reduce
> 
> Reducing images depends on qcow2 image-end-offset. We can tell which
> is the highest offset used by inactive disk:
> https://github.com/oVirt/vdsm/blob/24f646383acb615b090078fc7aeddaf7097afe57/lib/vdsm/storage/blockVolume.py#L403
> 
> and reduce the logical volume to this size.
> 
> But this will not works since qcow2 image-end-offset is not decreased by
> 
> virt-sparsify --in-place

Right - this doesn't "defragment" the qcow2 file, ie. moving clusters
to the beginning - so (except by accident) it won't make the qcow2
file smaller.

Virt-sparsify in copying mode will actually do what you want, but
obviously is much more heavyweight and complex to use.

> So it is true that sparsify releases unused space on storage level, but it 
> does
> not decrease the qcow2 image allocation, so we cannot reduce the logical
> volumes.
> 
> > At the moment I don't see how we could achieve the correct values. If
> > anyone has any idea feel free to entertain me. The only option seems to
> > be to switch to LVM thin pools. Do we have any plans on doing that?
> 
> No, thin pools do not support clustering, this can be used only on a single
> host. oVirt lvm based volumes are accessed on multiple hosts at the same
> time.
> 
> Here is an example sparisfy test showing the issue:
> 
> Before writing data to new disk
> 
> guest:
> 
> # df -h /data
> Filesystem  Size  Used Avail Use% Mounted on
> /dev/sda110G  104M  9.9G   2% /data
> 
> storage:
> 
> $ ls -lhs /home/target/2/00
> 2.1G -rw-r--r--. 1 root root 100G Oct  2 00:57 /home/target/2/00
> 
> host:
> 
> # qemu-img info
> /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/42cf66df-43ad-4cfa-ab57-a943516155d1
> image: 
> /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/42cf66df-43ad-4cfa-ab57-a943516155d1
> file format: qcow2
> virtual size: 10 GiB (10737418240 bytes)
> disk size: 0 B
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> compression type: zlib
> lazy refcounts: false
> refcount bits: 16
> corrupt: false
> 
> # qemu-img check
> /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/42cf66df-43ad-4cfa-ab57-a943516155d1
> No errors were found on the image.
> 168/163840 = 0.10% allocated, 0.60% fragmented, 0.00% compressed clusters
> Image end offset: 12582912
> 
> 
> After writing 5g file to file system on this disk in the guest:
> 
> guest:
> 
> $ dd if=/dev/zero bs=8M count=640 of=/data/test oflag=direct
> conv=fsync status=progress
> 
> # df -h /data
> Filesystem  Size  Used Avail Use% Mounted on
> /dev/sda110G  5.2G  4.9G  52% /data
> 
> storage:
> 
> $ ls -lhs /home/target/2/00
> 7.1G -rw-r--r--. 1 root root 100G Oct  2 01:06 /home/target/2/00
> 
> host:
> 
> # qemu-img check
> /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/42cf66df-43ad-4cfa-ab57-a943516155d1
> No errors were found on the image.
> 82088/163840 = 50.10% allocated, 5.77% fragmented, 0.00% compressed clusters
> Image end offset: 5381423104
> 
> 
> After deleting the 5g file:
> 
> guest:
> 
> # df -h /data
> Filesystem  Size  Used Avail Use% Mounted on
> /dev/sda110G  104M  9.9G   2% /data
> 
> storage:
> 
> $ ls -lhs /home/target/2/00
> 7.1G -rw-r--r--. 1 root root 100G Oct  2 01:12 /home/target/2/00
> 
> host:
> 
> # qemu-img check
> /dev/27f2b637-ffb1-48f9-8f68-63ed227392b9/42cf66df-43ad-4cfa-ab57-a943516155d1
> No errors were found on the image.
> 82088/163840 = 50.10% allocated, 5.77% fragmented, 0.00% compressed clusters
> Image end offset: 5381423104
> 
> 
> After sparsi

[ovirt-devel] Re: [ARM64] Possiblity to support oVirt on ARM64

[Richard W.M. Jones]

On Wed, Jul 22, 2020 at 04:29:15PM +0800, Zhenyu Zheng wrote:
> Hi,
> 
> Any other comments for this topic?

libguestfs, nbdkit and virt-v2v have also worked (upstream)
for years out of the box on aarch64, so there should be no
problem there.

If you're interested in RHV (ie. oVirt downstream on RHEL)
then note we don't build virt-v2v for RHEL on aarch64.

Virt-v2v is not very useful on aarch64 since there is not
(yet) any public VMware version for aarch64, although VMware
do have an internal build and rumoured plans for ESXi on aarch64.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/X3SW2NQU6WV754FQXNVYGGOAYUNQD56C/

[ovirt-devel] Re: v2v: -o rhv-upload: Upload via NBD

[Richard W.M. Jones]

On Sun, Dec 09, 2018 at 07:11:28PM +0200, Nir Soffer wrote:
> I posted this patch for vdsm, adding NBD APIs:
> https://gerrit.ovirt.org/c/96079/

Great stuff.  What's the minimum version of oVirt which will have this
feature and what's the approximate date when we can assume most users
of oVirt will have this?

I don't have time now to implement the rest but I filed an RFE bug
referring back to your email:

https://bugzilla.redhat.com/show_bug.cgi?id=1657713

Rich.

> The main purpose of these APIs are enabling incremental restore, but they
> also
> enable a more efficient rhv-upload via NBD, and importing to thin disks,
> which is
> not possible with current solution.
> 
> The same idea can work for KubeVirt or other targets, minimizing specific
> target code.
> 
> Here is how rhv-upload can work using NBD:
> 
> 1. rhv-upload plugin separated to pre and post scripts
> 
> - pre - prepare disk and start image transfer
> - post - finialize image transfer, create vm, etc.
> 
> 2. rhr-upload-pre plugin create a transfer with transport="nbd"
> 
> POST /imagetransfers
> 
> 
> 
> upload
> raw
> nbd
> 
> 
> Engine does not implement  yet, but this should be an easy
> change.
> 
> This will use the new NBD APIs to export the disk using NBD over unix
> socket.
> We can support later also NBD over TLS/PSK.
> 
> Engine will return NBD url in transfer_url:
> 
> nbd:unix:/run/vdsm/nbd/.sock
> 
> 3. v2v use the trasfer_url to start qem-img with the NBD server:
> 
> nbdkit (vddk) -> qemu-img convert -> qemu-nbd
> 
> Note that nbdkit is removed from the rhv side of the pipeline. This is
> expected to
> improve the throughput significantly, since imageio is not very good with
> lot of
> small requests generated by qemu-img.
> 
> 4. rhv-upload-post script invoked to complete the transfer
> 
> What do you think?
> 
> Nir

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/NPMGQSAAB34VCTPHF6VOG76MLSGJSZPJ/

[ovirt-devel] Re: Mailing-Lists upgrade

[Richard W.M. Jones]

On Fri, May 11, 2018 at 05:55:57PM +0100, Richard W.M. Jones wrote:
> 
> This change has dropped the X-BeenThere header, ie:
> 
> X-BeenThere: devel@ovirt.org
> 
> X-BeenThere: us...@ovirt.org
> 
> which means filtering the mails is no longer possible.  There doesn't
> appear to be any other distinguishing header.

Oh wait, List-id ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

[ovirt-devel] Re: Mailing-Lists upgrade

[Richard W.M. Jones]

This change has dropped the X-BeenThere header, ie:

X-BeenThere: devel@ovirt.org

X-BeenThere: us...@ovirt.org

which means filtering the mails is no longer possible.  There doesn't
appear to be any other distinguishing header.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

Re: [ovirt-devel] [ovirt-users] Unremovable disks created through the API

[Richard W.M. Jones]

On Wed, Mar 07, 2018 at 03:12:58PM +0200, Arik Hadas wrote:
> If for some predefined period of time no new disk is added or an upload
> doesn't make any progress (assuming the uploads are done sequentially), to
> fail the import operation and that would roll back the resources (disks,
> VMs) that were created as part of the import process.

At the moment we're actually trying to remove the disk on failure.
However the disk_service.remove() API does nothing (doesn't even
fail).  Perhaps because the transfer isn't finalized on the error
path?

Anyway the code - which needs review - is:

https://www.redhat.com/archives/libguestfs/2018-March/msg00024.html

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [ovirt-users] Unremovable disks created through the API

[Richard W.M. Jones]

On Wed, Mar 07, 2018 at 01:26:39PM +0200, Arik Hadas wrote:
> Interesting, that contradicts my intuition - I would imagine that most of
> the things are actually known (the things that appear in the top-level part
> of the domain xml: memory size, memory size, num of CPUs, name,.. ) and
> only things that depend on the content of the disks or things that depend
> on installations during the conversion are unknown.
> But anyway, it is enough IMO to send the name, memory, CPU and size of the
> disks to present something useful to the user and make the necessary
> validations at that point.

Some of those things are known, but they didn't seem to me to be that
interesting for oVirt to know in advance.  In any case what's
precisely known before conversion is:

(1) The 'source' struct and sub-structs:

https://github.com/libguestfs/libguestfs/blob/ba53251ab912b8bac9e00c1022adc6ba9bdf70a3/v2v/types.mli#L59

(2) The 'overlay' struct (one per disk):

https://github.com/libguestfs/libguestfs/blob/ba53251ab912b8bac9e00c1022adc6ba9bdf70a3/v2v/types.mli#L175

Note only virtual disk size is known, which is near to useless for
provisioning storage.

(3) The 'target' struct (one per disk):

https://github.com/libguestfs/libguestfs/blob/ba53251ab912b8bac9e00c1022adc6ba9bdf70a3/v2v/types.mli#L191

What's unknown are guest capabilities (hence nothing about what
devices should be presented to the guest), inspection data, target bus
mapping, real size of disks, etc.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [ovirt-users] Unremovable disks created through the API

[Richard W.M. Jones]

On Wed, Mar 07, 2018 at 10:42:31AM +0200, Arik Hadas wrote:
> (Moving to devel-list)
> BTW, I think that the import process should include a preliminary phase
> where ovirt-engine is informed that the import process starts.
> 
> Currently, IIUC, the new process is designed to be:
> 1. virt-v2v uploads the disks
> 2. virt-v2v calls an API with the OVF configuration so ovirt-engine will
> add the VM and attach the uploaded disks to that VM

Yes, this is how it happens now, see:

https://www.redhat.com/archives/libguestfs/2018-March/msg00021.html

> IMHO, the process should be comprised of:
> 1. virt-v2v calls an API with the (probably partial since the OS and other
> things are unknown at that point) OVF configuration

Almost nothing is known at this point, I'm not sure what we could
provide.  Perhaps just number and virtual size of disks.  It doesn't
sound like it would be OVF, but something else.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Packages for Python SDK

[Richard W.M. Jones]

On Mon, Feb 12, 2018 at 04:46:18PM +0100, Tomáš Golembiovský wrote:
> On Mon, 12 Feb 2018 15:31:22 +
> "Richard W.M. Jones"  wrote:
> 
> > On Mon, Feb 12, 2018 at 02:26:09PM +0100, Tomáš Golembiovský wrote:
> > > Hi,
> > > 
> > > are there any plans to have the packages for Python SDK in base
> > > repositories of CentOS and Fedora? They seem to be available only from
> > > oVirt and Virt SIG repos. 
> > > 
> > > I saw that Juan used to build the v3 packages for Fedora.
> > > 
> > > The rationale behind is that virt-v2v is considering to add support for
> > > the disk upload API and would like to use Python snippets so as not to
> > > re-implement the calls in curl.  
> > 
> > Also where is the source RPM for python3-ovirt-engine-sdk4?
> > 
> > I got the python 2 package (python-ovirt-engine-sdk4) to rebuild on
> > Fedora starting from the CentOS source RPM found here:
> > 
> >   http://resources.ovirt.org/pub/ovirt-4.2/rpm/el7Server/SRPMS/
> > 
> > but I couldn't find the source RPM for the python3 package.
> 
> Here:
> 
> http://resources.ovirt.org/pub/ovirt-4.2-snapshot/rpm/fc27/SRPMS/python-ovirt-engine-sdk4-4.2.4-1.20180209gita93f0f2.fc27.src.rpm

That worked, thanks.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Packages for Python SDK

[Richard W.M. Jones]

On Mon, Feb 12, 2018 at 02:26:09PM +0100, Tomáš Golembiovský wrote:
> Hi,
> 
> are there any plans to have the packages for Python SDK in base
> repositories of CentOS and Fedora? They seem to be available only from
> oVirt and Virt SIG repos. 
> 
> I saw that Juan used to build the v3 packages for Fedora.
> 
> The rationale behind is that virt-v2v is considering to add support for
> the disk upload API and would like to use Python snippets so as not to
> re-implement the calls in curl.

Also where is the source RPM for python3-ovirt-engine-sdk4?

I got the python 2 package (python-ovirt-engine-sdk4) to rebuild on
Fedora starting from the CentOS source RPM found here:

  http://resources.ovirt.org/pub/ovirt-4.2/rpm/el7Server/SRPMS/

but I couldn't find the source RPM for the python3 package.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Removal of Export Storage Domain and virt-v2v

[Richard W.M. Jones]

On Wed, Jun 14, 2017 at 02:52:56PM +0100, Richard W.M. Jones wrote:
> Yaniv wrote:
> > And indeed, we won't deprecate it without a suitable replacement.
> > While you can already upload disks, we are working towards upload/download
> > of VMs.
> > See
> > http://www.ovirt.org/develop/release-management/features/virt/enhance-import-export-with-ova/
> 
> My concern with this (if I've understood it correctly) is it's going
> to be slow and require considerable extra storage on the conversion
> server.
> 
> At the moment -- using NFS -- we can stream from VMware to oVirt,
> requiring almost no local storage on the conversion server.

Thinking about this overnight, another way to do this would be to
allow the import of "OVA directories".  OVA doesn't need to be packed
into a tarball, the standard allows an OVA to be an unpacked directory
(directory containing disk images + OVF file).

Create a ‘/import’ directory on the NFS server, and allow OVA
directories to be uploaded to there.  We could stream the disk content
as now.

Then provide a REST API to import an OVA (directory) from /import on
the NFS server to complete to operation.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Removal of Export Storage Domain and virt-v2v

[Richard W.M. Jones]

Yaniv wrote:
> And indeed, we won't deprecate it without a suitable replacement.
> While you can already upload disks, we are working towards upload/download
> of VMs.
> See
> http://www.ovirt.org/develop/release-management/features/virt/enhance-import-export-with-ova/

My concern with this (if I've understood it correctly) is it's going
to be slow and require considerable extra storage on the conversion
server.

At the moment -- using NFS -- we can stream from VMware to oVirt,
requiring almost no local storage on the conversion server.

On Wed, Jun 14, 2017 at 03:30:12PM +0300, Arik Hadas wrote:
> On Wed, Jun 14, 2017 at 2:04 PM, Richard W.M. Jones 
> wrote:
> > Although virt-v2v can still work via the GUI, this isn't really
> > suitable for bulk, scripted upload of hundreds or thousands of VMs.
> >
> 
> We now enable triggering that process also via REST-API (this is mostly
> intended for importing from VMware using ManageIQ, which is expected to be
> available soon).

That requires virt-v2v to be installed on the oVirt nodes so it's not
quite a replacement for what you can do with ‘virt-v2v -o rhv’ today.

Can we have a REST API so that we can upload the disk images and OVF
metadata to some directory on the NFS server (eg. an ‘/import’
directory), and then the call the REST API which would complete the
import -- ie. update the database, and move the images to their final
places (bonus points if it can handle chown-ing the images too).

> Other than that, there is a rumor about introucing a backup-data-domain.
> Hopefully, that plan will also be shared publicly and we'll know more
> details about it. If it will be close in concept to the export domain, we
> may be able to replace the export domain with that domain in virt-v2v.

Any more details of this?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] Removal of Export Storage Domain and virt-v2v

[Richard W.M. Jones]

As you may know virt-v2v can use the Export Storage Domain (ESD) to
upload converted virtual machines to oVirt.  It was brought to my
attention yesterday that the ESD feature is being dropped, so this
will no longer work at some point in the future.  (BTW I would
appreciate notice if you're going to drop major features that we rely on.)

Although virt-v2v can still work via the GUI, this isn't really
suitable for bulk, scripted upload of hundreds or thousands of VMs.

The ESD method was never very good.  It was sort of an undocumented
back door into oVirt, and it was slow, and still required manual
intervention (after virt-v2v had done its job, you still needed to go
through the GUI and import the guests into the Data Domain).

What we really need is a fully scripted method to upload VMs --
metadata and disk images -- to oVirt.  Maybe one exists already?  If
not, what's the best way to do this?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [ovirt-users] Feature: enhanced OVA support

[Richard W.M. Jones]

On Sun, May 14, 2017 at 04:56:56PM +0300, Arik Hadas wrote:
> Hi everyone,
> 
> We would like to share our plan for extending the currently provided
> support for OVA files with:
> 1. Support for uploading OVA.
> 2. Support for exporting a VM/template as OVA.
> 3. Support for importing OVA that was generated by oVirt (today, we only
> support those that are VMware-compatible).
> 4. Support for downloading OVA.
> 
> This can be found on the feature page
> 
> .
> 
> Your feedback and cooperation will be highly appreciated.

The plan as stated seems fine, but I have some reservations which I
don't think are answered by the page:

(1) How will oVirt know the difference between an OVA generated
by oVirt and one generated by VMware (or indeed other sources)?
A VMware OVF has an XML comment:



but not any official metadata that I could see.

(By the way, I don't think importing via virt-v2v vs directly will be
any quicker.  The v2v conversion / device installation takes only a
fraction of the time.  Most of the time is consumed doing the format
conversion from VMDK to qcow2.  However you are correct that when you
know that the source is oVirt/KVM, you should not run virt-v2v.)

(2) I think you're going to have a lot of fun generating OVAs which
work on VMware.  As Yaniv says, the devices aren't the same so you'd
be having to do some virt-v2v -like driver installation / registry
modification.  Plus the OVF file is essentially a VMware data dump
encoded as XML.  OVF isn't a real standard.  I bet there are a million
strange corner cases.  Even writing VMDK files is full of pitfalls.

VMware has a reasonable V2V import tool (actually their P2V tooling is
very decent).  Of course it's proprietary, but then so is their
hypervisor.  Maybe oVirt can drive their tools?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] execCmd() and storing stdout and stderr in log file

[Richard W.M. Jones]

On Thu, Jul 14, 2016 at 02:40:06PM +0200, Tomáš Golembiovský wrote:
> This is something I have tried as first, as you can see here:
> 
> https://gerrit.ovirt.org/#/c/59834/4/lib/vdsm/v2v.py@404
> 
> The problem with this approach is that we don't get proper exit code from 
> virt-v2v
> because of the pipe. Fixing this in POSIX shell is not trivial and would lead
> to more complex shell code here. (Would be fairly easy to fix if we could
> resort to using bash here.)

Script mentioned previously also solves the exit code problem.
However you do need to use a shell for it (or perhaps reimplement that
wrapper script in a tiny bit of Python?).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] execCmd() and storing stdout and stderr in log file

[Richard W.M. Jones]

On Thu, Jul 14, 2016 at 03:07:29PM +0300, Nir Soffer wrote:
> This should work with execCmd, since the special subprocess.STDOUT
> parameter is handled in subprocess.Popen, and cpopen.CPopen inherit
> this code. However this is not tested with cpopen, so it may be broken.
> 
> But merging stdout and stderr is likely to break v2v output parser, and vdsm
> log is not the place for virt-v2v debug logs.
> 
> I understand that the issue is keeping virt-v2v debug logs (using --verbose?),
> and the logs are spread in stdout and stderr. Did you discuss this issue
> with Richard?

FWIW we just faced this same issue with virt-p2v (which runs virt-v2v
as a remote subprocess).  The resolution was to write a wrapper script
that separates out the stdout & stderr.  Stdout is displayed to the
user.  At the same time, stdout + stderr contains all the debugging
information and that is sent to a log file.  You can find the code
here:

https://github.com/libguestfs/libguestfs/blob/master/p2v/conversion.c#L959

There were also some corresponding changes to virt-v2v itself to more
sensibly apportion the output between stdout & stderr, but those are
all included in RHEL 7.3 (not 7.2).

The only problem was that if virt-v2v fails, the final error gets sent
to stderr (hence to the debugging log file) and not displayed to the
user, so on error the wrapper script has to dump out the last 50 lines
of the log file.

We also wrote a simple ANSI colour parser so that the full colourized
output of 'virt-v2v --colours' shows up in virt-p2v (ie. what comes
via stdout and is displayed to the user).  The code is:

https://github.com/libguestfs/libguestfs/blob/master/p2v/gui.c#L1782

> I would use tee to write stdout and stderr to an import log file, without
> changing the code checking import progress.
> 
> Maybe virt-v2v can add a --logfile option appending to given log file?

Tricky to implement.  In any case, starting with RHEL 7.3, you can
achieve the same thing using the ' >> log | tee -a log ' technique
used by the wrapper script, see link above.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] In-place sparsify support in VDSM

[Richard W.M. Jones]

I was working on this piece of code today:

https://github.com/libguestfs/libguestfs/blob/3fb35e66272ab675bb335a7ae1b6c0299b96bd18/src/launch-direct.c#L1377-L1459

which reminded me of this email thread.  The above function in
libguestfs encodes which versions of qemu and which block drivers in
qemu have discard support.

Unfortunately it doesn't encode the NFS knowledge nor the local
drive/partition business, because it's rather hard to detect those.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] In-place sparsify support in VDSM

[Richard W.M. Jones]

On Tue, May 17, 2016 at 09:50:07PM +0300, Nir Soffer wrote:
> On Tue, May 17, 2016 at 7:14 PM, Shmuel Melamud  wrote:
> 
> > Hi!
> >
> > There is an RFE being implemented currently (
> > https://bugzilla.redhat.com/show_bug.cgi?id=734120) to use --inplace
> > option in virt-sparsify to sparsify a disk image in-place, without copying
> > it.
> >
> > The problem is that in-place sparsify works on NFS only starting from NFS
> > 4.2, while the copying implementation supposedly works with any storage.
> >
> > From my point of view, it is better to remove the old code and start with
> > the new code that uses --inplace and just add to document that one will
> > need NFS >= 4.2 for sparsify feature to work. Although the old
> > implementation exists, it is not used currently so there are no actual
> > users that may be affected by the change. If it will be a crying need to
> > use sparsify on older NFS or some other incompatible storage, we can add it
> > later on the base of the new code. The old code is far from ideal and we
> > will need to rewrite it in any case, and there is not sense to do this
> > without real need.
> >
> 
> Richard,
> 
> Currently vdsm is using:
> 
> virt-sparsify --tmp prebuilt:tmp_vol --format src_format --convert
> dst_format src_vol dst_vol

> (--format and --dst_format are optional)
> 
> tmp_vol, src_vol, and dst_vol can be either file on nfs/glusterfs/other
> posix shared filesystem,
> or an lv created on top of lun (iscsi/fc).
> 
> can you confirm this method works on all the storage types I mentioned? or
> this depends
> on the underlying storage server?

Copying mode requires that the dst_vol supports sparseness.  The most
common case where this would *not* be true is partitions on local hard
disks.  You can't make a partition and/or local hard disk sparse.

However all of the ones you've mentioned above support sparseness, so
you should be good.

*If* you were using --inplace only, you could nuke the --tmp option,
and indeed all the code associated with "prebuilt" qcow2 files for
scratch space.

That's because --inplace mode uses hardly any temporary storage, so
you can just have it use regular /var/tmp or $TMPDIR.

However ...
 
> The new inplace method is much nicer, but something that works only on
> latest NFS version
> is not useful for our most important users, using block storage.

... it's a shame, but yes.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] OVA imports into virt-v2v (was: Re: [ovirt-users] virt-v2v integration feature)

[Richard W.M. Jones]

On Thu, Jul 10, 2014 at 10:11:24AM +0100, Richard W.M. Jones wrote:
> On Thu, Jul 10, 2014 at 10:41:16AM +0300, Itamar Heim wrote:
> > On 07/10/2014 10:29 AM, Sven Kieske wrote:
> > >
> > >
> > >Am 09.07.2014 20:30, schrieb Arik Hadas:
> > >>Hi All,
> > >>
> > >>The proposed feature will introduce a new process of import virtual 
> > >>machines from external systems using virt-v2v in oVirt.
> > >>I've created a wiki page that contains initial thoughts and design for it:
> > >>http://www.ovirt.org/Features/virt-v2v_Integration
> > >>
> > >>You are more than welcome to share your thoughts and insights.
> > >>
> > >>Thanks,
> > >>Arik
> > >
> > >Am I right that this still involves a full operational e.g. esxi host
> > >to import vmware vms?
> > >
> > >There is a huge user demand on a simpler process for just converting and
> > >importing an vmware disk image. This feature will not solve this
> > >use case, will it?
> > 
> > I agree it should. need to check if virt-v2v can cover this. if not,
> > need to fix it so it will...
> 
> >From live ESX, yes.
> 
> Offline VMware images (OVAs), not yet.  However patches are welcome.
> We quite literally have no one available to implement this.

Just catching up on this ..

virt-v2v now supports OVA imports, and it works pretty well.  Please
make sure you use the new version, available here:

https://www.redhat.com/archives/libguestfs/2014-May/msg00090.html

or in Fedora >= 21 or Debian experimental.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] virt-v2v intergration with vdsm

[Richard W.M. Jones]

On Thu, Oct 02, 2014 at 04:56:55AM -0400, Arik Hadas wrote:
> Rich, I remember that we came into conclusion that we don't have a faster
> way than doing the full conversion that will give us better estimation than
> taking the virtual size + some buffer, right?

No - see previous email.  We do better estimates than using just the
virtual size.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] virt-v2v intergration with vdsm

[Richard W.M. Jones]

On Thu, Oct 02, 2014 at 04:23:31AM -0400, Arik Hadas wrote:
> - the second option will be based on estimations of the actual size
>   of the converted disks and since all we know is their
>   virtual-sizes, we might allocated much more space that we need so
>   we'll need another phase of shrinking the disks according to the
>   amount of data that was actually written - in this case the whole
>   operation will be slower than it could be

Although estimation is difficult, it's a bit more sophisticated than
just the virtual size.  See:

https://github.com/libguestfs/libguestfs/blob/master/v2v/v2v.ml#L488

Actually I need to test this code on some sample guests to see how
close the estimate is to the reality.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] virt-v2v intergration with vdsm

[Richard W.M. Jones]

On Tue, Sep 30, 2014 at 01:19:46PM +0100, Richard W.M. Jones wrote:
>  - hostname of RHEL 5 Xen server

- Optional port number of ssh server on RHEL 5 Xen server

>  - username
>  - password
>  - name of the guest

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] virt-v2v intergration with vdsm

[Richard W.M. Jones]

To expand on the previous email, the web page is quite vague about
what precise sources you want to import from.

The ones supported by virt-v2v are:

 - VMware live vCenter: I assume you would want this.  (Note ESXi is
   mentioned in the web page, but virt-v2v does not support import
   from ESXi directly)

 - OVA file: As Shahar wrote this bit, I'll assume you'll want to use
   it.

 - RHEL 5 Xen: SSH key management is going to be a problem here.

 - Local disk files: Seems unlikely you'll want this.

 - Citrix Xen, Hyper-V, etc.: Requires development work.

 - Physical machines

VMware vCenter
--

See: http://libguestfs.org/virt-v2v.1.html#input-from-vmware-vcenter-server

The user will need to specify:
 - hostname of VMware vCenter server
 - optional port number
 - username
 - password
 - name of VMware Datacenter
 - hostname of VMware ESXi server [may be able to relax this in future]
 - whether or not to check HTTPS certs
 - name of the guest

As detailed in the link above, there are various virsh commands you
can run from VDSM to establish whether conversion is likely to work,
and to get a list of guests which you can present to the user.

OVA file


The user will need to specify:
 - OVA file (eg. location)

I've no idea how the file will end up being accessible to VDSM.  NFS mount?

RHEL 5 Xen
--

See: http://libguestfs.org/virt-v2v.1.html#input-from-rhel-5-xen

I think we're going to have a problem with ssh keys, because this mode
currently only works when the virt-v2v client can get passwordless
access to the RHEL 5 Xen server using ssh-agent (note that Kerberos
does not work).  We might fix this in future, but at the moment you're
somehow going to have to ensure that the oVirt node has a private key
which is listed in the authorized_keys in the RHEL 5 Xen server.  I
simply cannot imagine a user interface that is going to make this
usable.

I should also say that few people care about RHEL 5 Xen imports.
Citrix Xen imports OTOH, certainly, but those were out of scope for
current upstream development for RHEL 7.1.

Anyway, the user will have to specify:

 - hostname of RHEL 5 Xen server
 - username
 - password
 - name of the guest

There are various virsh commands you can run from VDSM to establish
whether conversion is likely to work, and to get a list of guests
which you can present to the user.

Physical machines
-

See also: http://libguestfs.org/virt-p2v.1.html
[Update from previous email: I have now updated this page to contain
the latest documentation.]

Virt-v2v can import physical machines, but you have to boot the
separate virt-p2v ISO on the physical machine, and generally speaking
the process is directed from the physical machine.

This makes it quite unlike the other input sources.  (Users will still
be able to use the self-boot + '-o rhev' method, ie. import into the
Export Storage Domain, as with RHEL 6).

If oVirt can (or does?) run a PXE server, then it's possible to serve a
virt-p2v bootable image to physical machines, and it's also possible
to automate it from the PXE server side.
[http://libguestfs.org/virt-p2v.1.html#kernel-command-line-configuration]

Environment variables
-

You may also want to set the following environment variables before
running virt-v2v:

LIBGUESTFS_BACKEND=direct

This is currently required for VMware and RHEL 5 Xen conversions owing
to a bug in libvirt (1134592) that was today pushed to RHEL 7.2.  You
should be able to drop this once libvirt is fixed.

TMPDIR=...

You might want to set this to control where large temporary files are
created during the v2v process.

Other environment variables that may affect virt-v2v are discussed on
both of these pages:
http://libguestfs.org/guestfs.3.html#environment-variables
http://libguestfs.org/virt-v2v.1.html#environment-variables

A note on the output mode
-

You're going to be using '-o vdsm -os /storage_domain' and the other
options: '--vdsm-image-uuid UUID', '--vdsm-vol-uuid UUID',
'--vdsm-vm-uuid UUID' [http://libguestfs.org/virt-v2v.1.html#options]

Implicit in the contract with '-o vdsm' is that VDSM is responsible
for creating the output directories, running virt-v2v as user 36:36,
and deleting the output directories if virt-v2v fails.

In this output mode virt-v2v doesn't create or clean up after itself
or try to chown files (unlike '-o rhev').

HTH,

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] virt-v2v intergration with vdsm

[Richard W.M. Jones]

On Tue, Sep 30, 2014 at 10:37:44AM +0200, Michal Skrivanek wrote:
> On Sep 16, 2014, at 14:17 , Shahar Havivi  wrote:
> > More details on virt-v2v integration can be found in its feature page:
> > http://www.ovirt.org/Features/virt-v2v_Integration
> 
> sounds good
> anyone else?:)

Is this directed at me?  Anyway I read it the first time, and I've
just scanned through it again.

I think it would be nice if the page talked about exactly what
virt-v2v command line(s) you are going to run.

In particular we support several sources.  See the '-i' options here:

  http://libguestfs.org/virt-v2v.1.html

Each source has slight quirks, so you'll need to take that into
account in the UI.  To give you just one example: RHEL 5 Xen import
needs an ssh-agent, username, server name, optional port, and guest
name -- I don't think you've taken in to account any of those.

The second thing is P2V, which is (or can be) directed from the
physical machine.  It needs a virt-v2v instance to talk to, and I
guess that rules out vdsm.  Lots more on this topic in the manual page
(especially the 1.27.56 man page which is *not* online yet, so please
consult the man page in the brew package for now).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] virt-v2v integration feature

[Richard W.M. Jones]

On Wed, Sep 10, 2014 at 03:11:37PM +0300, Itamar Heim wrote:
> On 09/10/2014 02:55 PM, Richard W.M. Jones wrote:
> >On Wed, Sep 10, 2014 at 01:36:59PM +0300, Itamar Heim wrote:
> >>- Windows VMs - v2v would need to make sure relevant iso is in the iso
> >>   domain probably with the drivers expected by v2v?
> >
> >I'm not 100% sure how this works, but I'll tell you what virt-v2v does
> >(which is the same as old virt-v2v).  It installs RHEV-APT (.exe) in
> >the guest and ensures it runs at first boot.  Does RHEV-APT need an
> >ISO to work?  I was under the impression that the executable contained
> >the drivers needed in itself.
> >
> >Rich.
> >
> 
> rhev-apt (can't remember the upstream name of the top of my head) is
> just a utility that automatically installs/updates the tools based
> on:
> - detecting there is an ISO
> - detecting the ISO is a rhev-tools iso
> - detecting the ISO is properly signed
> - run the various installers
> 
> so yes, we need to make sure if the converted VM is a windows VM,
> there is an ISO (and attach it to the VM).
> 
> how/where does v2v get rhev-apt from btw?

Very good question!  Both old and current virt-v2v use a binary (added
to the RHEL build) which Matt extracted from rhev-apt, probably an old
version.  I have no idea where it comes from, and I doubt it is up to
date.

  $ grep RHEV-App libguestfs.spec 
  Source7:   RHEV-Application_Provisioning_Tool_46267.exe

> (not sure the newly introduced ovirt windows guest driver iso has
> installers yet).

Something we can build from source using the mingw-* chain would be
useful, otherwise we can never add this to Fedora.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] virt-v2v integration feature

[Richard W.M. Jones]

On Wed, Sep 10, 2014 at 01:36:59PM +0300, Itamar Heim wrote:
> - Windows VMs - v2v would need to make sure relevant iso is in the iso
>   domain probably with the drivers expected by v2v?

I'm not 100% sure how this works, but I'll tell you what virt-v2v does
(which is the same as old virt-v2v).  It installs RHEV-APT (.exe) in
the guest and ensures it runs at first boot.  Does RHEV-APT need an
ISO to work?  I was under the impression that the executable contained
the drivers needed in itself.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] virt-v2v integration feature

[Richard W.M. Jones]

On Wed, Sep 10, 2014 at 10:41:27AM +0300, Shahar Havivi wrote:
> Rich: Can we get the estimated size when we run with --no-copy?

BTW you need to file bugs for all feature changes needed in virt-v2v.
If there's no bug it will get forgotten about.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] virt-v2v integration feature

[Richard W.M. Jones]

On Wed, Sep 10, 2014 at 10:41:27AM +0300, Shahar Havivi wrote:
> On 02.09.14 09:26, Arik Hadas wrote:
> > Hi All,
> > 
> > Better late than never..
> > 
> > Thanks for all the feedback, it was really constructive.
> > I made major changes in the wiki page to address the comments,
> > Please take another look:
> > http://www.ovirt.org/Features/virt-v2v_Integration
> > 
> > Thanks,
> > Arik
> Few issues came up after talking with storage team:
> 
> The actual import VM will take place in two steps:
> 1. SPM call to create the volume.
> 2. HSM call virt-v2v with the created UUIDs by the first SPM call.
>The virt-v2v don't need to create any directory or volume since
>the SPM create volume will create the structure for us.
>Rich: is that ok with the current virt-v2v?
> 
> Before creating the volume we are running a virt-v2v convert with --no-copy,
> which mean we don't copy the disk only get the ovf, by doing that we don't
> have the actual size of the disk - which cause a problem in block device where
> we need to preallocate the volume and to make sure that the destination have
> room for the new disk.
> Rich: Can we get the estimated size when we run with --no-copy?

Currently you get the virtual size accurately.  We could add an
estimate of the used size, but I doubt it will be very accurate.  It
would probably be an upper limit, which I guess is useful.

Rich.

> Some notes:
> Since the second call is an HSM call we need to make sure that we take the
> virt-v2v "tasks" into consideration when it comes to host life cycle.
> 
> Thanks,
>   Shahar.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [ovirt-users] virt-v2v integration feature

[Richard W.M. Jones]

On Thu, Jul 10, 2014 at 10:41:16AM +0300, Itamar Heim wrote:
> On 07/10/2014 10:29 AM, Sven Kieske wrote:
> >
> >
> >Am 09.07.2014 20:30, schrieb Arik Hadas:
> >>Hi All,
> >>
> >>The proposed feature will introduce a new process of import virtual 
> >>machines from external systems using virt-v2v in oVirt.
> >>I've created a wiki page that contains initial thoughts and design for it:
> >>http://www.ovirt.org/Features/virt-v2v_Integration
> >>
> >>You are more than welcome to share your thoughts and insights.
> >>
> >>Thanks,
> >>Arik
> >
> >Am I right that this still involves a full operational e.g. esxi host
> >to import vmware vms?
> >
> >There is a huge user demand on a simpler process for just converting and
> >importing an vmware disk image. This feature will not solve this
> >use case, will it?
> 
> I agree it should. need to check if virt-v2v can cover this. if not,
> need to fix it so it will...

>From live ESX, yes.

Offline VMware images (OVAs), not yet.  However patches are welcome.
We quite literally have no one available to implement this.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] virt-v2v integration feature

[Richard W.M. Jones]

On Thu, Jul 10, 2014 at 01:04:48AM -0400, Oved Ourfali wrote:
> As for installing virt-v2v, I don't know if that's an issue to add a
> dependency on it, and to include it in all hosts supporting cluster
> level >= 3.6.  However, if it is an issue, then it can be selected
> at the host level when adding it, and the bootstrapping will install
> it. Then it will be used only on hosts that have this checked.

The new virt-v2v has considerably fewer dependencies than the previous
version.  In particular it no longer requires Perl, and libguestfs has
split dependencies too so you only "pay for" features required by your
guests (eg. if you don't need to support XFS guests, then you don't
need to pull in the XFS tools).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel