Hi Andrija, thank you so much for your support.
It worked perfectly. I used the oVirt 4.3 Repository: yum install https://resources.ovirt.org/pub/yum-repo/ovirt-release43.rpm And limited the repo to the libvirt and qemu packages: includepkgs=qemu-* libvirt-* I had some issues with the configuration of my 2nd Storage NFS and had to enable rpc.statd on the nfs server, otherwise I was not able to mount ISO images from 2nd storage. With the packages from CentOS-Base this was no problem. After changing to the oVirt packages, I was able to online migrate a volume between two storage repositories using the "virsh --copy- storage-all --xml" mechanism. Afterwards I updated the CloudStack database setting pool_id in volumes to the new storage for the migrated volume and everyting looked perfectly. I am still unsure how far to use this "hack" in production, but I feel reassured, that I do now have the opportunity to use this feature for urgend cases where no VM downtime is possible for a storage migration. If there is interest I can offer to translate my notes to english to provide them to others with the same need. And @Andrija, you mentioned a "detailed step-by-step .docx guide".. I would really be interested, maybe theres further information I missed. I would really like you to forward this to me. Greetings, Melanie Am Mittwoch, den 25.09.2019, 13:51 +0200 schrieb Andrija Panic: > Hi Mellanie, > > so Ubuntu 14.04+ - i.e. 16.04 working fine, 18.04 also being > supported in > later releases... > CentOS7 is THE recommended OS (or more recent Ubuntu) - but yes, RHEL > makes > small surprises sometimes (until CentOS 7.1, if not mistaken, they > also > didn't provide RBD/Ceph support, only in paid RHEV - won't comment on > this > lousy behaviour...) > > Afaik, for KVM specifically, no pooling of volumes' location, and you > would > need to update DB (pay attention also to usage records if that is of > your > interest) > You'll need to test this kind of migration and DB schema thoroughly > (including changing disk offerings and such in DB, in case your > source/destination storage solution have different Storage TAGs in > ACS) > > I'm trying to stay away from any clustered file systems, 'cause when > they > break, they break bad...so can't comment there. > You are using those as preSetup in KVM/CloudStack I guess - if it > works, > then all good. > But...move on I suggest, if possible :) > > Best > Andrija > > On Wed, 25 Sep 2019 at 13:16, Melanie Desaive < > m.desa...@heinlein-support.de> > wrote: > > > Hi Andrija, > > > > thank you so much for your detailled explanation! Looks like the my > > problem can be solved. :) > > > > To summarize the information you provided: > > > > As long as CloudStack does not support volume live migration I > > could be > > using > > > > virsh with --copy-storage --xml. > > > > BUT: CentOS7 is lacking necessary features! Bad luck. I started out > > with CentOS7 as Disto. > > > > You suggest, that it could be worth trying the qemu/libvirt > > packages > > from the oVirt repository. I will look into this now. > > > > But if that gets complicated: Cloudstack documentation lists > > CentOS7 > > and Ubuntu 14.04 as supported Distros. Are there other not > > officially > > supported Distros/Version I could be using? I wanted to avoid the > > quite > > outdated Ubuntu 14.04 and did for that reason decide towards > > CentOS7. > > > > And another general question: How is CloudStack getting along with > > the > > Volumes of its VMs changing the storage repository without beeing > > informed about it. Does it get this information through polling, or > > do > > I have to manipulate the database? > > > > And to make things clearer: At the moment I am using storage > > attached > > through Gibrechannel using clustered LVM logic. Could also be > > changing > > to GFS2 on cLVM. Never heard anyone mentioning such a setup by now. > > Am > > I the only one running KVM on a proprietary storage system over > > Fibrechannel, are there limitation/problems to be expected from > > such a > > setup? > > > > Greetings, > > > > Melanie > > > > > > Am Mittwoch, den 25.09.2019, 11:46 +0200 schrieb Andrija Panic: > > > So, let me explain. > > > > > > Doing "online storage migration" aka live storage migration is > > > working for > > > CEPH/NFS --> SolidFire, starting from 4.11+ > > > Internally it is done in the same way as "virsh with --copy- > > > storage- > > > all > > > --xml" in short > > > > > > Longer explanation: > > > Steps: > > > You create new volumes on the destination storage (SolidFire in > > > this > > > case), > > > set QoS etc - simply prepare the destination volumes (empty > > > volumes > > > atm). > > > On source host/VM, dump VM XML, edit XML, change disk section to > > > point to > > > new volume path, protocol, etc - and also the IP address for the > > > VNC > > > (cloudstack requirement), save XMLT > > > Then you do "virsh with --copy-storage-all --xml. myEditedVM.xml > > > ..." > > > stuff > > > that does the job. > > > Then NBD driver will be used to copy blocks from the source > > > volumes > > > to the > > > destination volumes while that virsh command is working... > > > (here's my > > > demo, > > > in details.. > > > > > https://www.youtube.com/watch?v=Eo8BuHBnVgg&list=PLEr0fbgkyLKyiPnNzPz7XDjxnmQNxjJWT&index=5&t=2s > > > ) > > > > > > This is yet to be extended/coded to support NFS-->NFS or CEPH > > > -->CEPH > > > or > > > CEPH/NFS-->CEPH/NFS... should not be that much work, the logic is > > > there > > > (bit part of the code) > > > Also, starting from 4.12, you can actually (I believe using > > > identical > > > logic) migrate only ROOT volume that are on the LOCAL storage > > > (local > > > disks) > > > to another host/local storage - but DATA disks are not supported. > > > > > > Now...imagine the feature is there - if using CentOS7, our > > > friends at > > > RedHat have removed support for actually using live storage > > > migration > > > (unless you are paying for RHEV - but it does work fine on > > > CentOS6, > > > and > > > Ubuntu 14.04+ > > > > > > I recall "we" had to use qemu/libvirt from the "oVirt" repo which > > > DOES > > > (DID) support storage live migration (normal EV packages from the > > > Special > > > Interest Group (2.12 tested) - did NOT include this...) > > > > > > I can send you step-by-step .docx guide for manually mimicking > > > what > > > is done > > > (in SolidFire, but identical logic for other storages) - but not > > > sure > > > if > > > that still helps you... > > > > > > > > > Andrija > > > > > > On Wed, 25 Sep 2019 at 10:51, Melanie Desaive < > > > m.desa...@heinlein-support.de> > > > wrote: > > > > > > > Hi all, > > > > > > > > I am currently doing my first steps with KVM as hypervisor for > > > > CloudStack. I was shocked to realize that currently live volume > > > > migration between different shared storages is not supported > > > > with > > > > KVM. > > > > This is a feature I use intensively with XenServer. > > > > > > > > How do you get along with this limitation? I do really expect > > > > you > > > > to > > > > use some workarounds, or do you all only accept vm downtimes > > > > for a > > > > storage migration? > > > > > > > > With my first investigation I found three techniques mentioned > > > > and > > > > would like to ask for suggestions which to investigate deeper: > > > > > > > > x Eric describes a technique using snapshosts and pauses to do > > > > a > > > > live > > > > storage migration in this mailing list tread. > > > > x Dag suggests using virsh with --copy-storage-all --xml. > > > > x I found articles about using virsh blockcopy for storage > > > > live > > > > migration. > > > > > > > > Greetings, > > > > > > > > Melanie > > > > > > > > Am Freitag, den 02.02.2018, 15:55 +0100 schrieb Andrija Panic: > > > > > @Dag, you might want to check with Mike Tutkowski, how he > > > > > implemented > > > > > this > > > > > for the "online storage migration" from other storages (CEPH > > > > > and > > > > > NFS > > > > > implemented so far as sources) to SolidFire. > > > > > > > > > > We are doing exactly the same demo/manual way (this is what > > > > > Mike > > > > > has > > > > > sent > > > > > me back in the days), so perhaps you want to see how to > > > > > translate > > > > > this into > > > > > general things (so ANY to ANY storage migration) inside > > > > > CloudStack. > > > > > > > > > > Cheers > > > > > > > > > > On 2 February 2018 at 10:28, Dag Sonstebo < > > > > > dag.sonst...@shapeblue.com > > > > > wrote: > > > > > > > > > > > All > > > > > > > > > > > > I am doing a bit of R&D around this for a client at the > > > > > > moment. > > > > > > I > > > > > > am > > > > > > semi-successful in getting live migrations to different > > > > > > storage > > > > > > pools to > > > > > > work. The method I’m using is as follows – this does not > > > > > > take > > > > > > into > > > > > > account > > > > > > any efficiency optimisation around the disk transfer (which > > > > > > is > > > > > > next > > > > > > on my > > > > > > list). The below should answer your question Eric about > > > > > > moving > > > > > > to a > > > > > > different location – and I am also working with your steps > > > > > > to > > > > > > see > > > > > > where I > > > > > > can improve the following. Keep in mind all of this is > > > > > > external > > > > > > to > > > > > > CloudStack – although CloudStack picks up the destination > > > > > > KVM > > > > > > host > > > > > > automatically it does not update the volume tables etc., > > > > > > neither > > > > > > does it do > > > > > > any housekeeping. > > > > > > > > > > > > 1) Ensure the same network bridges are up on source and > > > > > > destination > > > > > > – > > > > > > these are found with: > > > > > > > > > > > > [root@kvm1 ~]# virsh dumpxml 9 | grep source > > > > > > <source file='/mnt/00e88a7b-985f-3be8-b717- > > > > > > 0a59d8197640/d0ab5dd5- > > > > > > e3dd-47ac-a326-5ce3d47d194d'/> > > > > > > <source bridge='breth1-725'/> > > > > > > <source path='/dev/pts/3'/> > > > > > > <source path='/dev/pts/3'/> > > > > > > > > > > > > So from this make sure breth1-725 is up on the destionation > > > > > > host > > > > > > (do it > > > > > > the hard way or cheat and spin up a VM from same account > > > > > > and > > > > > > network on > > > > > > that host) > > > > > > > > > > > > 2) Find size of source disk and create stub disk in > > > > > > destination > > > > > > (this part > > > > > > can be made more efficient to speed up disk transfer – by > > > > > > doing > > > > > > similar > > > > > > things to what Eric is doing): > > > > > > > > > > > > [root@kvm1 ~]# qemu-img info /mnt/00e88a7b-985f-3be8-b717- > > > > > > 0a59d8197640/d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d > > > > > > image: /mnt/00e88a7b-985f-3be8-b717-0a59d8197640/d0ab5dd5- > > > > > > e3dd- > > > > > > 47ac-a326-5ce3d47d194d > > > > > > file format: qcow2 > > > > > > virtual size: 8.0G (8589934592 bytes) > > > > > > disk size: 32M > > > > > > cluster_size: 65536 > > > > > > backing file: /mnt/00e88a7b-985f-3be8-b717- > > > > > > 0a59d8197640/3caaf4c9- > > > > > > eaec- > > > > > > 11e7-800b-06b4a401075c > > > > > > > > > > > > ###################### > > > > > > > > > > > > [root@kvm3 50848ff7-c6aa-3fdd-b487-27899bf2129c]# qemu-img > > > > > > create > > > > > > -f > > > > > > qcow2 d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d 8G > > > > > > Formatting 'd0ab5dd5-e3dd-47ac-a326-5ce3d47d194d', > > > > > > fmt=qcow2 > > > > > > size=8589934592 encryption=off cluster_size=65536 > > > > > > [root@kvm3 50848ff7-c6aa-3fdd-b487-27899bf2129c]# qemu-img > > > > > > info > > > > > > d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d > > > > > > image: d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d > > > > > > file format: qcow2 > > > > > > virtual size: 8.0G (8589934592 bytes) > > > > > > disk size: 448K > > > > > > cluster_size: 65536 > > > > > > > > > > > > 3) Rewrite the new VM XML file for the destination with: > > > > > > a) New disk location, in this case this is just a new path > > > > > > (Eric – > > > > > > this > > > > > > answers your question) > > > > > > b) Different IP addresses for VNC – in this case 10.0.0.1 > > > > > > to > > > > > > 10.0.0.2 > > > > > > and carry out migration. > > > > > > > > > > > > [root@kvm1 ~]# virsh dumpxml 9 | sed -e 's/00e88a7b-985f- > > > > > > 3be8- > > > > > > b717- > > > > > > 0a59d8197640/50848ff7-c6aa-3fdd-b487-27899bf2129c/g' | sed > > > > > > -e > > > > > > 's/ > > > > > > 10.0.0.1/10.0.0.2/g' > /root/i-2-14-VM.xml > > > > > > > > > > > > [root@kvm1 ~]# virsh migrate --live --persistent --copy- > > > > > > storage-all > > > > > > --xml > > > > > > /root/i-2-14-VM.xml i-2-14-VM qemu+tcp://10.0.0.2/system -- > > > > > > verbose > > > > > > --abort-on-error > > > > > > Migration: [ 25 %] > > > > > > > > > > > > 4) Once complete delete the source file. This can be done > > > > > > with > > > > > > extra > > > > > > switches on the virsh migrate command if need be. > > > > > > = = = > > > > > > > > > > > > In the simplest tests this works – destination VM remains > > > > > > online > > > > > > and has > > > > > > storage in new location – but it’s not persistent – > > > > > > sometimes > > > > > > the > > > > > > destination VM ends up in a paused state, and I’m working > > > > > > on > > > > > > how to > > > > > > get > > > > > > around this. I also noted virsh migrate has a migrate- > > > > > > setmaxdowntime which > > > > > > I think can be useful here. > > > > > > > > > > > > Regards, > > > > > > Dag Sonstebo > > > > > > Cloud Architect > > > > > > ShapeBlue > > > > > > > > > > > > On 01/02/2018, 20:30, "Andrija Panic" < > > > > > > andrija.pa...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > Actually, we have this feature (we call this > > > > > > internally > > > > > > online-storage-migration) to migrate volume from > > > > > > CEPH/NFS > > > > > > to > > > > > > SolidFire > > > > > > (thanks to Mike Tutkowski) > > > > > > > > > > > > There is libvirt mechanism, where basically you start > > > > > > another > > > > > > PAUSED > > > > > > VM on > > > > > > another host (same name and same XML file, except the > > > > > > storage > > > > > > volumes > > > > > > are > > > > > > pointing to new storage, different paths, etc and maybe > > > > > > VNC > > > > > > listening > > > > > > address needs to be changed or so) and then you issue > > > > > > on > > > > > > original > > > > > > host/VM > > > > > > the live migrate command with few parameters... the > > > > > > libvirt > > > > > > will > > > > > > transaprently handle the copy data process from Soruce > > > > > > to > > > > > > New > > > > > > volumes, > > > > > > and > > > > > > after migration the VM will be alive (with new XML > > > > > > since > > > > > > have > > > > > > new > > > > > > volumes) > > > > > > on new host, while the original VM on original host is > > > > > > destroyed.... > > > > > > > > > > > > (I can send you manual for this, that is realted to SF, > > > > > > but > > > > > > idea is the > > > > > > same and you can exercies this on i.e. 2 NFS volumes on > > > > > > 2 > > > > > > different > > > > > > storages) > > > > > > > > > > > > This mechanism doesn't exist in ACS in general (AFAIK), > > > > > > except > > > > > > for when > > > > > > migrating to SolidFire. > > > > > > > > > > > > Perhaps community/DEV can help extend Mike's code to do > > > > > > same > > > > > > work on > > > > > > different storage types... > > > > > > > > > > > > Cheers > > > > > > > > > > > > > > > > > > dag.sonst...@shapeblue.com > > > > > > www.shapeblue.com > > > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > > > > @shapeblue > > > > > > > > > > > > > > > > > > > > > > > > On 19 January 2018 at 18:45, Eric Green < > > > > > > eric.lee.gr...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > KVM is able to live migrate entire virtual machines > > > > > > complete > > > > > > with > > > > > > local > > > > > > > volumes (see 'man virsh') but does require nbd > > > > > > (Network > > > > > > Block > > > > > > Device) to be > > > > > > > installed on the destination host to do so. It may > > > > > > need > > > > > > installation > > > > > > of > > > > > > > later libvirt / qemu packages from OpenStack > > > > > > repositories > > > > > > on > > > > > > Centos > > > > > > 6, I'm > > > > > > > not sure, but just works on Centos 7. In any event, I > > > > > > have > > > > > > used this > > > > > > > functionality to move virtual machines between > > > > > > virtualization > > > > > > hosts > > > > > > on my > > > > > > > home network. It works. > > > > > > > > > > > > > > What is missing is the ability to live-migrate a disk > > > > > > from > > > > > > one shared > > > > > > > storage to another. The functionality built into > > > > > > virsh > > > > > > live- > > > > > > migrates > > > > > > the > > > > > > > volume ***to the exact same location on the new > > > > > > host***, > > > > > > so > > > > > > obviously is > > > > > > > useless for migrating the disk to a new location on > > > > > > shared > > > > > > storage. I > > > > > > > looked everywhere for the ability of KVM to live > > > > > > migrate > > > > > > a > > > > > > disk from > > > > > > point > > > > > > > A to point B all by itself, and found no such thing. > > > > > > libvirt/qemu > > > > > > has the > > > > > > > raw capabilities needed to do this, but it is not > > > > > > currently > > > > > > exposed > > > > > > as a > > > > > > > single API via the qemu console or virsh. It can be > > > > > > emulated > > > > > > via > > > > > > scripting > > > > > > > however: > > > > > > > > > > > > > > 1. Pause virtual machine > > > > > > > 2. Do qcow2 snapshot. > > > > > > > 3. Detach base disk, attach qcow2 snapshot > > > > > > > 4. unpause virtual machine > > > > > > > 5. copy qcow2 base file to new location > > > > > > > 6. pause virtual machine > > > > > > > 7. detach snapshot > > > > > > > 8. unsnapshot qcow2 snapshot at its new location. > > > > > > > 9. attach new base at new location. > > > > > > > 10. unpause virtual machine. > > > > > > > > > > > > > > Thing is, if that entire process is not built into > > > > > > the > > > > > > underlying > > > > > > > kvm/qemu/libvirt infrastructure as tested > > > > > > functionality > > > > > > with > > > > > > a > > > > > > defined API, > > > > > > > there's no guarantee that it will work seamlessly and > > > > > > will > > > > > > continue > > > > > > working > > > > > > > with the next release of the underlying > > > > > > infrastructure. > > > > > > This > > > > > > is using > > > > > > > multiple different tools to manipulate the qcow2 file > > > > > > and > > > > > > attach/detach > > > > > > > base disks to the running (but paused) kvm domain, > > > > > > and > > > > > > would > > > > > > have to > > > > > > be > > > > > > > tested against all variations of those tools on all > > > > > > supported > > > > > > Cloudstack > > > > > > > KVM host platforms. The test matrix looks pretty > > > > > > grim. > > > > > > > > > > > > > > By contrast, the migrate-with-local-storage process > > > > > > is > > > > > > built > > > > > > into > > > > > > virsh > > > > > > > and is tested by the distribution vendor and the set > > > > > > of > > > > > > tools > > > > > > provided with > > > > > > > the distribution is guaranteed to work with the virsh > > > > > > / > > > > > > libvirt/ qemu > > > > > > > distributed by the distribution vendor. That makes > > > > > > the > > > > > > test > > > > > > matrix > > > > > > for > > > > > > > move-with-local-storage look a lot simpler -- "is > > > > > > this > > > > > > functionality > > > > > > > supported by that version of virsh on that > > > > > > distribution? > > > > > > Yes? > > > > > > Enable > > > > > > it. > > > > > > > No? Don't enable it." > > > > > > > > > > > > > > I'd love to have live migration of disks on shared > > > > > > storage > > > > > > with > > > > > > Cloudstack > > > > > > > KVM, but not at the expense of reliability. Shutting > > > > > > down > > > > > > a > > > > > > virtual > > > > > > machine > > > > > > > in order to migrate one of its disks from one shared > > > > > > datastore to > > > > > > another > > > > > > > is not ideal, but at least it's guaranteed reliable. > > > > > > > > > > > > > > > > > > > > > > On Jan 19, 2018, at 04:54, Rafael Weingärtner < > > > > > > > rafaelweingart...@gmail.com> wrote: > > > > > > > > > > > > > > > > Hey Marc, > > > > > > > > It is very interesting that you are going to pick > > > > > > this > > > > > > up > > > > > > for KVM. > > > > > > I am > > > > > > > > working in a related issue for XenServer [1]. > > > > > > > > If you can confirm that KVM is able to live migrate > > > > > > local > > > > > > volumes > > > > > > to > > > > > > > other > > > > > > > > local storage or shared storage I could make the > > > > > > feature I > > > > > > am > > > > > > working on > > > > > > > > available to KVM as well. > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-10240 > > > > > > > > > > > > > > > > On Thu, Jan 18, 2018 at 11:35 AM, Marc-Aurèle > > > > > > Brothier > > > > > > < > > > > > > > ma...@exoscale.ch> > > > > > > > > wrote: > > > > > > > > > > > > > > > >> There's a PR waiting to be fixed about live > > > > > > migration > > > > > > with > > > > > > local > > > > > > volume > > > > > > > for > > > > > > > >> KVM. So it will come at some point. I'm the one > > > > > > who > > > > > > made > > > > > > this PR > > > > > > but I'm > > > > > > > >> not using the upstream release so it's hard for me > > > > > > to > > > > > > debug the > > > > > > problem. > > > > > > > >> You can add yourself to the PR to get notify when > > > > > > things > > > > > > are > > > > > > moving on > > > > > > > it. > > > > > > > >> > > > > > > > >> https://github.com/apache/cloudstack/pull/1709 > > > > > > > >> > > > > > > > >> On Wed, Jan 17, 2018 at 10:56 AM, Eric Green < > > > > > > eric.lee.gr...@gmail.com> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >>> Theoretically on Centos 7 as the host KVM OS it > > > > > > could > > > > > > be > > > > > > done > > > > > > with a > > > > > > > >>> couple of pauses and the snapshotting mechanism > > > > > > built > > > > > > into > > > > > > qcow2, but > > > > > > > >> there > > > > > > > >>> is no simple way to do it directly via virsh, the > > > > > > libvirtd/qemu > > > > > > control > > > > > > > >>> program that is used to manage virtualization. > > > > > > It's > > > > > > not > > > > > > as with > > > > > > > issuing a > > > > > > > >>> simple vmotion 'migrate volume' call in Vmware. > > > > > > > >>> > > > > > > > >>> I scripted out how it would work without that > > > > > > direct > > > > > > support in > > > > > > > >>> libvirt/virsh and after looking at all the points > > > > > > where > > > > > > things > > > > > > could go > > > > > > > >>> wrong, honestly, I think we need to wait until > > > > > > there > > > > > > is > > > > > > support > > > > > > in > > > > > > > >>> libvirt/virsh to do this. virsh clearly has the > > > > > > capability > > > > > > internally > > > > > > > to > > > > > > > >> do > > > > > > > >>> live migration of storage, since it does this for > > > > > > live > > > > > > domain > > > > > > migration > > > > > > > >> of > > > > > > > >>> local storage between machines when migrating KVM > > > > > > domains > > > > > > from > > > > > > one host > > > > > > > >> to > > > > > > > >>> another, but that capability is not currently > > > > > > exposed > > > > > > in > > > > > > a way > > > > > > > Cloudstack > > > > > > > >>> could use, at least not on Centos 7. > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> On Jan 17, 2018, at 01:05, Piotr Pisz < > > > > > > pp...@pulab.pl> > > > > > > wrote: > > > > > > > >>>> > > > > > > > >>>> Hello, > > > > > > > >>>> > > > > > > > >>>> Is there a chance that one day it will be > > > > > > possible > > > > > > to > > > > > > migrate > > > > > > volume > > > > > > > >>> (root disk) of a live VM in KVM between storage > > > > > > pools > > > > > > (in > > > > > > CloudStack)? > > > > > > > >>>> Like a storage vMotion in Vmware. > > > > > > > >>>> > > > > > > > >>>> Best regards, > > > > > > > >>>> Piotr > > > > > > > >>>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > > > > > > -- > > > > -- > > > > Heinlein Support GmbH > > > > Schwedter Str. 8/9b, 10119 Berlin > > > > > > > > https://www.heinlein-support.de > > > > > > > > Tel: 030 / 40 50 51 - 62 > > > > Fax: 030 / 40 50 51 - 19 > > > > > > > > Amtsgericht Berlin-Charlottenburg - HRB 93818 B > > > > Geschäftsführer: Peer Heinlein - Sitz: Berlin > > > > > > -- > > -- > > Heinlein Support GmbH > > Schwedter Str. 8/9b, 10119 Berlin > > > > https://www.heinlein-support.de > > > > Tel: 030 / 40 50 51 - 62 > > Fax: 030 / 40 50 51 - 19 > > > > Amtsgericht Berlin-Charlottenburg - HRB 93818 B > > Geschäftsführer: Peer Heinlein - Sitz: Berlin > > > > -- -- Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 40 50 51 - 62 Fax: 030 / 40 50 51 - 19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
signature.asc
Description: This is a digitally signed message part