--- Begin Message ---
El 5/7/21 a las 12:36, Eneko Lacunza escribió:
Hi all,
We have split the BIG 88 node cluster in 6 clusters of 15 nodes each
(there where some spare servers); now things seem much better :)
Sadly, we are seeing some issues when VDI management system (USD
Enterprise) is performing mass (in the order of 100s or even 1000s)
destruction and creation of VMs. In a fraction of the clone
operations, clone will fail with the following message:
"Error: clone failed. Failed to change directory to
'/mnt/pve/vdi-prod1/images/103': No such file or directory at
/usr/share/perl5/PVE/Storage/Plugin.pm line 708."
This happens when destroy for that VMID was some seconds before
(5s-14s for example). When another clone tries to use that VMID later
(as soon as 54s after destruction), it works ok.
PVE version is 6.4 ISO (details below), and storage is NFS 4.2 with
pNFS with two pairs of NetApp servers in HA.
Seems like a "race condition" is happening, where the node that is
cloning sees the storage directory removed by destruction late (?).
I have checked "qemu-server.git/PVE/QemuServer.pm:sub destroy_vm" and
I see first storage disk are freed and after that VM config is
removed, which seems quite correct. Could it be the NFS servers that
are a bit "late" propagating directory removal to the client nodes?
We seem to have isolated this to the default NFS mount option
"lookupcach/e"/, it defaults to "all". Changing to "none" seems to fix
the issue, so it's node's local NFS cache.
UDS Enterprise has released a fix so that VMIDs don't get reused so
early (they now secuentially use all VMID space until restarting with
low VMIDs), that fixes the issue without disabling NFS lookupcache.
This may impact other uses of NFS storage, so I suggest a bit more
checking should be done:
https://git.proxmox.com/?p=pve-storage.git;a=blob;f=PVE/Storage/Plugin.pm;h=26c74304aef46da195a5db1a1d266ccec7c3fb8c;hb=refs/heads/stable-6
676 sub clone_image {
677 my ($class, $scfg, $storeid, $volname, $vmid, $snap) = @_;
678
679 # this only works for file based storage types
680 die "storage definintion has no path\n" if !$scfg->{path};
681
682 my ($vtype, $basename, $basevmid, undef, undef, $isBase,
$format) =
683 $class->parse_volname($volname);
684
685 die "clone_image on wrong vtype '$vtype'\n" if $vtype ne 'images';
686
687 die "this storage type does not support clone_image on
snapshot\n" if $snap;
688
689 die "this storage type does not support clone_image on
subvolumes\n" if $format eq 'subvol';
690
691 die "clone_image only works on base images\n" if !$isBase;
692
693 my $imagedir = $class->get_subdir($scfg, 'images');
694 $imagedir .= "/$vmid";
695
*696 mkpath $imagedir;*
697
698 my $name = $class->find_free_diskname($imagedir, $scfg, $vmid,
"qcow2", 1);
699
700 warn "clone $volname: $vtype, $name, $vmid to $name
(base=../$basevmid/$basename)\n";
701
702 my $newvol = "$basevmid/$basename/$vmid/$name";
703
704 my $path = $class->filesystem_path($scfg, $newvol);
705
706 # Note: we use relative paths, so we need to call chdir before
qemu-img
707 eval {
*708 local $CWD = $imagedir;*
709
710 my $cmd = ['/usr/bin/qemu-img', 'create', '-b',
"../$basevmid/$basename",
711 '-f', 'qcow2', $path];
712
713 run_command($cmd);
714 };
715 my $err = $@;
716
717 die $err if $err;
718
719 return $newvol;
720 }
I'm not a PERL coder but maybe doing "local $CWD = $imagedir" before
mkpath, or some more explicit test before mkpath that will make it
create the dir, instead of believing data in lookupcache.
Cheers,
Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project
Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
--- End Message ---
_______________________________________________
pve-user mailing list
[email protected]
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
