[pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support

2024-09-30 Thread Alexandre Derumier via pve-devel
--- Begin Message --- This patch series implement qcow2 external snapshot support for files && lvm volumes The current internal qcow2 snapshots have a lot of performance problems. I have tested through nfs and also local filesystem https://lore.proxmox.com/pve-devel/6a44716a-88bc-4523-b210-d6703

[pve-devel] [PATCH v2 qemu-server 1/1] implement external snapshot

2024-09-30 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Signed-off-by: Alexandre Derumier --- PVE/QemuServer.pm | 108 -- 1 file changed, 95 insertions(+), 13 deletions(-) diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm index b26da505..1523df15 100644 --- a/PVE/QemuServer.pm +++ b/P

[pve-devel] [PATCH v2 pve-storage 2/2] add lvmqcow2 plugin: (lvm with external qcow2 snapshot)

2024-09-30 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Signed-off-by: Alexandre Derumier --- src/PVE/Storage.pm| 2 + src/PVE/Storage/LvmQcow2Plugin.pm | 460 ++ src/PVE/Storage/Makefile | 3 +- 3 files changed, 464 insertions(+), 1 deletion(-) create mode 100644 src/PVE/

[pve-devel] [PATCH v2 pve-storage 1/2] add external snasphot support

2024-09-30 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Signed-off-by: Alexandre Derumier --- src/PVE/Storage/DirPlugin.pm | 1 + src/PVE/Storage/Plugin.pm| 225 +++ 2 files changed, 201 insertions(+), 25 deletions(-) diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm i

[pve-devel] [PATCH pve-storage] lvmplugin: fix: activate|deactivate volume, add missing storeid param in path sub

2024-09-25 Thread Alexandre Derumier via pve-devel
--- Begin Message --- $storeid param is missing and $snapname is used as third param. This seem to works actually because $snapname is always empty in lvm Signed-off-by: Alexandre Derumier --- src/PVE/Storage/LVMPlugin.pm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/s

[pve-devel] [PATCH SERIES pve-storage/qemu-server] add external qcow2 snapshot support

2024-09-19 Thread Alexandre Derumier via pve-devel
--- Begin Message --- This patch series implement qcow2 external snapshot support. (for directory plugin currently,but it can be extended to nfs, gluster,...) The current internal qcow2 snapshots have a lot of performance problems. I have tested through nfs and also local filesystem https://lore

[pve-devel] [PATCH qemu-server 1/1] implement external snapshot

2024-09-19 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Signed-off-by: Alexandre Derumier --- PVE/QemuServer.pm | 74 +++ 1 file changed, 62 insertions(+), 12 deletions(-) diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm index b26da505..646e3476 100644 --- a/PVE/QemuServer.pm +++ b/P

[pve-devel] [PATCH pve-storage 1/1] add external snasphot support

2024-09-19 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Signed-off-by: Alexandre Derumier --- src/PVE/Storage/DirPlugin.pm | 1 + src/PVE/Storage/Plugin.pm| 220 +++ 2 files changed, 196 insertions(+), 25 deletions(-) diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm i

[pve-devel] [PATCH pve-storage 5/5] add volume_underlay_shrink

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- reduce underlay volume size to minimum used Signed-off-by: Alexandre Derumier --- src/PVE/Storage.pm| 15 +++ src/PVE/Storage/LVMQcow2Plugin.pm | 31 +++ src/PVE/Storage/Plugin.pm | 7 +++ 3 files changed,

[pve-devel] [PATCH SERIES storage/qemu-server/-manager] RFC : add lvmqcow2 storage support

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- This patch series add support for a new lvmqcow2 storage format. Currently, we can't do snasphot && thin provisionning on shared block devices because lvm thin can't share his metavolume. I have a lot of onprem vmware customers where it's really blocking the proxmox migratio

[pve-devel] [PATCH pve-storage 4/5] add refresh volume

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Signed-off-by: Alexandre Derumier --- src/PVE/Storage.pm| 13 + src/PVE/Storage/LVMQcow2Plugin.pm | 8 src/PVE/Storage/Plugin.pm | 6 ++ 3 files changed, 27 insertions(+) diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm

[pve-devel] [PATCH pve-manager 1/1] pvestatd: lvmqcow2 : extend disk on io-error

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- if the write are really too fast, and the auto extend from qmp event is too slow, the vm could try to write to an qcow2 offset highter than the lvm underlay. In this case, the vm will be paused in "io-error" mode. To fix it, try to extend drive and resume the vm. Signed-off

[pve-devel] [PATCH qemu-server 4/6] migration: refresh remote disk size before resume

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- A blockextend occur on source just before the switch. we want to be sure that lvm size on target is correctly refreshed, or we could have io-error is the vm is writing to non available sectors Signed-off-by: Alexandre Derumier --- PVE/CLI/qm.pm | 2

[pve-devel] [PATCH pve-storage 3/5] add volume_underlay_resize

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Allow to resize the underlay vm volume Signed-off-by: Alexandre Derumier --- src/PVE/Storage.pm| 18 ++ src/PVE/Storage/LVMQcow2Plugin.pm | 12 src/PVE/Storage/Plugin.pm | 7 +++ 3 files changed, 37 insertions(+) di

[pve-devel] [PATCH qemu-server 6/6] clone: allocate && shrink lvmcow2 underlay

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- We allocated the full size for underlay, as we can't easily known what is the used size of the source. (we could use qemu-img measure, but for some storage like nfs, it need full read to detected sparsiness for example). Then, after the copy, we use qemu-img measure to detec

[pve-devel] [PATCH pve-storage 2/5] vdisk_alloc: add underlay_size option

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Signed-off-by: Alexandre Derumier --- src/PVE/Storage.pm| 4 ++-- src/PVE/Storage/LVMQcow2Plugin.pm | 5 - 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm index 97e77c8..385150e 100755 --- a/src/PVE/S

[pve-devel] [PATCH qemu-server 3/6] qmevent: call qm disk blockextend when write_threshold event is received

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Signed-off-by: Alexandre Derumier --- qmeventd/qmeventd.c | 27 +++ 1 file changed, 27 insertions(+) diff --git a/qmeventd/qmeventd.c b/qmeventd/qmeventd.c index d8f3ee7..4f45b6f 100644 --- a/qmeventd/qmeventd.c +++ b/qmeventd/qmeventd.c @@ -229,6 +2

[pve-devel] [PATCH qemu-server 5/6] qemu_img_format: lvmqcow2 is a path_storage

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- needed for snapshot Signed-off-by: Alexandre Derumier --- PVE/QemuServer.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm index fb67ac9..a6a40cd 100644 --- a/PVE/QemuServer.pm +++ b/PVE/QemuServer.pm @@ -7922,7 +79

[pve-devel] [PATCH pve-storage 1/5] add lvmqcow2 plugin

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Signed-off-by: Alexandre Derumier --- src/PVE/Storage.pm| 2 + src/PVE/Storage/LVMQcow2Plugin.pm | 218 ++ src/PVE/Storage/Makefile | 3 +- 3 files changed, 222 insertions(+), 1 deletion(-) create mode 100644 src/PVE/

[pve-devel] [PATCH qemu-server 1/6] lvmqcow2: set disk write threshold

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- on vm start or when we hotplug a disk, we add a write threshold. The threshold is: size of the lvm - (chunk_usage_percent * chunksize) qemu will emit an event when the vm write on an offset higher than the threshold, and the counter is reset to 0. (So, we'll need to set thr

[pve-devel] [PATCH qemu-server 2/6] qm cli: add blockextend

2024-08-26 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Increase the underlay storage size to an higher value than current max offset or 1 chunk bigger we max write offset is 0 Signed-off-by: Alexandre Derumier --- PVE/CLI/qm.pm | 31 PVE/QemuServer.pm | 52 +++

[pve-devel] [PATCH pve-storage] qcow2 format: enable subcluster allocation by default

2024-07-03 Thread Alexandre Derumier via pve-devel
--- Begin Message --- extended_l2 is an optimisation to reduce write amplification. Currently,without it, when a vm write 4k, a full 64k cluster need to be writen. When enabled, the cluster is splitted in 32 subclusters. We use a 128k cluster by default, to have 32 * 4k subclusters https://blogs

[pve-devel] [PATCH pve-network 1/1] vnets : add ports isolation

2024-04-25 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Add support for bridge ports isolation https://github.com/torvalds/linux/commit/7d850abd5f4edb1b1ca4b4141a4453305736f564 This allow to drop traffic between all ports having isolation enabled on the local bridge, but allow traffic with non isolated ports. Here,we isolate traf

[pve-devel] [PATCH pve-manager 1/1] sdn: vnet: add ports-isolation option.

2024-04-25 Thread Alexandre Derumier via pve-devel
--- Begin Message --- also move vlan-aware in advanced section Signed-off-by: Alexandre Derumier --- www/manager6/sdn/VnetEdit.js | 12 1 file changed, 12 insertions(+) diff --git a/www/manager6/sdn/VnetEdit.js b/www/manager6/sdn/VnetEdit.js index cdd83ed4..a00f83e6 100644 --- a/ww

[pve-devel] [PATCH pve-network/pve-common/pve-manager] fix #4300 : sdn: add bridge ports isolation

2024-04-25 Thread Alexandre Derumier via pve-devel
--- Begin Message --- This patches series add support for sdn vnet bridge ports isolation pve-network: Alexandre Derumier (1): vnets : add ports isolation src/PVE/Network/SDN/VnetPlugin.pm | 5 + src/PVE/Network/SDN/Zones/Plugin.pm | 1 + 2 files changed, 6 insertions(+) pve-common:

[pve-devel] [PATCH pve-common 1/1] tap_plug: add support for bridge port isolation

2024-04-25 Thread Alexandre Derumier via pve-devel
--- Begin Message --- This is allow to block traffic/isolation traffic between all ports on the bridge with isolation (so between the vms), ans still allow incoming traffic from uplink. Signed-off-by: Alexandre Derumier --- src/PVE/Network.pm | 14 -- 1 file changed, 12 insertions(+)

[pve-devel] [PATCH pve-network] fix #5398: vxlan: only ipv4 is currently supported

2024-04-24 Thread Alexandre Derumier via pve-devel
--- Begin Message --- ifupdown2 only support ipv4 (could be fixed) frr/evpn: don't support ipv6 at all Signed-off-by: Alexandre Derumier --- src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 2 +- src/PVE/Network/SDN/Zones/VxlanPlugin.pm | 2 +- 2 files changed, 2 insertions(+), 2 deletions(

[pve-devel] [PATCH pve-network] fix #5343 : isis: fix ipv6 && custom router config

2024-04-16 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Signed-off-by: Alexandre Derumier --- src/PVE/Network/SDN/Controllers/IsisPlugin.pm| 3 ++- src/test/zones/evpn/isis/expected_controller_config | 2 ++ src/test/zones/evpn/isis_loopback/expected_controller_config | 2 ++ 3 files changed, 6 insertions

[pve-devel] [PATCH pve-network] fix #5319: frr.local: add support for bgp-community

2024-04-16 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Need to be inserted after ip prefix-list and before route map Signed-off-by: Alexandre Derumier --- src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 15 +-- 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/src/PVE/Network/SDN/Controllers/EvpnPlugin.p

[pve-devel] [PATCH pve-network] fix #5344: isis: add isis networkid parser

2024-04-16 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Signed-off-by: Alexandre Derumier --- src/PVE/Network/SDN/Controllers/IsisPlugin.pm | 12 +++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/src/PVE/Network/SDN/Controllers/IsisPlugin.pm b/src/PVE/Network/SDN/Controllers/IsisPlugin.pm index afea82e..

[pve-devel] [PATCH v2 pve-network] fix #5364: bgp|evpn: derivated router-id from mac address for ipv6 underlay

2024-04-12 Thread Alexandre Derumier via pve-devel
--- Begin Message --- for ipv4, we use the iface ipv4 router-id as router-id need to 32bit. That's doesn't work for pure ipv6 underlay network. since https://www.rfc-editor.org/rfc/rfc6286, we can use any 32bit id, it's just need to be unique in the ASN. Simply use the last 4 bytes of iface mac

[pve-devel] [PATCH pve-network] fix #5364: bgp|evpn: derivated router-id from mac address for ipv6 underlay

2024-04-12 Thread Alexandre Derumier via pve-devel
--- Begin Message --- for ipv4, we use the iface ipv4 router-id as router-id need to 32bit. That's doesn't work for pure ipv6 underlay network. since https://www.rfc-editor.org/rfc/rfc6286, we can use any 32bit id, it's just need to be unique in the ASN. Simply use the last 4 bytes of iface mac

[pve-devel] [PATCH pve-network] fix #5361: evpn: fix ipv6 route-map

2024-04-12 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Signed-off-by: Alexandre Derumier --- src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 2 +- src/test/zones/evpn/exitnode/expected_controller_config | 2 +- .../evpn/exitnode_local_routing/expected_controller_config| 2 +- src/test/zones/evpn/exitnod

[pve-devel] [PATCH v3 pve-manager 0/2] cloudinit pending behaviour change

2022-06-22 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Implement new cloudinit api from last qemu-server patch serie. I don't have tuned it yet, the rstore is polling the api each second, so extract the config from cloudinit drive each time. I need to check how to load it once or when regenerated only. need qemu-server patch s