[pve-devel] [PATCH zfsonlinux 1/2] update zfs submodule to 2.2.6
patches did not need adaptation Signed-off-by: Stoiko Ivanov --- upstream | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/upstream b/upstream index 25665920..baa50314 16 --- a/upstream +++ b/upstream @@ -1 +1 @@ -Subproject commit 2566592045780e7be7afc899c2496b1ae3af4f4d +Subproject commit baa50314567afd986a00838f0fa65fdacbd12daf -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux 2/2] debian: remove libzfsbootenv1linux.install
When initially packaging libzfsbootenv1linux a .install file was commited in addtion to the .install.in (which contains the proper path with the multiarch component). This wasn't noticed during building because the .install got clobbered while building Fixes: fd0cc4becda91c46991e723ae6f0b08dcdf661e6 Signed-off-by: Stoiko Ivanov --- debian/libzfsbootenv1linux.install | 1 - 1 file changed, 1 deletion(-) delete mode 100644 debian/libzfsbootenv1linux.install diff --git a/debian/libzfsbootenv1linux.install b/debian/libzfsbootenv1linux.install deleted file mode 100644 index 01d3f0c3.. --- a/debian/libzfsbootenv1linux.install +++ /dev/null @@ -1 +0,0 @@ -lib/libzfsbootenv.so.* -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux 0/2] update ZFS to 2.2.6
This patchset updates ZFS to 2.2.6, which was released last night. It supersedes the update to 2.2.5: https://lore.proxmox.com/pve-devel/20240820164512.1532793-1-s.iva...@proxmox.com/ The usrmerge part of the above series will be resent independently later, as we probably don't want to do that in between releases (as talked off-list with Thomas). In addition to the changes from 2.2.5, 2.2.6 contains a fixes discard on zvols, that might affect kernel 6.8: https://github.com/openzfs/zfs/pull/16462 https://github.com/openzfs/zfs/pull/16454 additionally it contains a few relevant fixes for more recent kernels (6.10+) It also pulls in a fix for bash-completion that was reported in our bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=5565 Tested minimally on a pair of VMs (migrate a container, storage replication). Relevant part of the cover-letter for the superseded series for ZFS-2.2.5: This patchset updates ZFS to 2.2.5 which was released two weeks ago [0] The changes don't look too scary, but could prevent a few issues in not so common use-cases for us e.g.: https://github.com/openzfs/zfs/pull/16359/commits/4d2f7f9839d12708417457cd57cf43d15cae5e92 https://github.com/openzfs/zfs/pull/16359/commits/6f27c4cadd29eb9b850c1c66bf71ef9ba119b955 https://github.com/openzfs/zfs/pull/16359/commits/ef08cb26dae6b3c2e930e66852f13329babb7c2e [0] https://github.com/openzfs/zfs/pull/16359 Stoiko Ivanov (2): update zfs submodule to 2.2.6 debian: remove libzfsbootenv1linux.install debian/libzfsbootenv1linux.install | 1 - upstream | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) delete mode 100644 debian/libzfsbootenv1linux.install -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH storage v2] api: disks: zfs: enable monthly scrub when creating a pool
On Tue, 27 Aug 2024 16:24:33 +0200 Maximiliano Sandoval wrote: > And disable it when destroying the pool. > can be fine in general - but why? our packages ship: /etc/cron.d/zfsutils-linux (and /usr/lib/zfs-linux/scrub) which does (afair) run a monthly scrub (and trim) under certain conditions (including by default on healthy pools) what do we gain by explicitly enabling a unit for scrubbing (which at least from experiences with the per-pool import unit, will cause odd warnings in the journal, resulting in support requests)? > Signed-off-by: Maximiliano Sandoval > --- > > Differences from v1: > - Add --now flag to commands so they are `started` on top of `enabled`. a short explanation on why this is/was necessary would help (maybe even with a link to the relevant documentation e.g.: https://documentation.suse.com/smart/systems-management/html/systemd-working-with-timers/index.html#systemd-timer-manage > > > src/PVE/API2/Disks/ZFS.pm | 12 > 1 file changed, 12 insertions(+) > > diff --git a/src/PVE/API2/Disks/ZFS.pm b/src/PVE/API2/Disks/ZFS.pm > index 6fb6bd6b..6d9ace5b 100644 > --- a/src/PVE/API2/Disks/ZFS.pm > +++ b/src/PVE/API2/Disks/ZFS.pm > @@ -490,6 +490,13 @@ __PACKAGE__->register_method ({ > run_command($cmd); > } > > + if (-e '/lib/systemd/system/zfs-scrub-monthly@.timer') { > + my $scrubtimer = 'zfs-scrub-monthly@'. > PVE::Systemd::escape_unit($name, undef) . '.timer'; > + $cmd = ['systemctl', 'enable', $scrubtimer, '--now']; > + print "# ", join(' ', @$cmd), "\n"; > + run_command($cmd); > + } > + > PVE::Diskmanage::udevadm_trigger($devs->@*); > > if ($param->{add_storage}) { > @@ -583,6 +590,11 @@ __PACKAGE__->register_method ({ > run_command(['systemctl', 'disable', $importunit]); > } > > + if (-e '/lib/systemd/system/zfs-scrub-monthly@.timer') { > + my $scrubtimer = 'zfs-scrub-monthly@' . > PVE::Systemd::escape_unit($name) . '.timer'; > + run_command(['systemctl', 'disable', $scrubtimer, '--now']); > + } > + > run_command(['zpool', 'destroy', $name]); > > my $config_err; ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux 4/5] d/zfsutils-linux.install: move most exectuables from sbin to bin
follows commit e1e64f07af5c4ca2a313625a15c24e4ad6fb42f1 in debian upstream [0]. context from upstream's commit message: - arc_summary and zilstat do not require privilege. - arcstat does not need root, but the name is taken by nordugrid-arc-client. - dbufstat needs root permission to read /proc/spl/kstat. differences: we never shipped zvol_wait and zilstat in /sbin, so no need for the compatibility symlink. [0] https://salsa.debian.org/zfsonlinux-team/zfs Signed-off-by: Stoiko Ivanov --- debian/rules | 6 -- debian/zfsutils-linux.install | 6 +++--- debian/zfsutils-linux.links | 2 ++ 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/debian/rules b/debian/rules index 9299985a..224fd9fd 100755 --- a/debian/rules +++ b/debian/rules @@ -61,12 +61,6 @@ override_dh_auto_install: @# Install the utilities. $(MAKE) install DESTDIR='$(CURDIR)/debian/tmp' - # Move from bin_dir to /usr/sbin - mkdir -p '$(CURDIR)/debian/tmp/usr/sbin/' - mv '$(CURDIR)/debian/tmp/usr/bin/arc_summary' '$(CURDIR)/debian/tmp/usr/sbin/arc_summary' - mv '$(CURDIR)/debian/tmp/usr/bin/arcstat' '$(CURDIR)/debian/tmp/usr/sbin/arcstat' - mv '$(CURDIR)/debian/tmp/usr/bin/dbufstat' '$(CURDIR)/debian/tmp/usr/sbin/dbufstat' - mkdir -p $(CURDIR)/debian/tmp/usr/lib/$(DEB_HOST_MULTIARCH) chmod a-x '$(CURDIR)/debian/tmp/etc/zfs/zfs-functions' diff --git a/debian/zfsutils-linux.install b/debian/zfsutils-linux.install index c6a97ce1..e120d6c7 100644 --- a/debian/zfsutils-linux.install +++ b/debian/zfsutils-linux.install @@ -28,9 +28,9 @@ usr/lib/udev/ usr/lib/zfs-linux/zfs_prepare_disk usr/lib/zfs-linux/zpool.d/ usr/lib/zfs-linux/zpool_influxdb -usr/sbin/arc_summary -usr/sbin/arcstat -usr/sbin/dbufstat +usr/bin/arc_summary +usr/bin/arcstat usr/sbin +usr/bin/dbufstat usr/sbin usr/sbin/fsck.zfs usr/sbin/mount.zfs usr/sbin/zdb diff --git a/debian/zfsutils-linux.links b/debian/zfsutils-linux.links index faa031bb..9a516905 100644 --- a/debian/zfsutils-linux.links +++ b/debian/zfsutils-linux.links @@ -1,3 +1,5 @@ usr/sbin/zfs usr/bin/zfs usr/sbin/zpoolusr/bin/zpool usr/lib/zfs-linux/zpool_influxdb bin/zpool_influxdb +usr/bin/arc_summary usr/sbin/arc_summary + -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux 0/5] update to ZFS 2.2.5 and opt-in to usrmerge
This patchset updates ZFS to 2.2.5 which was released two weeks ago [0] The changes don't look too scary, but could prevent a few issues in not so common use-cases for us e.g.: https://github.com/openzfs/zfs/pull/16359/commits/4d2f7f9839d12708417457cd57cf43d15cae5e92 https://github.com/openzfs/zfs/pull/16359/commits/6f27c4cadd29eb9b850c1c66bf71ef9ba119b955 https://github.com/openzfs/zfs/pull/16359/commits/ef08cb26dae6b3c2e930e66852f13329babb7c2e Additionally I checked Debian's zfs repository [1], and pulled in the changes done there. Most prominently this is opt-in to usrmerge. We could address the conflict of arcstat (a different executable is also called arcstat in `nordugrid-arc-client` (a grid middleware project, which is rather unlikely to be co-installed on a machine running either of our products)) differently (simply by conflicting with nordugrid-arc-client, but I opted to stick close to upstream's packaging. Similarly I'm not sure if we want to include the last patch (adding a NEWS file for the usrmerge, notifying users prominently about the changed paths), but included it as a RFC - feel free to drop it. minimally tested on 2 VMs having a few containers with replication configured. additionally installed a VM with ZFS on / and checked that the usrmerge did not prevent it from booting. [0] https://github.com/openzfs/zfs/pull/16359 [1] https://salsa.debian.org/zfsonlinux-team/zfs Stoiko Ivanov (5): update zfs submodule to 2.2.5 debian: remove libzfsbootenv1linux.install debian: install files to /usr d/zfsutils-linux.install: move most exectuables from sbin to bin d/NEWS: add news for usr-merge debian/NEWS | 13 debian/libnvpair3linux.install.in | 2 +- debian/libpam-zfs.install | 2 - debian/libpam-zfs.install.in | 2 + debian/libuutil3linux.install.in | 2 +- debian/libzfs4linux.install.in| 4 +- debian/libzfsbootenv1linux.install| 1 - debian/libzfsbootenv1linux.install.in | 2 +- debian/libzfslinux-dev.install.in | 6 +- debian/libzpool5linux.install.in | 2 +- debian/rules | 32 +++-- .../usr/share/initramfs-tools/hooks/zdev | 10 +-- debian/zfs-test.install | 4 +- debian/zfs-zed.install| 2 +- debian/zfsutils-linux.install | 66 +-- debian/zfsutils-linux.links | 6 +- upstream | 2 +- 17 files changed, 79 insertions(+), 79 deletions(-) create mode 100644 debian/NEWS delete mode 100644 debian/libpam-zfs.install create mode 100644 debian/libpam-zfs.install.in delete mode 100644 debian/libzfsbootenv1linux.install -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux 2/5] debian: remove libzfsbootenv1linux.install
When initially packaging libzfsbootenv1linux a .install file was commited in addtion to the .install.in (which contains the proper path with the multiarch component). This wasn't noticed during building because the .install got clobbered while building Fixes: fd0cc4becda91c46991e723ae6f0b08dcdf661e6 Signed-off-by: Stoiko Ivanov --- debian/libzfsbootenv1linux.install | 1 - 1 file changed, 1 deletion(-) delete mode 100644 debian/libzfsbootenv1linux.install diff --git a/debian/libzfsbootenv1linux.install b/debian/libzfsbootenv1linux.install deleted file mode 100644 index 01d3f0c3.. --- a/debian/libzfsbootenv1linux.install +++ /dev/null @@ -1 +0,0 @@ -lib/libzfsbootenv.so.* -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux 3/5] debian: install files to /usr
follows commit 902967808693510631a198414dc57dc4ca564afe in debian upstream [0], which was done to support usrmerge [0] https://salsa.debian.org/zfsonlinux-team/zfs Signed-off-by: Stoiko Ivanov --- debian/libnvpair3linux.install.in | 2 +- debian/libpam-zfs.install | 2 - debian/libpam-zfs.install.in | 2 + debian/libuutil3linux.install.in | 2 +- debian/libzfs4linux.install.in| 4 +- debian/libzfsbootenv1linux.install.in | 2 +- debian/libzfslinux-dev.install.in | 6 +- debian/libzpool5linux.install.in | 2 +- debian/rules | 26 +++- .../usr/share/initramfs-tools/hooks/zdev | 10 ++-- debian/zfs-test.install | 4 +- debian/zfs-zed.install| 2 +- debian/zfsutils-linux.install | 60 +-- debian/zfsutils-linux.links | 4 +- 14 files changed, 60 insertions(+), 68 deletions(-) delete mode 100644 debian/libpam-zfs.install create mode 100644 debian/libpam-zfs.install.in diff --git a/debian/libnvpair3linux.install.in b/debian/libnvpair3linux.install.in index ed7b541e..fce54227 100644 --- a/debian/libnvpair3linux.install.in +++ b/debian/libnvpair3linux.install.in @@ -1 +1 @@ -lib/@DEB_HOST_MULTIARCH@/libnvpair.so.* +usr/lib/@DEB_HOST_MULTIARCH@/libnvpair.so.* diff --git a/debian/libpam-zfs.install b/debian/libpam-zfs.install deleted file mode 100644 index c33123f6.. --- a/debian/libpam-zfs.install +++ /dev/null @@ -1,2 +0,0 @@ -lib/*/security/pam_zfs_key.so -usr/share/pam-configs/zfs_key diff --git a/debian/libpam-zfs.install.in b/debian/libpam-zfs.install.in new file mode 100644 index ..036b2e8b --- /dev/null +++ b/debian/libpam-zfs.install.in @@ -0,0 +1,2 @@ +usr/lib/@DEB_HOST_MULTIARCH@/security/pam_zfs_key.so +usr/share/pam-configs/zfs_key diff --git a/debian/libuutil3linux.install.in b/debian/libuutil3linux.install.in index a197d030..bb333867 100644 --- a/debian/libuutil3linux.install.in +++ b/debian/libuutil3linux.install.in @@ -1 +1 @@ -lib/@DEB_HOST_MULTIARCH@/libuutil.so.* +usr/lib/@DEB_HOST_MULTIARCH@/libuutil.so.* diff --git a/debian/libzfs4linux.install.in b/debian/libzfs4linux.install.in index 6765aaee..a9054c14 100644 --- a/debian/libzfs4linux.install.in +++ b/debian/libzfs4linux.install.in @@ -1,2 +1,2 @@ -lib/@DEB_HOST_MULTIARCH@/libzfs.so.* -lib/@DEB_HOST_MULTIARCH@/libzfs_core.so.* +usr/lib/@DEB_HOST_MULTIARCH@/libzfs.so.* +usr/lib/@DEB_HOST_MULTIARCH@/libzfs_core.so.* diff --git a/debian/libzfsbootenv1linux.install.in b/debian/libzfsbootenv1linux.install.in index 49216742..b61b8ab6 100644 --- a/debian/libzfsbootenv1linux.install.in +++ b/debian/libzfsbootenv1linux.install.in @@ -1 +1 @@ -lib/@DEB_HOST_MULTIARCH@/libzfsbootenv.so.* +usr/lib/@DEB_HOST_MULTIARCH@/libzfsbootenv.so.* diff --git a/debian/libzfslinux-dev.install.in b/debian/libzfslinux-dev.install.in index e515ddb8..7605b4db 100644 --- a/debian/libzfslinux-dev.install.in +++ b/debian/libzfslinux-dev.install.in @@ -1,4 +1,4 @@ -lib/@DEB_HOST_MULTIARCH@/*.a usr/lib/@DEB_HOST_MULTIARCH@/ -usr/include +usr/lib/@DEB_HOST_MULTIARCH@/*.a usr/lib/@DEB_HOST_MULTIARCH@/*.so -/usr/lib/@DEB_HOST_MULTIARCH@/pkgconfig/*.pc +usr/lib/@DEB_HOST_MULTIARCH@/pkgconfig +usr/include diff --git a/debian/libzpool5linux.install.in b/debian/libzpool5linux.install.in index b9e872df..0e087a27 100644 --- a/debian/libzpool5linux.install.in +++ b/debian/libzpool5linux.install.in @@ -1 +1 @@ -lib/@DEB_HOST_MULTIARCH@/libzpool.so.* +usr/lib/@DEB_HOST_MULTIARCH@/libzpool.so.* diff --git a/debian/rules b/debian/rules index dc1c1c51..9299985a 100755 --- a/debian/rules +++ b/debian/rules @@ -27,18 +27,19 @@ override_dh_auto_configure: @# Build the userland, but don't build the kernel modules. dh_auto_configure -- \ --bindir=/usr/bin \ - --sbindir=/sbin \ - --libdir=/lib/"$(DEB_HOST_MULTIARCH)" \ - --with-udevdir=/lib/udev \ + --sbindir=/usr/sbin \ + --with-mounthelperdir=/usr/sbin \ + --libdir=/usr/lib/"$(DEB_HOST_MULTIARCH)" \ + --with-udevdir=/usr/lib/udev \ --with-zfsexecdir=/usr/lib/zfs-linux \ --enable-systemd \ --enable-pyzfs \ --with-python=python3 \ - --with-pammoduledir='/lib/$(DEB_HOST_MULTIARCH)/security' \ + --with-pammoduledir='/usr/lib/$(DEB_HOST_MULTIARCH)/security' \ --with-pkgconfigdir='/usr/lib/$(DEB_HOST_MULTIARCH)/pkgconfig' \ - --with-systemdunitdir=/lib/systemd/system \ - --with-systemdpresetdir=/lib/systemd/system-preset \ - --with-systemdgeneratordir=/lib/systemd/system-generators \ + --with-systemdunitdir=/usr/lib/systemd/system \ + --with-systemdpresetdir=/usr/lib/systemd/system-preset \ + --with-systemdgeneratordi
[pve-devel] [PATCH zfsonlinux 1/5] update zfs submodule to 2.2.5
patches did not need adaptation Signed-off-by: Stoiko Ivanov --- upstream | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/upstream b/upstream index 25665920..33174af1 16 --- a/upstream +++ b/upstream @@ -1 +1 @@ -Subproject commit 2566592045780e7be7afc899c2496b1ae3af4f4d +Subproject commit 33174af15112ed5c53299da2d28e763b0163f428 -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux 5/5] d/NEWS: add news for usr-merge
to raise awareness for the changed paths (NEWS entries get displayed when installing updates). follows commit 20e286d2f5c8752ac1372cd084729147cf276f3f in debian upstream [0]. [0] https://salsa.debian.org/zfsonlinux-team/zfs Signed-off-by: Stoiko Ivanov --- debian/NEWS | 13 + 1 file changed, 13 insertions(+) create mode 100644 debian/NEWS diff --git a/debian/NEWS b/debian/NEWS new file mode 100644 index ..464999be --- /dev/null +++ b/debian/NEWS @@ -0,0 +1,13 @@ +zfs-linux (2.2.5-pve1) UNRELEASED; urgency=medium + + zfs-linux has now opted-in usr-merge. No behavioral changes should be noted. + + arc_summary was moved from /usr/sbin to /usr/bin and a link was added to + avoid breaking existing scripts. Users calling these executables should + update all absolute paths used. + + As an exception, the name /usr/bin/arcstat is taken by another different + package, nordugrid-arc-client. Thus zfs-linux still uses /usr/sbin/arcstat. + + -- Proxmox Support Team Mon, 19 Aug 2024 19:00:21 +0200 + -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH storage] storage: add bzip2 support
In general I think the addition of bz2 for ISOs makes sense (if only for OPNSense's sake) - Thanks for addressing this! You want to add bzip2 as dependency to debian/control we probably should also add lzop, zstd, gzip to the dependencies although they are pulled in as dependency of pve-manger (due to vzdump being there) - please do so in a separate commit On Fri, 2 Aug 2024 16:34:14 +0200 Maximiliano Sandoval wrote: > This was requested at [1]. add a 'close #5267' to the commit message as suggested in the developer-docs: https://pve.proxmox.com/wiki/Developer_Documentation#Commits_and_Commit_Messages Your colleagues preparing the release-notes will be grateful. > > [1] https://bugzilla.proxmox.com/show_bug.cgi?id=5267 > > Suggested-by: Stoiko Ivanov > Signed-off-by: Maximiliano Sandoval > --- > > For the sake of not having a commit where tests fail, this is all in one > commit. > Please tell me if I should split it. having each commit working is worth a lot if you have to bisect at some point - and in this case following the changes works well for me - so I'd agree to keep this as one commit > > The idea is to expose this algorithm only for bz2 decompression in the web UI > when downloading ISOs. A followup will contain the patch for pve-manager. you say you only want it for ISOs here ... > > An popular iso that is compressed with bz2 is opnsense [2]. That motivation could also be useful in the commit-message for the future. (I did not expect that bz2 as sole compressed format is something I'd run into nowadays) will give it a spin with the pve-manager commit. > > [2] https://opnsense.org/download/ > > src/PVE/Storage.pm | 3 +++ > src/PVE/Storage/Plugin.pm | 2 +- > src/test/archive_info_test.pm | 6 -- > src/test/list_volumes_test.pm | 11 ++- > src/test/parse_volname_test.pm | 4 ++-- > src/test/path_to_volume_id_test.pm | 13 - > 6 files changed, 28 insertions(+), 11 deletions(-) > > diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm > index 57b2038..0caac71 100755 > --- a/src/PVE/Storage.pm > +++ b/src/PVE/Storage.pm > @@ -1550,16 +1550,19 @@ sub decompressor_info { > gz => ['tar', '-z'], > lzo => ['tar', '--lzop'], > zst => ['tar', '--zstd'], > + bz2 => ['tar', '--bzip2'], > }, > vma => { > gz => ['zcat'], > lzo => ['lzop', '-d', '-c'], > zst => ['zstd', '-q', '-d', '-c'], > + bz2 => ['bzcat', '-q'], ... but add it for tar and vma here? (haven't tried if adding it only for one of the formats works explicitly though) also - but this is really in the realm of cosmetics - why do we have zcat/bzcat, but not zstdcat (lzop does not ship a link to lzopcat ..)? (but again - I don't think there's a need to change stuff here) > }, > iso => { > gz => ['zcat'], > lzo => ['lzop', '-d', '-c'], > zst => ['zstd', '-q', '-d', '-c'], > + bz2 => ['bzcat', '-q'], > }, > }; > > diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm > index 6444390..e5ef9de 100644 > --- a/src/PVE/Storage/Plugin.pm > +++ b/src/PVE/Storage/Plugin.pm > @@ -19,7 +19,7 @@ use JSON; > > use base qw(PVE::SectionConfig); > > -use constant KNOWN_COMPRESSION_FORMATS => ('gz', 'lzo', 'zst'); > +use constant KNOWN_COMPRESSION_FORMATS => ('gz', 'lzo', 'zst', 'bz2'); > use constant COMPRESSOR_RE => join('|', KNOWN_COMPRESSION_FORMATS); > > use constant LOG_EXT => ".log"; > diff --git a/src/test/archive_info_test.pm b/src/test/archive_info_test.pm > index cf03c3d..53e37be 100644 > --- a/src/test/archive_info_test.pm > +++ b/src/test/archive_info_test.pm > @@ -121,11 +121,13 @@ my $decompressor = { > gz => ['tar', '-z'], > lzo => ['tar', '--lzop'], > zst => ['tar', '--zstd'], > + bz2 => ['tar', '--bzip2'], > }, > vma => { > gz => ['zcat'], > lzo => ['lzop', '-d', '-c'], > zst => ['zstd', '-q', '-d', '-c'], > + bz2 => ['bzcat', '-q'], >
[pve-devel] [PATCH manager] pve7to8: fix typo in log message
Signed-off-by: Stoiko Ivanov --- noticed while pulling the changes over to pmg7to8 PVE/CLI/pve7to8.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PVE/CLI/pve7to8.pm b/PVE/CLI/pve7to8.pm index 940fb5c6..416ec79b 100644 --- a/PVE/CLI/pve7to8.pm +++ b/PVE/CLI/pve7to8.pm @@ -235,7 +235,7 @@ sub check_pve_packages { } } elsif ($get_pkg->($kinstalled)) { # with 6.2 kernel being available in both we might want to fine-tune the check? - log_warn("a suitable kernel ($kinstalled) is intalled, but an unsuitable ($kernel_ver) is booted, missing reboot?!"); + log_warn("a suitable kernel ($kinstalled) is installed, but an unsuitable ($kernel_ver) is booted, missing reboot?!"); } else { log_warn("unexpected running and installed kernel '$kernel_ver'."); } -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH proxmox-offline-mirror 0/2] remove snapshot directories vanished on source also on medium
superseded-by: https://lists.proxmox.com/pipermail/pve-devel/2024-July/064545.html On Mon, 24 Jun 2024 21:57:10 +0200 Stoiko Ivanov wrote: > This patchset fixes a small glitch that we noticed in a pom-setup, creating > regular snapshots, without cleaning them up regularly. > > Eventually medium sync becomes quite slow. > After removing many snapshots and running garbage collection both on the > mirror as well as on the medium the run-time for the sync still took quite > long. strace showed that the process still walked through the directories > for each snapshot on the medium - they were not cleaned up after all the > files inside were removed. > > tested the patch locally (which is the reason for patch 1/2). > > Stoiko Ivanov (2): > bump proxmox-time to 2.0 > medium: remove snapshot link directories not present in source mirror > > Cargo.toml | 2 +- > debian/control | 2 +- > src/medium.rs | 18 +- > src/types.rs | 2 +- > 4 files changed, 20 insertions(+), 4 deletions(-) > ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-offline-mirror v2 1/5] bump proxmox-apt to 0.11 and adapt to changes.
The recent changes to proxmox-apt along with the introduction of the proxmox_apt_apit_types crate led to the following changes. Signed-off-by: Stoiko Ivanov --- Cargo.toml | 3 ++- debian/control | 3 ++- src/lib.rs | 5 +++-- src/mirror.rs | 11 +-- 4 files changed, 12 insertions(+), 10 deletions(-) diff --git a/Cargo.toml b/Cargo.toml index edcdb87..39461c5 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -26,7 +26,8 @@ sequoia-openpgp = { version = "1.12" } walkdir = "2.3.1" xz2 = "0.1" -proxmox-apt = { version = "0.10.9" } +proxmox-apt = "0.11" +proxmox-apt-api-types = "1.0" proxmox-async = "0.4" proxmox-http = { version = "0.9", features = [ "client-sync", "client-trait" ]} proxmox-router = { version = "2", features = [ "cli" ], default_features = false } diff --git a/debian/control b/debian/control index 86b6cfb..8063c2b 100644 --- a/debian/control +++ b/debian/control @@ -17,7 +17,8 @@ Build-Depends: bash-completion, librust-lazy-static-1+default-dev (>= 1.4-~~), librust-nix-0.26+default-dev (>= 0.26.1-~~), librust-openssl-0.10+default-dev, - librust-proxmox-apt-0.10+default-dev (>= 0.10.9~~), + librust-proxmox-apt-api-types-dev, + librust-proxmox-apt-0.11+default-dev, librust-proxmox-async-0.4+default-dev, librust-proxmox-http-0.9+client-sync-dev, librust-proxmox-http-0.9+client-trait-dev, diff --git a/src/lib.rs b/src/lib.rs index 8de1f33..8d5ca3f 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -16,7 +16,8 @@ use std::{ use anyhow::{format_err, Error}; use medium::MirrorInfo; -use proxmox_apt::repositories::{APTRepository, APTRepositoryFile, APTRepositoryFileType}; +use proxmox_apt::repositories::{APTRepositoryFileImpl, APTRepositoryImpl}; +use proxmox_apt_api_types::{APTRepository, APTRepositoryFile, APTRepositoryFileType}; use types::Snapshot; /// Main configuration file containing definitions of mirrors, external media and subscription keys. @@ -140,7 +141,7 @@ pub fn generate_repo_file_line( repo.uris = vec![format!("file://{}", snapshot_path)]; repo.options -.push(proxmox_apt::repositories::APTRepositoryOption { +.push(proxmox_apt_api_types::APTRepositoryOption { key: "check-valid-until".to_string(), values: vec!["false".to_string()], }); diff --git a/src/mirror.rs b/src/mirror.rs index 073df86..40ee120 100644 --- a/src/mirror.rs +++ b/src/mirror.rs @@ -20,13 +20,12 @@ use crate::{ types::{Diff, Snapshot, SNAPSHOT_REGEX}, FetchResult, Progress, }; -use proxmox_apt::{ -deb822::{ -CheckSums, CompressionType, FileReference, FileReferenceType, PackagesFile, ReleaseFile, -SourcesFile, -}, -repositories::{APTRepository, APTRepositoryPackageType}, + +use proxmox_apt::deb822::{ +CheckSums, CompressionType, FileReference, FileReferenceType, PackagesFile, ReleaseFile, +SourcesFile, }; +use proxmox_apt_api_types::{APTRepository, APTRepositoryPackageType}; use crate::helpers; -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-offline-mirror v2 4/5] pool: gc: remove empty directories under link_dir
garbage collection currently is quite aggressive in removing all files under the link_dir, which are not a hard-link to a checksum file. removing directories that remain empty below the link_dir should thus not too dangerous. without this patch, removing a snapshot on a mirror, running gc there, and syncing everything to a medium, leaves the medium with an hierarchy of empty directories below the removed snapshot (the files get cleaned up the directories remain). using WalkDir::content_first() seems better than to check for emptiness after each file-removal [0] [0] https://docs.rs/walkdir/latest/walkdir/struct.WalkDir.html#method.contents_first Signed-off-by: Stoiko Ivanov --- src/pool.rs | 13 ++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/src/pool.rs b/src/pool.rs index b4f2a6a..56dcbb4 100644 --- a/src/pool.rs +++ b/src/pool.rs @@ -451,6 +451,7 @@ impl PoolLockGuard<'_> { /// Run a garbage collection, removing /// - any checksum files that have no links outside of `pool_dir` /// - any files in `link_dir` that have no corresponding checksum files +/// - any empty directories below `link_dir` remaining after the file removal pub(crate) fn gc(&self) -> Result<(usize, u64), Error> { let (inode_map, _link_count) = self.get_inode_csum_map()?; @@ -459,7 +460,8 @@ impl PoolLockGuard<'_> { let handle_entry = |entry: Result, count: &mut usize, -size: &mut u64| +size: &mut u64, +remove_empty_dir: bool| -> Result<(), Error> { let path = entry?.into_path(); if path == self.lock_path() { @@ -467,6 +469,10 @@ impl PoolLockGuard<'_> { } let meta = path.metadata()?; +if remove_empty_dir && meta.is_dir() && path.read_dir()?.next().is_none() { +std::fs::remove_dir(path)?; +return Ok(()); +} if !meta.is_file() { return Ok(()); }; @@ -507,11 +513,12 @@ impl PoolLockGuard<'_> { }; WalkDir::new(&self.pool.link_dir) +.contents_first(true) .into_iter() -.try_for_each(|entry| handle_entry(entry, &mut count, &mut size))?; +.try_for_each(|entry| handle_entry(entry, &mut count, &mut size, true))?; WalkDir::new(&self.pool.pool_dir) .into_iter() -.try_for_each(|entry| handle_entry(entry, &mut count, &mut size))?; +.try_for_each(|entry| handle_entry(entry, &mut count, &mut size, false))?; Ok((count, size)) } -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-offline-mirror v2 3/5] pool: unlink_file: fix check for empty directory
path.is_empty() checks for the empty-path, not an empty directory [0]. as the check that the path is below the link_dir happens anyways in the if we can directly call std::fs::remove_dir (which is even safer than the std::fs::remove_dir_all call used in pool::remove_dir()). the oversight seems to have been in place since the intial commit. I ran across the issue when removing many snapshots of a Debian Bookworm repository, syncing this to a medium, and still having a vast amount of empty directories left behind (as debian has one directory per package), which in turn increases the sync run-time. [0] https://docs.rs/nix/latest/nix/trait.NixPath.html#tymethod.is_empty Signed-off-by: Stoiko Ivanov --- src/pool.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/pool.rs b/src/pool.rs index 611dcc9..b4f2a6a 100644 --- a/src/pool.rs +++ b/src/pool.rs @@ -428,11 +428,11 @@ impl PoolLockGuard<'_> { while let Some(parent) = path.parent() { path = parent; -if !self.pool.path_in_link_dir(path) || !path.is_empty() { +if !self.pool.path_in_link_dir(path) || path.read_dir()?.next().is_some() { break; } -remove_dir(path)?; +std::fs::remove_dir(path)?; } Ok(()) -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-offline-mirror v2 5/5] pool: remove unused imports
Signed-off-by: Stoiko Ivanov --- src/pool.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/pool.rs b/src/pool.rs index 56dcbb4..7db976e 100644 --- a/src/pool.rs +++ b/src/pool.rs @@ -1,14 +1,14 @@ use std::{ cmp::max, collections::{hash_map::Entry, HashMap}, -fs::{hard_link, remove_dir, File, Metadata}, +fs::{hard_link, File, Metadata}, ops::Deref, os::linux::fs::MetadataExt, path::{Path, PathBuf}, }; use anyhow::{bail, format_err, Error}; -use nix::{unistd, NixPath}; +use nix::unistd; use proxmox_apt::deb822::CheckSums; use proxmox_sys::fs::{create_path, file_get_contents, replace_file, CreateOptions}; -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-offline-mirror v2 0/5] remove snapshot directories vanished on source also on medium
supersedes: https://lists.proxmox.com/pipermail/pve-devel/2024-June/064278.html v1->v2: * had quite a few chats with Fabian off-list, as noted by him - Thanks! * noticed that the core-issue was probably an error in using path.is_empty (which checks for an empty path '', not for an empty directory) fixed with patch 3/5 * noticed that the current gc-code removes orphaned files (a.k.a. files not hardlinked to a checksum file) in any case - so dropped the idea of an explicit command-line switch (as sync_pool calls gc on the target pool as well) - can gladly rework both to add this as explicit safe-guard, but currently do not think it has much merit * patch 1/5 addresses the recent changes to proxmox-apt - I hope it's ok as is. * patch 2/5 is unrelated, but it confused me enough while going through the code. tested with a local setup. original cover-letter for v1: This patchset fixes a small glitch that we noticed in a pom-setup, creating regular snapshots, without cleaning them up regularly. Eventually medium sync becomes quite slow. After removing many snapshots and running garbage collection both on the mirror as well as on the medium the run-time for the sync still took quite long. strace showed that the process still walked through the directories for each snapshot on the medium - they were not cleaned up after all the files inside were removed. tested the patch locally (which is the reason for patch 1/2). Stoiko Ivanov (5): bump proxmox-apt to 0.11 and adapt to changes. pool: drop superfluous check for impossible path combination pool: unlink_file: fix check for empty directory pool: gc: remove empty directories under link_dir pool: remove unused imports Cargo.toml | 3 ++- debian/control | 3 ++- src/lib.rs | 5 +++-- src/mirror.rs | 11 +-- src/pool.rs| 24 ++-- 5 files changed, 26 insertions(+), 20 deletions(-) -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-offline-mirror v2 2/5] pool: drop superfluous check for impossible path combination
commit c598cb154ef3fa6e1a8af840ded40f713d973d3d changed the pool layout to have the pool directory (.pool for a mirror) on the same level as the link directory (instead of below), to enable pool-sharing across multiple mirrors. the condition will never be true, drop the if statement to avoid confusion in the future. Signed-off-by: Stoiko Ivanov --- src/pool.rs | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/pool.rs b/src/pool.rs index f20f5ea..611dcc9 100644 --- a/src/pool.rs +++ b/src/pool.rs @@ -265,9 +265,6 @@ impl PoolLockGuard<'_> { for link_entry in WalkDir::new(&self.pool.link_dir).into_iter() { let path = link_entry?.into_path(); -if self.path_in_pool(&path) { -continue; -}; let meta = path.metadata()?; if !meta.is_file() { -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-offline-mirror 2/2] medium: remove snapshot link directories not present in source mirror
Currently a medium sync takes care to remove all files and the links in the individual snapshot directories, but leaves the empty snapshot hierarchy in place. This causes a medium sync to take longer than necessary, as we walk through the complete mirror on the medium, including all empty link directories. The status output is misleading as well, still showing more snapshots than actually present (in the sense of containing any files). Noticed on a system, which creates snapshots on a daily basis, after a cleanup removing many snapshots and running garbage collection on the mirror and the medium, still caused the medium sync to be quite slow. Signed-off-by: Stoiko Ivanov --- src/medium.rs | 18 +- src/types.rs | 2 +- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/src/medium.rs b/src/medium.rs index 53df50d..2c2d24e 100644 --- a/src/medium.rs +++ b/src/medium.rs @@ -15,7 +15,7 @@ use serde::{Deserialize, Serialize}; use crate::{ config::{self, ConfigLockGuard, MediaConfig, MirrorConfig}, -generate_repo_file_line, +generate_repo_file_line, mirror, mirror::pool, pool::Pool, types::{Diff, Snapshot, SNAPSHOT_REGEX}, @@ -404,6 +404,22 @@ pub fn sync( Pool::create(&mirror_base, &mirror_pool)? }; +let source_snapshots: HashSet = +mirror::list_snapshots(&mirror)?.iter().cloned().collect(); +let target_snapshots: HashSet = list_snapshots(medium_base, &mirror.id)? +.iter() +.cloned() +.collect(); +let target_only_snapshots = target_snapshots.difference(&source_snapshots); +for snapshot in target_only_snapshots.into_iter() { +let path = target_pool.get_path(Path::new(&snapshot.to_string()))?; +println!( +"Removing snapshot no longer present on mirror: {:?}", +&snapshot.to_string() +); +target_pool.lock()?.remove_dir(&path)?; +} + let source_pool: Pool = pool(&mirror)?; source_pool.lock()?.sync_pool(&target_pool, medium.verify)?; diff --git a/src/types.rs b/src/types.rs index 7544d5e..94eee79 100644 --- a/src/types.rs +++ b/src/types.rs @@ -71,7 +71,7 @@ const_regex! { type: String, format: &ApiStringFormat::Pattern(&SNAPSHOT_REGEX), )] -#[derive(Debug, Clone, Copy, PartialEq, PartialOrd, Eq, Ord)] +#[derive(Debug, Clone, Copy, PartialEq, PartialOrd, Eq, Ord, Hash)] /// Mirror snapshot pub struct Snapshot(i64); -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-offline-mirror 0/2] remove snapshot directories vanished on source also on medium
This patchset fixes a small glitch that we noticed in a pom-setup, creating regular snapshots, without cleaning them up regularly. Eventually medium sync becomes quite slow. After removing many snapshots and running garbage collection both on the mirror as well as on the medium the run-time for the sync still took quite long. strace showed that the process still walked through the directories for each snapshot on the medium - they were not cleaned up after all the files inside were removed. tested the patch locally (which is the reason for patch 1/2). Stoiko Ivanov (2): bump proxmox-time to 2.0 medium: remove snapshot link directories not present in source mirror Cargo.toml | 2 +- debian/control | 2 +- src/medium.rs | 18 +- src/types.rs | 2 +- 4 files changed, 20 insertions(+), 4 deletions(-) -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-offline-mirror 1/2] bump proxmox-time to 2.0
Signed-off-by: Stoiko Ivanov --- Cargo.toml | 2 +- debian/control | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.toml b/Cargo.toml index bf7d754..edcdb87 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -35,4 +35,4 @@ proxmox-section-config = "2" proxmox-serde = "0.1" proxmox-subscription = { version = "0.4.2", features = [ "api-types" ] } proxmox-sys = { version = "0.5", features = [ "timer" ] } -proxmox-time = { version = "1.1.5" } +proxmox-time = "2" diff --git a/debian/control b/debian/control index b8c8ddc..86b6cfb 100644 --- a/debian/control +++ b/debian/control @@ -31,7 +31,7 @@ Build-Depends: bash-completion, librust-proxmox-subscription-0.4+default-dev (>= 0.4.2~), librust-proxmox-sys-0.5+default-dev, librust-proxmox-sys-0.5+timer-dev, - librust-proxmox-time-1+default-dev (>= 1.1.5~), + librust-proxmox-time-2+default-dev, librust-regex-1+default-dev (>= 1.5.5-~~), librust-sequoia-openpgp-1+default-dev (>= 1.7-~~), librust-serde-1+default-dev, -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux v2 1/2] update zfs submodule to 2.2.4 and refresh patches
mostly - drop all patches we had queued up to get kernel 6.8 supported. Signed-off-by: Stoiko Ivanov --- ...md-unit-for-importing-specific-pools.patch | 4 +- ...-move-manpage-arcstat-1-to-arcstat-8.patch | 2 +- ...-guard-access-to-l2arc-MFU-MRU-stats.patch | 12 +- ...hten-bounds-for-noalloc-stat-availab.patch | 4 +- ...rectly-handle-partition-16-and-later.patch | 52 -- ...-use-splice_copy_file_range-for-fall.patch | 135 .../0014-linux-5.4-compat-page_size.patch | 121 .../patches/0015-abd-add-page-iterator.patch | 334 - ...-existing-functions-to-vdev_classic_.patch | 349 - ...v_disk-reorganise-vdev_disk_io_start.patch | 111 --- ...-read-write-IO-function-configurable.patch | 69 -- ...e-BIO-filling-machinery-to-avoid-spl.patch | 671 -- ...dule-parameter-to-select-BIO-submiss.patch | 104 --- ...se-bio_chain-to-submit-multiple-BIOs.patch | 363 -- ...on-t-use-compound-heads-on-Linux-4.5.patch | 96 --- ...ault-to-classic-submission-for-2.2.x.patch | 90 --- ...ion-caused-by-mmap-flushing-problems.patch | 104 --- ...touch-vbio-after-its-handed-off-to-t.patch | 57 -- debian/patches/series | 14 - upstream | 2 +- 20 files changed, 12 insertions(+), 2682 deletions(-) delete mode 100644 debian/patches/0012-udev-correctly-handle-partition-16-and-later.patch delete mode 100644 debian/patches/0013-Linux-6.8-compat-use-splice_copy_file_range-for-fall.patch delete mode 100644 debian/patches/0014-linux-5.4-compat-page_size.patch delete mode 100644 debian/patches/0015-abd-add-page-iterator.patch delete mode 100644 debian/patches/0016-vdev_disk-rename-existing-functions-to-vdev_classic_.patch delete mode 100644 debian/patches/0017-vdev_disk-reorganise-vdev_disk_io_start.patch delete mode 100644 debian/patches/0018-vdev_disk-make-read-write-IO-function-configurable.patch delete mode 100644 debian/patches/0019-vdev_disk-rewrite-BIO-filling-machinery-to-avoid-spl.patch delete mode 100644 debian/patches/0020-vdev_disk-add-module-parameter-to-select-BIO-submiss.patch delete mode 100644 debian/patches/0021-vdev_disk-use-bio_chain-to-submit-multiple-BIOs.patch delete mode 100644 debian/patches/0022-abd_iter_page-don-t-use-compound-heads-on-Linux-4.5.patch delete mode 100644 debian/patches/0023-vdev_disk-default-to-classic-submission-for-2.2.x.patch delete mode 100644 debian/patches/0024-Fix-corruption-caused-by-mmap-flushing-problems.patch delete mode 100644 debian/patches/0025-vdev_disk-don-t-touch-vbio-after-its-handed-off-to-t.patch diff --git a/debian/patches/0007-Add-systemd-unit-for-importing-specific-pools.patch b/debian/patches/0007-Add-systemd-unit-for-importing-specific-pools.patch index 8232978c..0600296f 100644 --- a/debian/patches/0007-Add-systemd-unit-for-importing-specific-pools.patch +++ b/debian/patches/0007-Add-systemd-unit-for-importing-specific-pools.patch @@ -18,7 +18,7 @@ Signed-off-by: Thomas Lamprecht --- etc/Makefile.am | 1 + etc/systemd/system/50-zfs.preset | 1 + - etc/systemd/system/zfs-imp...@.service.in | 18 + etc/systemd/system/zfs-imp...@.service.in | 18 ++ 3 files changed, 20 insertions(+) create mode 100644 etc/systemd/system/zfs-imp...@.service.in @@ -48,7 +48,7 @@ index e4056a92c..030611419 100644 enable zfs-share.service diff --git a/etc/systemd/system/zfs-imp...@.service.in b/etc/systemd/system/zfs-imp...@.service.in new file mode 100644 -index 0..9b4ee9371 +index 0..5bd19fb79 --- /dev/null +++ b/etc/systemd/system/zfs-imp...@.service.in @@ -0,0 +1,18 @@ diff --git a/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch b/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch index c11c1ae8..9a4aea56 100644 --- a/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch +++ b/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch @@ -15,7 +15,7 @@ Signed-off-by: Thomas Lamprecht rename man/{man1/arcstat.1 => man8/arcstat.8} (99%) diff --git a/man/Makefile.am b/man/Makefile.am -index 45156571e..3713e9371 100644 +index 43bb014dd..a9293468a 100644 --- a/man/Makefile.am +++ b/man/Makefile.am @@ -2,7 +2,6 @@ dist_noinst_man_MANS = \ diff --git a/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch b/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch index f8cb3539..2e7c207d 100644 --- a/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch +++ b/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch @@ -27,7 +27,7 @@ Signed-off-by: Thomas Lamprecht 2 files changed, 21 insertions(+), 21 deletions(-) diff --git a/cmd/arc_summary b/cmd/arc_summary -index 9c69ec4f8..edf94ea2a 100755 +index 100fb1987..86b2260a1 100755 --- a/cmd/arc_summary +++ b/
[pve-devel] [PATCH zfsonlinux v2 2/2] update arc_summary arcstat patch with new introduced values
ZFS 2.2.4 added new kstats for speculative prefetch in: 026fe796465e3da7b27d06ef5338634ee6dd30d8 Adapt our patch introduced with ZFS 2.1 (for the then added MFU/MRU stats), to also deal with the now introduced values not being present (because an old kernel-module does not offer them). Signed-off-by: Stoiko Ivanov --- ...-guard-access-to-freshly-introduced-.patch | 438 ++ ...-guard-access-to-l2arc-MFU-MRU-stats.patch | 113 - debian/patches/series | 2 +- 3 files changed, 439 insertions(+), 114 deletions(-) create mode 100644 debian/patches/0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch delete mode 100644 debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch diff --git a/debian/patches/0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch b/debian/patches/0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch new file mode 100644 index ..bc7db2a9 --- /dev/null +++ b/debian/patches/0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch @@ -0,0 +1,438 @@ +From Mon Sep 17 00:00:00 2001 +From: Thomas Lamprecht +Date: Wed, 10 Nov 2021 09:29:47 +0100 +Subject: [PATCH] arc stat/summary: guard access to freshly introduced stats + +l2arc MFU/MRU and zfetch past future and stride stats were introduced +in 2.1 and 2.2.4 respectively: + +commit 085321621e79a75bea41c2b6511da6ebfbf2ba0a added printing MFU +and MRU stats for 2.1 user space tools, but those keys are not +available in the 2.0 module. That means it may break the arcstat and +arc_summary tools after upgrade to 2.1 (user space), before a reboot +to the new 2.1 ZFS kernel-module happened, due to python raising a +KeyError on the dict access then. + +Move those two keys to a .get accessor with `0` as fallback, as it +should be better to show some possible wrong data for new stat-keys +than throwing an exception. + +also move l2_mfu_asize l2_mru_asize l2_prefetch_asize +l2_bufc_data_asize l2_bufc_metadata_asize to .get accessor +(these are only present with a cache device in the pool) + +guard access to iohits and uncached state introduced in +792a6ee462efc15a7614f27e13f0f8aaa9414a08 + +guard access to zfetch past future stride stats introduced in +026fe796465e3da7b27d06ef5338634ee6dd30d8 + +These are present in the current kernel, but lead to an exception, if +running the new user-space with an old kernel module. + +Signed-off-by: Stoiko Ivanov +--- + cmd/arc_summary | 132 + cmd/arcstat.in | 48 +- + 2 files changed, 90 insertions(+), 90 deletions(-) + +diff --git a/cmd/arc_summary b/cmd/arc_summary +index 100fb1987..30f5d23e9 100755 +--- a/cmd/arc_summary b/cmd/arc_summary +@@ -551,21 +551,21 @@ def section_arc(kstats_dict): + arc_target_size = arc_stats['c'] + arc_max = arc_stats['c_max'] + arc_min = arc_stats['c_min'] +-meta = arc_stats['meta'] +-pd = arc_stats['pd'] +-pm = arc_stats['pm'] +-anon_data = arc_stats['anon_data'] +-anon_metadata = arc_stats['anon_metadata'] +-mfu_data = arc_stats['mfu_data'] +-mfu_metadata = arc_stats['mfu_metadata'] +-mru_data = arc_stats['mru_data'] +-mru_metadata = arc_stats['mru_metadata'] +-mfug_data = arc_stats['mfu_ghost_data'] +-mfug_metadata = arc_stats['mfu_ghost_metadata'] +-mrug_data = arc_stats['mru_ghost_data'] +-mrug_metadata = arc_stats['mru_ghost_metadata'] +-unc_data = arc_stats['uncached_data'] +-unc_metadata = arc_stats['uncached_metadata'] ++meta = arc_stats.get('meta', 0) ++pd = arc_stats.get('pd', 0) ++pm = arc_stats.get('pm', 0) ++anon_data = arc_stats.get('anon_data', 0) ++anon_metadata = arc_stats.get('anon_metadata', 0) ++mfu_data = arc_stats.get('mfu_data', 0) ++mfu_metadata = arc_stats.get('mfu_metadata', 0) ++mru_data = arc_stats.get('mru_data', 0) ++mru_metadata = arc_stats.get('mru_metadata', 0) ++mfug_data = arc_stats.get('mfu_ghost_data', 0) ++mfug_metadata = arc_stats.get('mfu_ghost_metadata', 0) ++mrug_data = arc_stats.get('mru_ghost_data', 0) ++mrug_metadata = arc_stats.get('mru_ghost_metadata', 0) ++unc_data = arc_stats.get('uncached_data', 0) ++unc_metadata = arc_stats.get('uncached_metadata', 0) + bonus_size = arc_stats['bonus_size'] + dnode_limit = arc_stats['arc_dnode_limit'] + dnode_size = arc_stats['dnode_size'] +@@ -655,13 +655,13 @@ def section_arc(kstats_dict): + prt_i1('L2 cached evictions:', f_bytes(arc_stats['evict_l
[pve-devel] [PATCH zfsonlinux v2 0/2] Update to ZFS 2.2.4
v1->v2: Patch 2/2 (adaptation of arc_summary/arcstat patch) modified: * right after sending the v1 I saw a report where pinning kernel 6.2 (thus ZFS 2.1) leads to a similar traceback - which I seem to have overlooked when packaging 2.2.0 ... adapted the patch by booting a VM with kernel 6.2 and the current userspace and running arc_summary /arcstat -a until no traceback was displayed with a single-disk pool. original cover-letter for v1: This patchset updates ZFS to the recently released 2.2.4 We had about half of the patches already in 2.2.3-2, due to the needed support for kernel 6.8. Compared to the last 2.2 point releases this one compares quite a few potential performance improvments: * for ZVOL workloads (relevant for qemu guests) multiple taskq were introduced [1] - this change is active by default (can be put back to the old behavior with explicitly setting `zvol_num_taskqs=1` * the interface for ZFS submitting operations to the kernel's block layer was augmented to better deal with split-pages [2] - which should also improve performance, and prevent unaligned writes which are rejected by e.g. the SCSI subsystem. - The default remains with the current code (`zfs_vdev_disk_classic=0` turns on the 'new' behavior...) * Speculative prefetching was improved [3], which introduced not kstats, which are reported by`arc_summary` and `arcstat`, as before with the MRU/MFU additions there was not guard for running the new user-space with an old kernel resulting in Python exceptions of both tools. I adapted the patch where Thomas fixed that back in the 2.1 release times. - sending as separate patch for easier review - and I hope it's ok that I dropped the S-o-b tag (as it's changed code) - glad to resend it, if this should be adapted. Minimally tested on 2 VMs (the arcstat/arc_summary changes by running with an old kernel and new user-space) [0] https://github.com/openzfs/zfs/releases/tag/zfs-2.2.4 [1] https://github.com/openzfs/zfs/pull/15992 [2] https://github.com/openzfs/zfs/pull/15588 [3] https://github.com/openzfs/zfs/pull/16022 Stoiko Ivanov (2): update zfs submodule to 2.2.4 and refresh patches update arc_summary arcstat patch with new introduced values ...md-unit-for-importing-specific-pools.patch | 4 +- ...-move-manpage-arcstat-1-to-arcstat-8.patch | 2 +- ...-guard-access-to-freshly-introduced-.patch | 438 ...-guard-access-to-l2arc-MFU-MRU-stats.patch | 113 --- ...hten-bounds-for-noalloc-stat-availab.patch | 4 +- ...rectly-handle-partition-16-and-later.patch | 52 -- ...-use-splice_copy_file_range-for-fall.patch | 135 .../0014-linux-5.4-compat-page_size.patch | 121 .../patches/0015-abd-add-page-iterator.patch | 334 - ...-existing-functions-to-vdev_classic_.patch | 349 - ...v_disk-reorganise-vdev_disk_io_start.patch | 111 --- ...-read-write-IO-function-configurable.patch | 69 -- ...e-BIO-filling-machinery-to-avoid-spl.patch | 671 -- ...dule-parameter-to-select-BIO-submiss.patch | 104 --- ...se-bio_chain-to-submit-multiple-BIOs.patch | 363 -- ...on-t-use-compound-heads-on-Linux-4.5.patch | 96 --- ...ault-to-classic-submission-for-2.2.x.patch | 90 --- ...ion-caused-by-mmap-flushing-problems.patch | 104 --- ...touch-vbio-after-its-handed-off-to-t.patch | 57 -- debian/patches/series | 16 +- upstream | 2 +- 21 files changed, 445 insertions(+), 2790 deletions(-) create mode 100644 debian/patches/0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch delete mode 100644 debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch delete mode 100644 debian/patches/0012-udev-correctly-handle-partition-16-and-later.patch delete mode 100644 debian/patches/0013-Linux-6.8-compat-use-splice_copy_file_range-for-fall.patch delete mode 100644 debian/patches/0014-linux-5.4-compat-page_size.patch delete mode 100644 debian/patches/0015-abd-add-page-iterator.patch delete mode 100644 debian/patches/0016-vdev_disk-rename-existing-functions-to-vdev_classic_.patch delete mode 100644 debian/patches/0017-vdev_disk-reorganise-vdev_disk_io_start.patch delete mode 100644 debian/patches/0018-vdev_disk-make-read-write-IO-function-configurable.patch delete mode 100644 debian/patches/0019-vdev_disk-rewrite-BIO-filling-machinery-to-avoid-spl.patch delete mode 100644 debian/patches/0020-vdev_disk-add-module-parameter-to-select-BIO-submiss.patch delete mode 100644 debian/patches/0021-vdev_disk-use-bio_chain-to-submit-multiple-BIOs.patch delete mode 100644 debian/patches/0022-abd_iter_page-don-t-use-compound-heads-on-Linux-4.5.patch delete mode 100644 debian/patches/0023-vdev_disk-default-to-classic-submission-for-2.2.x.patch delete mode 100644 debian/patches/0024-Fix-corruption-caused-by-mmap-flushing-problems.patch delete mode 100644 debian/patche
Re: [pve-devel] [PATCH zfsonlinux 2/2] update arc_summary arcstat patch with new introduced values
10 minutes after sending this - I saw a report about pvereport ending in a Python stacktrace - took me a while to see that a similar issue is present between 2.1 and 2.2 - will send the series again with those changes also added (this time the method was changing the source until no more stacktraces were present with the current userspace and kernel 6.2 (with ZFS 2.1) running). Not sure if dropping the whole patch or alternatively cleaning it up once every major PVE release would also be an option (although tbh - I expect quite a few monitoring tools to collect data from these utils - and having that throw exceptions will probably cause some discomfort to our users...) On Tue, 7 May 2024 15:38:36 +0200 Stoiko Ivanov wrote: > ZFS 2.2.4 added new kstats for speculative prefetch in: > 026fe796465e3da7b27d06ef5338634ee6dd30d8 > > Adapt our patch introduced with ZFS 2.1 (for the then added MFU/MRU > stats), to also deal with the now introduced values not being present > (because an old kernel-module does not offer them). > > Signed-off-by: Stoiko Ivanov > --- > ...guard-access-to-freshly-introduced-.patch} | 79 --- > debian/patches/series | 2 +- > 2 files changed, 69 insertions(+), 12 deletions(-) > rename > debian/patches/{0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch > => 0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch} (61%) > > diff --git > a/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch > > b/debian/patches/0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch > similarity index 61% > rename from > debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch > rename to > debian/patches/0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch > index 2e7c207d..a0768923 100644 > --- > a/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch > +++ > b/debian/patches/0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch > @@ -1,7 +1,10 @@ > From Mon Sep 17 00:00:00 2001 > From: Thomas Lamprecht > Date: Wed, 10 Nov 2021 09:29:47 +0100 > -Subject: [PATCH] arc stat/summary: guard access to l2arc MFU/MRU stats > +Subject: [PATCH] arc stat/summary: guard access to freshly introduced stats > + > +l2arc MFU/MRU and zfetch past future and stride stats were introduced > +in 2.1 and 2.2.4 respectively: > > commit 085321621e79a75bea41c2b6511da6ebfbf2ba0a added printing MFU > and MRU stats for 2.1 user space tools, but those keys are not > @@ -14,20 +17,24 @@ Move those two keys to a .get accessor with `0` as > fallback, as it > should be better to show some possible wrong data for new stat-keys > than throwing an exception. > > -Signed-off-by: Thomas Lamprecht > - > also move l2_mfu_asize l2_mru_asize l2_prefetch_asize > l2_bufc_data_asize l2_bufc_metadata_asize to .get accessor > (these are only present with a cache device in the pool) > + > +guard access to zfetch past future stride stats introduced in > +026fe796465e3da7b27d06ef5338634ee6dd30d8 > + > +These are present in the current kernel, but lead to an exception, if > +running the new user-space with an old kernel module. > + > Signed-off-by: Stoiko Ivanov > -Signed-off-by: Thomas Lamprecht > --- > - cmd/arc_summary | 28 ++-- > - cmd/arcstat.in | 14 +++--- > - 2 files changed, 21 insertions(+), 21 deletions(-) > + cmd/arc_summary | 40 > + cmd/arcstat.in | 26 +- > + 2 files changed, 33 insertions(+), 33 deletions(-) > > diff --git a/cmd/arc_summary b/cmd/arc_summary > -index 100fb1987..86b2260a1 100755 > +index 100fb1987..5fb2cdbbc 100755 > --- a/cmd/arc_summary > +++ b/cmd/arc_summary > @@ -655,13 +655,13 @@ def section_arc(kstats_dict): > @@ -48,6 +55,39 @@ index 100fb1987..86b2260a1 100755 > prt_i1('L2 ineligible evictions:', > f_bytes(arc_stats['evict_l2_ineligible'])) > print() > +@@ -794,26 +794,26 @@ def section_dmu(kstats_dict): > + zfetch_stats = isolate_section('zfetchstats', kstats_dict) > + > + zfetch_access_total = int(zfetch_stats['hits']) +\ > +-int(zfetch_stats['future']) + int(zfetch_stats['stride']) +\ > +-int(zfetch_stats['past']) + int(zfetch_stats['misses']) > ++int(zfetch_stats.get('future', 0)) + int(zfetch_stats.get('stride', > 0)) +\ > ++int(zfetch_stats.get('past', 0)) + int(zfetch_stats['misses']) > + > + prt_1('D
[pve-devel] [PATCH zfsonlinux 1/2] update zfs submodule to 2.2.4 and refresh patches
mostly - drop all patches we had queued up to get kernel 6.8 supported. Signed-off-by: Stoiko Ivanov --- ...md-unit-for-importing-specific-pools.patch | 4 +- ...-move-manpage-arcstat-1-to-arcstat-8.patch | 2 +- ...-guard-access-to-l2arc-MFU-MRU-stats.patch | 12 +- ...hten-bounds-for-noalloc-stat-availab.patch | 4 +- ...rectly-handle-partition-16-and-later.patch | 52 -- ...-use-splice_copy_file_range-for-fall.patch | 135 .../0014-linux-5.4-compat-page_size.patch | 121 .../patches/0015-abd-add-page-iterator.patch | 334 - ...-existing-functions-to-vdev_classic_.patch | 349 - ...v_disk-reorganise-vdev_disk_io_start.patch | 111 --- ...-read-write-IO-function-configurable.patch | 69 -- ...e-BIO-filling-machinery-to-avoid-spl.patch | 671 -- ...dule-parameter-to-select-BIO-submiss.patch | 104 --- ...se-bio_chain-to-submit-multiple-BIOs.patch | 363 -- ...on-t-use-compound-heads-on-Linux-4.5.patch | 96 --- ...ault-to-classic-submission-for-2.2.x.patch | 90 --- ...ion-caused-by-mmap-flushing-problems.patch | 104 --- ...touch-vbio-after-its-handed-off-to-t.patch | 57 -- debian/patches/series | 14 - upstream | 2 +- 20 files changed, 12 insertions(+), 2682 deletions(-) delete mode 100644 debian/patches/0012-udev-correctly-handle-partition-16-and-later.patch delete mode 100644 debian/patches/0013-Linux-6.8-compat-use-splice_copy_file_range-for-fall.patch delete mode 100644 debian/patches/0014-linux-5.4-compat-page_size.patch delete mode 100644 debian/patches/0015-abd-add-page-iterator.patch delete mode 100644 debian/patches/0016-vdev_disk-rename-existing-functions-to-vdev_classic_.patch delete mode 100644 debian/patches/0017-vdev_disk-reorganise-vdev_disk_io_start.patch delete mode 100644 debian/patches/0018-vdev_disk-make-read-write-IO-function-configurable.patch delete mode 100644 debian/patches/0019-vdev_disk-rewrite-BIO-filling-machinery-to-avoid-spl.patch delete mode 100644 debian/patches/0020-vdev_disk-add-module-parameter-to-select-BIO-submiss.patch delete mode 100644 debian/patches/0021-vdev_disk-use-bio_chain-to-submit-multiple-BIOs.patch delete mode 100644 debian/patches/0022-abd_iter_page-don-t-use-compound-heads-on-Linux-4.5.patch delete mode 100644 debian/patches/0023-vdev_disk-default-to-classic-submission-for-2.2.x.patch delete mode 100644 debian/patches/0024-Fix-corruption-caused-by-mmap-flushing-problems.patch delete mode 100644 debian/patches/0025-vdev_disk-don-t-touch-vbio-after-its-handed-off-to-t.patch diff --git a/debian/patches/0007-Add-systemd-unit-for-importing-specific-pools.patch b/debian/patches/0007-Add-systemd-unit-for-importing-specific-pools.patch index 8232978c..0600296f 100644 --- a/debian/patches/0007-Add-systemd-unit-for-importing-specific-pools.patch +++ b/debian/patches/0007-Add-systemd-unit-for-importing-specific-pools.patch @@ -18,7 +18,7 @@ Signed-off-by: Thomas Lamprecht --- etc/Makefile.am | 1 + etc/systemd/system/50-zfs.preset | 1 + - etc/systemd/system/zfs-imp...@.service.in | 18 + etc/systemd/system/zfs-imp...@.service.in | 18 ++ 3 files changed, 20 insertions(+) create mode 100644 etc/systemd/system/zfs-imp...@.service.in @@ -48,7 +48,7 @@ index e4056a92c..030611419 100644 enable zfs-share.service diff --git a/etc/systemd/system/zfs-imp...@.service.in b/etc/systemd/system/zfs-imp...@.service.in new file mode 100644 -index 0..9b4ee9371 +index 0..5bd19fb79 --- /dev/null +++ b/etc/systemd/system/zfs-imp...@.service.in @@ -0,0 +1,18 @@ diff --git a/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch b/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch index c11c1ae8..9a4aea56 100644 --- a/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch +++ b/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch @@ -15,7 +15,7 @@ Signed-off-by: Thomas Lamprecht rename man/{man1/arcstat.1 => man8/arcstat.8} (99%) diff --git a/man/Makefile.am b/man/Makefile.am -index 45156571e..3713e9371 100644 +index 43bb014dd..a9293468a 100644 --- a/man/Makefile.am +++ b/man/Makefile.am @@ -2,7 +2,6 @@ dist_noinst_man_MANS = \ diff --git a/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch b/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch index f8cb3539..2e7c207d 100644 --- a/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch +++ b/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch @@ -27,7 +27,7 @@ Signed-off-by: Thomas Lamprecht 2 files changed, 21 insertions(+), 21 deletions(-) diff --git a/cmd/arc_summary b/cmd/arc_summary -index 9c69ec4f8..edf94ea2a 100755 +index 100fb1987..86b2260a1 100755 --- a/cmd/arc_summary +++ b/
[pve-devel] [PATCH zfsonlinux 2/2] update arc_summary arcstat patch with new introduced values
ZFS 2.2.4 added new kstats for speculative prefetch in: 026fe796465e3da7b27d06ef5338634ee6dd30d8 Adapt our patch introduced with ZFS 2.1 (for the then added MFU/MRU stats), to also deal with the now introduced values not being present (because an old kernel-module does not offer them). Signed-off-by: Stoiko Ivanov --- ...guard-access-to-freshly-introduced-.patch} | 79 --- debian/patches/series | 2 +- 2 files changed, 69 insertions(+), 12 deletions(-) rename debian/patches/{0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch => 0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch} (61%) diff --git a/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch b/debian/patches/0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch similarity index 61% rename from debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch rename to debian/patches/0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch index 2e7c207d..a0768923 100644 --- a/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch +++ b/debian/patches/0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch @@ -1,7 +1,10 @@ From Mon Sep 17 00:00:00 2001 From: Thomas Lamprecht Date: Wed, 10 Nov 2021 09:29:47 +0100 -Subject: [PATCH] arc stat/summary: guard access to l2arc MFU/MRU stats +Subject: [PATCH] arc stat/summary: guard access to freshly introduced stats + +l2arc MFU/MRU and zfetch past future and stride stats were introduced +in 2.1 and 2.2.4 respectively: commit 085321621e79a75bea41c2b6511da6ebfbf2ba0a added printing MFU and MRU stats for 2.1 user space tools, but those keys are not @@ -14,20 +17,24 @@ Move those two keys to a .get accessor with `0` as fallback, as it should be better to show some possible wrong data for new stat-keys than throwing an exception. -Signed-off-by: Thomas Lamprecht - also move l2_mfu_asize l2_mru_asize l2_prefetch_asize l2_bufc_data_asize l2_bufc_metadata_asize to .get accessor (these are only present with a cache device in the pool) + +guard access to zfetch past future stride stats introduced in +026fe796465e3da7b27d06ef5338634ee6dd30d8 + +These are present in the current kernel, but lead to an exception, if +running the new user-space with an old kernel module. + Signed-off-by: Stoiko Ivanov -Signed-off-by: Thomas Lamprecht --- - cmd/arc_summary | 28 ++-- - cmd/arcstat.in | 14 +++--- - 2 files changed, 21 insertions(+), 21 deletions(-) + cmd/arc_summary | 40 + cmd/arcstat.in | 26 +- + 2 files changed, 33 insertions(+), 33 deletions(-) diff --git a/cmd/arc_summary b/cmd/arc_summary -index 100fb1987..86b2260a1 100755 +index 100fb1987..5fb2cdbbc 100755 --- a/cmd/arc_summary +++ b/cmd/arc_summary @@ -655,13 +655,13 @@ def section_arc(kstats_dict): @@ -48,6 +55,39 @@ index 100fb1987..86b2260a1 100755 prt_i1('L2 ineligible evictions:', f_bytes(arc_stats['evict_l2_ineligible'])) print() +@@ -794,26 +794,26 @@ def section_dmu(kstats_dict): + zfetch_stats = isolate_section('zfetchstats', kstats_dict) + + zfetch_access_total = int(zfetch_stats['hits']) +\ +-int(zfetch_stats['future']) + int(zfetch_stats['stride']) +\ +-int(zfetch_stats['past']) + int(zfetch_stats['misses']) ++int(zfetch_stats.get('future', 0)) + int(zfetch_stats.get('stride', 0)) +\ ++int(zfetch_stats.get('past', 0)) + int(zfetch_stats['misses']) + + prt_1('DMU predictive prefetcher calls:', f_hits(zfetch_access_total)) + prt_i2('Stream hits:', +f_perc(zfetch_stats['hits'], zfetch_access_total), +f_hits(zfetch_stats['hits'])) +-future = int(zfetch_stats['future']) + int(zfetch_stats['stride']) ++future = int(zfetch_stats.get('future', 0)) + int(zfetch_stats.get('stride', 0)) + prt_i2('Hits ahead of stream:', f_perc(future, zfetch_access_total), +f_hits(future)) + prt_i2('Hits behind stream:', +- f_perc(zfetch_stats['past'], zfetch_access_total), +- f_hits(zfetch_stats['past'])) ++ f_perc(zfetch_stats.get('past', 0), zfetch_access_total), ++ f_hits(zfetch_stats.get('past', 0))) + prt_i2('Stream misses:', +f_perc(zfetch_stats['misses'], zfetch_access_total), +f_hits(zfetch_stats['misses'])) + prt_i2('Streams limit reached:', +f_perc(zfetch_stats['max_streams'], zfetch_stats['misses']),
[pve-devel] [PATCH zfsonlinux 0/2] Update to ZFS 2.2.4
This patchset updates ZFS to the recently released 2.2.4 We had about half of the patches already in 2.2.3-2, due to the needed support for kernel 6.8. Compared to the last 2.2 point releases this one compares quite a few potential performance improvments: * for ZVOL workloads (relevant for qemu guests) multiple taskq were introduced [1] - this change is active by default (can be put back to the old behavior with explicitly setting `zvol_num_taskqs=1` * the interface for ZFS submitting operations to the kernel's block layer was augmented to better deal with split-pages [2] - which should also improve performance, and prevent unaligned writes which are rejected by e.g. the SCSI subsystem. - The default remains with the current code (`zfs_vdev_disk_classic=0` turns on the 'new' behavior...) * Speculative prefetching was improved [3], which introduced not kstats, which are reported by`arc_summary` and `arcstat`, as before with the MRU/MFU additions there was not guard for running the new user-space with an old kernel resulting in Python exceptions of both tools. I adapted the patch where Thomas fixed that back in the 2.1 release times. - sending as separate patch for easier review - and I hope it's ok that I dropped the S-o-b tag (as it's changed code) - glad to resend it, if this should be adapted. Minimally tested on 2 VMs (the arcstat/arc_summary changes by running with an old kernel and new user-space) [0] https://github.com/openzfs/zfs/releases/tag/zfs-2.2.4 [1] https://github.com/openzfs/zfs/pull/15992 [2] https://github.com/openzfs/zfs/pull/15588 [3] https://github.com/openzfs/zfs/pull/16022 Stoiko Ivanov (2): update zfs submodule to 2.2.4 and refresh patches update arc_summary arcstat patch with new introduced values ...md-unit-for-importing-specific-pools.patch | 4 +- ...-move-manpage-arcstat-1-to-arcstat-8.patch | 2 +- ...guard-access-to-freshly-introduced-.patch} | 81 ++- ...hten-bounds-for-noalloc-stat-availab.patch | 4 +- ...rectly-handle-partition-16-and-later.patch | 52 -- ...-use-splice_copy_file_range-for-fall.patch | 135 .../0014-linux-5.4-compat-page_size.patch | 121 .../patches/0015-abd-add-page-iterator.patch | 334 - ...-existing-functions-to-vdev_classic_.patch | 349 - ...v_disk-reorganise-vdev_disk_io_start.patch | 111 --- ...-read-write-IO-function-configurable.patch | 69 -- ...e-BIO-filling-machinery-to-avoid-spl.patch | 671 -- ...dule-parameter-to-select-BIO-submiss.patch | 104 --- ...se-bio_chain-to-submit-multiple-BIOs.patch | 363 -- ...on-t-use-compound-heads-on-Linux-4.5.patch | 96 --- ...ault-to-classic-submission-for-2.2.x.patch | 90 --- ...ion-caused-by-mmap-flushing-problems.patch | 104 --- ...touch-vbio-after-its-handed-off-to-t.patch | 57 -- debian/patches/series | 16 +- upstream | 2 +- 20 files changed, 76 insertions(+), 2689 deletions(-) rename debian/patches/{0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch => 0009-arc-stat-summary-guard-access-to-freshly-introduced-.patch} (59%) delete mode 100644 debian/patches/0012-udev-correctly-handle-partition-16-and-later.patch delete mode 100644 debian/patches/0013-Linux-6.8-compat-use-splice_copy_file_range-for-fall.patch delete mode 100644 debian/patches/0014-linux-5.4-compat-page_size.patch delete mode 100644 debian/patches/0015-abd-add-page-iterator.patch delete mode 100644 debian/patches/0016-vdev_disk-rename-existing-functions-to-vdev_classic_.patch delete mode 100644 debian/patches/0017-vdev_disk-reorganise-vdev_disk_io_start.patch delete mode 100644 debian/patches/0018-vdev_disk-make-read-write-IO-function-configurable.patch delete mode 100644 debian/patches/0019-vdev_disk-rewrite-BIO-filling-machinery-to-avoid-spl.patch delete mode 100644 debian/patches/0020-vdev_disk-add-module-parameter-to-select-BIO-submiss.patch delete mode 100644 debian/patches/0021-vdev_disk-use-bio_chain-to-submit-multiple-BIOs.patch delete mode 100644 debian/patches/0022-abd_iter_page-don-t-use-compound-heads-on-Linux-4.5.patch delete mode 100644 debian/patches/0023-vdev_disk-default-to-classic-submission-for-2.2.x.patch delete mode 100644 debian/patches/0024-Fix-corruption-caused-by-mmap-flushing-problems.patch delete mode 100644 debian/patches/0025-vdev_disk-don-t-touch-vbio-after-its-handed-off-to-t.patch -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 0/2] fix 2 cosmetic glitches with the tests.
while testing a patch-series today I saw quite a few: ``` Use of uninitialized value... ``` warnings from running the tests. While the issues are cosmetic, and don't harm, the actual build - they were enough to distract me for 10 minutes, which I want to spare others (including my future self). Stoiko Ivanov (2): d/control: add geoip-bin to Build-Depends tests: prevent uninitialized value warning with undef as fqdn debian/control | 1 + test/parse-fqdn.pl | 5 +++-- 2 files changed, 4 insertions(+), 2 deletions(-) -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 2/2] tests: prevent uninitialized value warning with undef as fqdn
cosmetic issue - but was distracting enough to make me look if there's an error. Signed-off-by: Stoiko Ivanov --- test/parse-fqdn.pl | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/test/parse-fqdn.pl b/test/parse-fqdn.pl index 6638fbe..47e0e21 100755 --- a/test/parse-fqdn.pl +++ b/test/parse-fqdn.pl @@ -24,9 +24,10 @@ sub is_parsed { sub is_invalid { my ($fqdn, $expected_err) = @_; +my $print_fqdn = $fqdn // '(undefined)'; my $parsed = eval { parse_fqdn($fqdn) }; -is($parsed, undef, "invalid FQDN did fail parsing: $fqdn"); -is($@, $expected_err, "invalid FQDN threw correct error: $fqdn"); +is($parsed, undef, "invalid FQDN did fail parsing: $print_fqdn"); +is($@, $expected_err, "invalid FQDN threw correct error: $print_fqdn"); } is_invalid(undef, ERR_EMPTY); -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 1/2] d/control: add geoip-bin to Build-Depends
else the tests running: `./proxmox-low-level-installer -t test.img dump-env` print quite a few warnings about the use of uninitialized values (though they still continue happily). This was a slight distraction for me. Signed-off-by: Stoiko Ivanov --- debian/control | 1 + 1 file changed, 1 insertion(+) diff --git a/debian/control b/debian/control index eb4d3be..afe3c70 100644 --- a/debian/control +++ b/debian/control @@ -4,6 +4,7 @@ Priority: optional Maintainer: Proxmox Support Team Build-Depends: cargo:native, debhelper-compat (= 12), + geoip-bin, iproute2, iso-codes, libgtk3-perl, -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH installer v3 0/2] assistant: keep prepared iso bootable on uefi with flash drives
for completeness sake - gave the v3 a quick spin as well - so also from my side the: Reviewed-by: Stoiko Ivanov Tested-by: Stoiko Ivanov still applies :) On Tue, 30 Apr 2024 12:46:07 +0200 Aaron Lauterer wrote: > booting a prepared iso in UEFI mode from a blockdev (e.g. usb flash > drive) did not work as grub could not find the partition. > > we now read the uuid / volume_date from the source iso and always set it > explictly to the same value when injecting files. > > more details in the actual commit message > > the second patch is a style patch > > this version should now include everything. sorry for the noise :) > > changes since: > v2: > * add import of format_err that was missed in v2 > v1: > * improve error handling in case xorriso does return empty output > > Aaron Lauterer (2): > assistant: keep prepared iso bootable on uefi with flash drives > assistant: use single dash for xorriso parameter > > proxmox-auto-install-assistant/src/main.rs | 48 +++--- > 1 file changed, 43 insertions(+), 5 deletions(-) > ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH 1/2] assistant: keep prepared iso bootable on uefi with flash drives
gave this and the next patch a spin on 2 test-servers and one VM, where I could reproduce the issue yesterday - * execsnoop-bpfcc says the patch does what it says on the tin * automated installs worked. tiny nit: subject prefix and a cover-letter would have helped the following goes for both patches: Reviewed-by: Stoiko Ivanov Tested-by: Stoiko Ivanov On Tue, 30 Apr 2024 10:54:33 +0200 Aaron Lauterer wrote: > By mapping files into the ISO, the UUID for the partitions change as > they depend on the timestamp. The result is, that grub cannot find its > partition anymore and the user ends up on the grub shell. > > This only happens when booting from a blockdev in UEFI mode. E.g. a USB > flash drive. Alternatively one can `dd` the ISO to a small (2GiB) VM > disk and mark it as the first boot device. > > Booting in legacy mode or via CDROM (e.g. pass through via IPMI), it > worked. > > Xorriso can report the commands needed to recreate the source ISO. The > '-volume_date uuid' is the one needed to override the same UUIDs. We > therefore read it first from the source iso and pass it as parameter > whenever we inject a file into the iso. > > Signed-off-by: Aaron Lauterer > --- > proxmox-auto-install-assistant/src/main.rs | 44 -- > 1 file changed, 41 insertions(+), 3 deletions(-) > > diff --git a/proxmox-auto-install-assistant/src/main.rs > b/proxmox-auto-install-assistant/src/main.rs > index 0debd29..e9213f7 100644 > --- a/proxmox-auto-install-assistant/src/main.rs > +++ b/proxmox-auto-install-assistant/src/main.rs > @@ -276,6 +276,7 @@ fn show_system_info(_args: &CommandSystemInfo) -> > Result<()> { > > fn prepare_iso(args: &CommandPrepareISO) -> Result<()> { > check_prepare_requirements(args)?; > +let uuid = get_iso_uuid(&args.input)?; > > if args.fetch_from == FetchAnswerFrom::Iso && args.answer_file.is_none() > { > bail!("Missing path to the answer file required for the fetch-from > 'iso' mode."); > @@ -331,10 +332,15 @@ fn prepare_iso(args: &CommandPrepareISO) -> Result<()> { > instmode_file_tmp.push("auto-installer-mode.toml"); > fs::write(&instmode_file_tmp, toml::to_string_pretty(&config)?)?; > > -inject_file_to_iso(&tmp_iso, &instmode_file_tmp, > "/auto-installer-mode.toml")?; > +inject_file_to_iso( > +&tmp_iso, > +&instmode_file_tmp, > +"/auto-installer-mode.toml", > +&uuid, > +)?; > > if let Some(answer_file) = &args.answer_file { > -inject_file_to_iso(&tmp_iso, answer_file, "/answer.toml")?; > +inject_file_to_iso(&tmp_iso, answer_file, "/answer.toml", &uuid)?; > } > > println!("Moving prepared ISO to target location..."); > @@ -371,11 +377,14 @@ fn final_iso_location(args: &CommandPrepareISO) -> > PathBuf { > target.to_path_buf() > } > > -fn inject_file_to_iso(iso: &PathBuf, file: &PathBuf, location: &str) -> > Result<()> { > +fn inject_file_to_iso(iso: &PathBuf, file: &PathBuf, location: &str, uuid: > &String) -> Result<()> { > let result = Command::new("xorriso") > .arg("--boot_image") > .arg("any") > .arg("keep") > +.arg("-volume_date") > +.arg("uuid") > +.arg(uuid) > .arg("-dev") > .arg(iso) > .arg("-map") > @@ -391,6 +400,35 @@ fn inject_file_to_iso(iso: &PathBuf, file: &PathBuf, > location: &str) -> Result<( > Ok(()) > } > > +fn get_iso_uuid(iso: &PathBuf) -> Result { > +let result = Command::new("xorriso") > +.arg("-dev") > +.arg(iso) > +.arg("-report_system_area") > +.arg("cmd") > +.output()?; > +if !result.status.success() { > +bail!( > +"Error determining the UUID of the source ISO: {}", > +String::from_utf8_lossy(&result.stderr) > +); > +} > +let mut uuid = String::new(); > +for line in String::from_utf8(result.stdout)?.lines() { > +if line.starts_with("-volume_date uuid") { > +uuid = line > +.split(' ') > +.last() > +.unwrap() > +.replace('\'', "") > +.trim() > +.into(); > +break; > +} > +} > +Ok(uuid) > +} > + > fn get_disks() -> Result>> { > let unwantend_block_devs = vec![ > "ram[0-9]*", ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH pve-installer] answer: perform basic input validation for keyboard
On Wed, 24 Apr 2024 10:48:50 +0200 Christian Ebner wrote: > Currently it is possible to validate and create an iso with an > invalid keyboad layout, only failing later during installation. > > Add a basic check for correct keyboard layout by defining an enum > with allowed variants. > > Signed-off-by: Christian Ebner > --- > proxmox-auto-installer/src/answer.rs | 39 +++- > proxmox-auto-installer/src/utils.rs | 8 -- > 2 files changed, 44 insertions(+), 3 deletions(-) > > diff --git a/proxmox-auto-installer/src/answer.rs > b/proxmox-auto-installer/src/answer.rs > index a6cf8b7..af7485a 100644 > --- a/proxmox-auto-installer/src/answer.rs > +++ b/proxmox-auto-installer/src/answer.rs > @@ -23,7 +23,7 @@ pub struct Answer { > pub struct Global { > pub country: String, > pub fqdn: Fqdn, > -pub keyboard: String, > +pub keyboard: KeyboardLayout, > pub mailto: String, > pub timezone: String, > pub root_password: String, > @@ -270,3 +270,40 @@ pub struct BtrfsOptions { > pub hdsize: Option, > pub raid: Option, > } > + > +#[derive(Clone, Deserialize, Serialize, Debug, PartialEq)] > +#[serde(rename_all = "kebab-case", deny_unknown_fields)] > +pub enum KeyboardLayout { > +De, > +DeCh, > +Dk, > +EnGb, > +EnUs, > +Es, > +Fi, > +Fr, > +FrBe, > +FrCa, > +FrCh, > +Hu, > +Is, > +It, > +Jp, > +Lt, > +Mk, > +Nl, > +No, > +Pl, > +Pt, > +PtBr, > +Se, > +Si, > +Tr, > +} > + quickly looked at that as well yesterday - and I also ran into the issue that we get the relevant data in the installer itself (where actual validation takes place). with the target to have the auto-install-assistant available as single static binary - I think your approach works well enough - the one thing that might be an improvment is to get the data from country.dat (a build-artefact output from country.pl based on /usr/share/iso-codes/json/iso_3166-1.json ) at build-time and embed it in the binary. (but I did not get around to checking how this is done sensibly in rust) additionally we could verify the country selection as well with that. > +impl std::fmt::Display for KeyboardLayout { > +fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { > +let keyboard_layout = > serde_json::to_value(self).unwrap().to_string(); > +write!(f, "{}", keyboard_layout.trim_matches('\"')) > +} > +} > diff --git a/proxmox-auto-installer/src/utils.rs > b/proxmox-auto-installer/src/utils.rs > index 7e1366c..202ad41 100644 > --- a/proxmox-auto-installer/src/utils.rs > +++ b/proxmox-auto-installer/src/utils.rs > @@ -281,7 +281,11 @@ pub fn verify_locale_settings(answer: &Answer, locales: > &LocaleInfo) -> Result<( > { > bail!("country code '{}' is not valid", &answer.global.country); > } > -if !locales.kmap.keys().any(|i| i == &answer.global.keyboard) { > +if !locales > +.kmap > +.keys() > +.any(|i| i == &answer.global.keyboard.to_string()) > +{ > bail!("keyboard layout '{}' is not valid", &answer.global.keyboard); > } > > @@ -328,7 +332,7 @@ pub fn parse_answer( > > country: answer.global.country.clone(), > timezone: answer.global.timezone.clone(), > -keymap: answer.global.keyboard.clone(), > +keymap: answer.global.keyboard.to_string(), > > password: answer.global.root_password.clone(), > mailto: answer.global.mailto.clone(), ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer] assistant: error out on set network config for dhcp
potentially will not happen too often in practice if the sample files always contain the right source. Still having settings in an answer file that get ignored does not seem right. tested with `validate-answer` on a file without `source` in the network section (which initially caused confusion for me) Signed-off-by: Stoiko Ivanov --- proxmox-auto-installer/src/answer.rs | 13 + 1 file changed, 13 insertions(+) diff --git a/proxmox-auto-installer/src/answer.rs b/proxmox-auto-installer/src/answer.rs index de8a360..df20db3 100644 --- a/proxmox-auto-installer/src/answer.rs +++ b/proxmox-auto-installer/src/answer.rs @@ -87,6 +87,19 @@ impl TryFrom for Network { }), }) } else { +if network.cidr.is_some() { +return Err("Field 'cidr' not supported for 'from-dhcp' config."); +} +if network.dns.is_some() { +return Err("Field 'dns' not supported for 'from-dhcp' config."); +} +if network.gateway.is_some() { +return Err("Field 'gateway' not supported for 'from-dhcp' config."); +} +if network.filter.is_some() { +return Err("Field 'filter' not supported for 'from-dhcp' config."); +} + Ok(Network { network_settings: NetworkSettings::FromDhcp, }) -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager] acme: ui: handle missing meta field in directory response
Gave this patch a spin against a quickly setup step-ca container I had lying around - the issue of not being able to register an account without EAB is gone with it. Tested-by: Stoiko Ivanov On Mon, 22 Apr 2024 11:01:02 +0200 Folke Gleumes wrote: > When none of the meta fields is set by the directory, the whole > dictionary is missing from the response, leading to an exception > when testing for fields inside it. > > Signed-off-by: Folke Gleumes > --- > www/manager6/node/ACME.js | 9 ++--- > 1 file changed, 6 insertions(+), 3 deletions(-) > > diff --git a/www/manager6/node/ACME.js b/www/manager6/node/ACME.js > index a0db51a6..7fe49171 100644 > --- a/www/manager6/node/ACME.js > +++ b/www/manager6/node/ACME.js > @@ -150,15 +150,18 @@ Ext.define('PVE.node.ACMEAccountCreate', { > directory: value, > }, > success: function(response, opt) { > - if (response.result.data.termsOfService) { > + if (response.result.data && > response.result.data.termsOfService) { > > field.setValue(response.result.data.termsOfService); > > disp.setValue(response.result.data.termsOfService); > checkbox.setHidden(false); > } else { > - checkbox.setValue(false); > + // Needed to pass input verification > and enable register button > + // has no influence on the submitted > form > + checkbox.setValue(true); > disp.setValue("No terms of service > agreement required"); > } > - vm.set('eabRequired', > !!response.result.data.externalAccountRequired); > + vm.set('eabRequired', > !!(response.result.data && > + > response.result.data.externalAccountRequired)); > }, > failure: function(response, opt) { > disp.setValue(undefined); ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH docs] firmware: adapt to proxmox packaged fwupd
We ship our own fwupd package, since it needs to handle the differently named efi_os_dir (proxmox vs debian). Due to our experience with `udisks2` causing issues on hypervisors, our package downgraded the Recommends udisks2, to a Suggests. The downside is, that users need to explicitly set their ESP mountpoint in the config file. Additionally a minor stylistic rephrasing (is an option vs. could be an option). Tested this today, while giving our fwupd package a spin. Suggested-by: Fabian Grünbichler Signed-off-by: Stoiko Ivanov --- firmware-updates.adoc | 21 +++-- 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/firmware-updates.adoc b/firmware-updates.adoc index 279cf11..8354955 100644 --- a/firmware-updates.adoc +++ b/firmware-updates.adoc @@ -43,13 +43,22 @@ ConnectX or https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli'] for Broadcom network cards. -* https://fwupd.org[LVFS] could also be an option if there is a cooperation with -a https://fwupd.org/lvfs/vendors/[vendor] and +* https://fwupd.org[LVFS] is also an option if there is a cooperation with +the https://fwupd.org/lvfs/vendors/[hardware vendor] and https://fwupd.org/lvfs/devices/[supported hardware] in use. The technical -requirement for this is that the system was manufactured after 2014, is booted -via UEFI and the easiest way is to mount the EFI partition from which you boot -(`mount /dev/disk/by-partuuid/ /boot/efi`) before installing -'fwupd'. +requirement for this is that the system was manufactured after 2014 and is +booted via UEFI. + +Since {pve} ships its own version of the `fwupd` package, for Secure Boot +Support with the Proxmox signing key, which does not recommend the `udisks2` +package, due to observed issues with its use on hypervisors setting the mount +point of the EFI partition in `/etc/fwupd/daemon.conf` is necessary: + +.File `/etc/fwupd/daemon.conf` + +# Override the location used for the EFI system partition (ESP) path. +EspLocation=/boot/efi + TIP: If the update instructions require a host reboot, make sure that it can be done safely. See also xref:ha_manager_node_maintenance[Node Maintenance]. -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH docs] secure boot: mention proxmox-secure-boot-support metapackage
Signed-off-by: Stoiko Ivanov --- Just had the opportunity to try this on a testsystem - it worked flawlessly :) I did consider dropping the explicit list of packages and replace it by the metapackage only, but think that the additional explanation of how they interact is worth keeping. system-booting.adoc | 9 + 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/system-booting.adoc b/system-booting.adoc index 9603fc0..3847354 100644 --- a/system-booting.adoc +++ b/system-booting.adoc @@ -378,7 +378,8 @@ Secure Boot Since {pve} 8.1, Secure Boot is supported out of the box via signed packages and integration in `proxmox-boot-tool`. -The following packages need to be installed for Secure Boot to be enabled: +The following packages needed for Secure Boot to work, are installed as +dependency of `proxmox-secure-boot-support`: - `shim-signed` (shim bootloader signed by Microsoft) - `shim-helpers-amd64-signed` (fallback bootloader and MOKManager, signed by @@ -404,9 +405,9 @@ well-tested backup of your {pve} host!** An existing UEFI installation can be switched over to Secure Boot if desired, without having to reinstall {pve} from scratch. -First, ensure all your system is up-to-date. Next, install all the required -pre-signed packages as listed above. GRUB automatically creates the needed EFI -boot entry for booting via the default shim. +First, ensure all your system is up-to-date. Next, install +`proxmox-secure-boot-support`. GRUB automatically creates the needed EFI boot +entry for booting via the default shim. .systemd-boot -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH pve-kernel] revert 2 changes in thermal driver causing an early kernel Oops.
The second patch, that is reverted (first): `thermal: trip: Drop lockdep assertion from thermal_zone_trip_id()` only touches code introduced by the first patch. The first patch causes the following Oops (reproduced on an old HP DL380 G8): ``` [2.960519] ACPI: button: Power Button [PWRF] [2.963126] BUG: kernel NULL pointer dereference, address: 000c [2.965667] #PF: supervisor read access in kernel mode [2.966954] #PF: error_code(0x) - not-present page [2.966954] PGD 0 P4D 0 [2.966954] Oops: [#1] PREEMPT SMP PTI [2.966954] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G I 6.5.13-4-pve #1 [2.966954] Hardware name: HP ProLiant DL380p Gen8, BIOS P70 05/24/2019 [2.966954] RIP: 0010:step_wise_throttle+0x48/0x360 [2.966954] Code: 04 25 28 00 00 00 48 89 45 d0 31 c0 48 63 c6 48 8d 14 40 48 8b 87 50 03 00 00 4c 8d 24 90 e8 cf d0 ff ff c6 45 bf 00 89 45 b4 <41> 8b 04 24 41 39 85 78 03 00 00 0f 8d a9 02 00 00 0f 1f 44 00 00 [2.966954] RSP: :9e2b8014bae8 EFLAGS: 00010246 [2.966954] RAX: 0002 RBX: 0001 RCX: [2.966954] RDX: RSI: RDI: [2.966954] RBP: 9e2b8014bb40 R08: R09: [2.966954] R10: R11: R12: 000c [2.966954] R13: 8c7ac421d000 R14: 0001 R15: [2.966954] FS: () GS:8c7def60() knlGS: [2.966954] CS: 0010 DS: ES: CR0: 80050033 [2.966954] CR2: 000c CR3: 000513a34001 CR4: 000606f0 [2.966954] Call Trace: [2.966954] ``` the relevant mainline kernels (6.6.15), corresponding to the Ubuntu-patchset (which mixes changes from 6.6.15, with ones from 6.1.76) [0] - also boot happily - so I strongly assume that the changes depend on one of the many commits introduced in linux-upstream between v6.5.1 and v6.6.1. As it looks like a refactoring (upon which later changes are based), and not a bug-fix in itself - simply dropping it seems sensible. Signed-off-by: Stoiko Ivanov --- ...rip-Drop-lockdep-assertion-from-ther.patch | 24 ++ ...ore-Store-trip-pointer-in-struct-the.patch | 343 ++ 2 files changed, 367 insertions(+) create mode 100644 patches/kernel/0014-Revert-thermal-trip-Drop-lockdep-assertion-from-ther.patch create mode 100644 patches/kernel/0015-Revert-thermal-core-Store-trip-pointer-in-struct-the.patch diff --git a/patches/kernel/0014-Revert-thermal-trip-Drop-lockdep-assertion-from-ther.patch b/patches/kernel/0014-Revert-thermal-trip-Drop-lockdep-assertion-from-ther.patch new file mode 100644 index ..413b1641a4b1 --- /dev/null +++ b/patches/kernel/0014-Revert-thermal-trip-Drop-lockdep-assertion-from-ther.patch @@ -0,0 +1,24 @@ +From Mon Sep 17 00:00:00 2001 +From: Stoiko Ivanov +Date: Thu, 4 Apr 2024 11:41:15 +0200 +Subject: [PATCH] Revert "thermal: trip: Drop lockdep assertion from + thermal_zone_trip_id()" + +This reverts commit c723c4fca6d2db3815623ff4dc0ea51667b56b89. +--- + drivers/thermal/thermal_trip.c | 2 ++ + 1 file changed, 2 insertions(+) + +diff --git a/drivers/thermal/thermal_trip.c b/drivers/thermal/thermal_trip.c +index 68bea8706c597..1d4fe63e09f77 100644 +--- a/drivers/thermal/thermal_trip.c b/drivers/thermal/thermal_trip.c +@@ -201,6 +201,8 @@ int thermal_zone_trip_id(struct thermal_zone_device *tz, + { + int i; + ++ lockdep_assert_held(&tz->lock); ++ + for (i = 0; i < tz->num_trips; i++) { + if (&tz->trips[i] == trip) + return i; diff --git a/patches/kernel/0015-Revert-thermal-core-Store-trip-pointer-in-struct-the.patch b/patches/kernel/0015-Revert-thermal-core-Store-trip-pointer-in-struct-the.patch new file mode 100644 index ..fe1ce3ed6632 --- /dev/null +++ b/patches/kernel/0015-Revert-thermal-core-Store-trip-pointer-in-struct-the.patch @@ -0,0 +1,343 @@ +From 0000 Mon Sep 17 00:00:00 2001 +From: Stoiko Ivanov +Date: Thu, 4 Apr 2024 11:41:17 +0200 +Subject: [PATCH] Revert "thermal: core: Store trip pointer in struct + thermal_instance" + +This reverts commit 643b451957369f28b7770af387d14d4e4712074b. +--- + drivers/thermal/gov_bang_bang.c | 23 +++ + drivers/thermal/gov_fair_share.c | 5 ++--- + drivers/thermal/gov_power_allocator.c | 11 +++ + drivers/thermal/gov_step_wise.c | 16 +--- + drivers/thermal/thermal_core.c| 15 +-- + drivers/thermal/thermal_core.h| 4 +--- + drivers/thermal/thermal_helpers.c | 5 + + drivers/thermal/thermal_sysfs.c | 3 +-- + drivers/thermal/thermal_trip.c| 15 --- + 9 files changed, 37 insertions(+), 6
[pve-devel] [PATCH pve-kernel] revert cifs backport to 6.1 added between 6.5.13-1 and 6.5.13-2
copying files within a cifs-share currently result in the following trace: ``` [ 495.388739] BUG: unable to handle page fault for address: fffe [ 495.388744] #PF: supervisor read access in kernel mode [ 495.388746] #PF: error_code(0x) - not-present page [ 495.388747] PGD 172c3f067 P4D 172c3f067 PUD 172c41067 PMD 0 [ 495.388752] Oops: [#2] PREEMPT SMP NOPTI [ 495.388754] CPU: 1 PID: 3894 Comm: cp Tainted: G D 6.5.0-32-generic #32-Ubuntu [ 495.388756] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 4.2023.08-4 02/15/2024 [ 495.388758] RIP: 0010:cifs_flush_folio+0x41/0xf0 [cifs] ... ``` a quick check identified proxmox-kernel-6.5.13-2 as the first affected version, and `2dc07a11e269bfbe5589e99b60cdbae0118be979` as likely source of the issue. The commit adapts the changes from `7b2404a886f8b91250c31855d287e632123e1746` to work with the code in kernel 6.1. This is not needed as the relevant changes were made in 6.4 and are already part of the 6.5 tree - `66dabbb65d673aef40dd17bf62c042be8f6d4a4b` reverting the commit fixes copying files within a samba share. Tested/reproduced with: * a VM with the kernel as cifs-client * one very crude samba-share allowing guest-write access on a Debian bookworm host * as well as a share using cifscreds + multiuser (`mount.cifs(8)`) * mounting the share, copying any file from one directory to another on the same share (with `cp` and Thunar and Nautilus). Reported to Ubuntu upstream at [1]. [0] https://lore.kernel.org/linux-mm/zzhrpnj3zxmr8...@eldamar.lan/ [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2055002 Reported-by: Daniela Häsler Signed-off-by: Stoiko Ivanov --- ...flushing-folio-regression-for-6.1-ba.patch | 23 +++ 1 file changed, 23 insertions(+) create mode 100644 patches/kernel/0014-Revert-cifs-fix-flushing-folio-regression-for-6.1-ba.patch diff --git a/patches/kernel/0014-Revert-cifs-fix-flushing-folio-regression-for-6.1-ba.patch b/patches/kernel/0014-Revert-cifs-fix-flushing-folio-regression-for-6.1-ba.patch new file mode 100644 index ..e033b68ac69f --- /dev/null +++ b/patches/kernel/0014-Revert-cifs-fix-flushing-folio-regression-for-6.1-ba.patch @@ -0,0 +1,23 @@ +From Mon Sep 17 00:00:00 2001 +From: Stoiko Ivanov +Date: Wed, 3 Apr 2024 10:29:59 +0200 +Subject: [PATCH] Revert "cifs: fix flushing folio regression for 6.1 backport" + +This reverts commit 2dc07a11e269bfbe5589e99b60cdbae0118be979. +--- + fs/smb/client/cifsfs.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/fs/smb/client/cifsfs.c b/fs/smb/client/cifsfs.c +index 55a6d0296ec82..82313b2534631 100644 +--- a/fs/smb/client/cifsfs.c b/fs/smb/client/cifsfs.c +@@ -1245,7 +1245,7 @@ static int cifs_flush_folio(struct inode *inode, loff_t pos, loff_t *_fstart, lo + int rc = 0; + + folio = filemap_get_folio(inode->i_mapping, index); +- if (!folio) ++ if (IS_ERR(folio)) + return 0; + + size = folio_size(folio); -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH zfsonlinux] fix #4835: order zfs-import@ before -cache/-scan
LGTM - Thanks for the idea and patch! gave it a spin by creating a pool through the GUI, rebooting, seeing the spurious warning, installing a patched version, rebooting multiple times (no warning), adding yet another pool and enabling zfs-import-scan.service (did not manage to boot without zpool.cache not being empty :) Reviewed-by: Stoiko Ivanov Tested-by: Stoiko Ivanov On Thu, 28 Mar 2024 11:06:07 +0100 Fabian Grünbichler wrote: > missed the subjectprefix, this is for ZFS ;) thanks @Stoiko for pointing > it out off-list > > On March 28, 2024 10:41 am, Fabian Grünbichler wrote: > > this should fix failures of the template instances because either of the two > > other import services picked up the pool in question first. > > > > Signed-off-by: Fabian Grünbichler > > --- > > ...dd-systemd-unit-for-importing-specific-pools.patch | 11 --- > > 1 file changed, 8 insertions(+), 3 deletions(-) > > > > diff --git > > a/debian/patches/0007-Add-systemd-unit-for-importing-specific-pools.patch > > b/debian/patches/0007-Add-systemd-unit-for-importing-specific-pools.patch > > index ed7adbe..8232978 100644 > > --- > > a/debian/patches/0007-Add-systemd-unit-for-importing-specific-pools.patch > > +++ > > b/debian/patches/0007-Add-systemd-unit-for-importing-specific-pools.patch > > @@ -10,13 +10,16 @@ by scanning /dev/disk/by-id, irrespective of the > > existence and content of > > the instance name is used unescaped (see systemd.unit(5)), since zpool > > names > > can contain characters which will be escaped by systemd. > > > > +Its instances are ordered before the other two "big" import services to > > avoid > > +races and spurious (cosmetic!) service failures. > > + > > Signed-off-by: Stoiko Ivanov > > Signed-off-by: Thomas Lamprecht > > --- > > etc/Makefile.am | 1 + > > etc/systemd/system/50-zfs.preset | 1 + > > - etc/systemd/system/zfs-imp...@.service.in | 16 > > - 3 files changed, 18 insertions(+) > > + etc/systemd/system/zfs-imp...@.service.in | 18 > > + 3 files changed, 20 insertions(+) > > create mode 100644 etc/systemd/system/zfs-imp...@.service.in > > > > diff --git a/etc/Makefile.am b/etc/Makefile.am > > @@ -48,7 +51,7 @@ new file mode 100644 > > index 0..9b4ee9371 > > --- /dev/null > > +++ b/etc/systemd/system/zfs-imp...@.service.in > > -@@ -0,0 +1,16 @@ > > +@@ -0,0 +1,18 @@ > > +[Unit] > > +Description=Import ZFS pool %i > > +Documentation=man:zpool(8) > > @@ -57,6 +60,8 @@ index 0..9b4ee9371 > > +After=cryptsetup.target > > +After=multipathd.target > > +Before=zfs-import.target > > ++Before=zfs-import-scan.service > > ++Before=zfs-import-cache.service > > + > > +[Service] > > +Type=oneshot > > -- > > 2.39.2 > > > > > > > > ___ > > pve-devel mailing list > > pve-devel@lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > > > > ___ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager 9/9] report: add microcode info to better assess possible system impacts
On Fri, 22 Mar 2024 14:59:33 +0100 Alexander Zeidler wrote: > * list availability and installation status of `*microcode` packages > * grep for applied "Early OS Microcode Updates" > * grep for (un)patched CPU vulnerability messages > > Signed-off-by: Alexander Zeidler > --- > PVE/Report.pm | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/PVE/Report.pm b/PVE/Report.pm > index fe497b43..18c554ec 100644 > --- a/PVE/Report.pm > +++ b/PVE/Report.pm > @@ -108,6 +108,8 @@ my $init_report_cmds = sub { > 'dmidecode -t bios -q', > 'dmidecode -t memory | grep -E > "Capacity|Devices|Size|Manu|Part" | sed -Ez "s/\n\t(M|P)[^:]*: (\S*)/\t\2/g" > | sort', > 'lscpu', > + 'apt list *microcode 2>/dev/null | column -tL', While `apt` works really well and its output hasn't changed since I started using it (wheezy or jessie) - I still want to mention it's output when piping: ``` WARNING: apt does not have a stable CLI interface. Use with caution in scripts. ``` potentially consider either using our code directly or switching to `dpkg -l`? (but as said `apt` has been pretty stable, and we simply dump the output - so probably the warning is not too relevant here) > + 'dmesg | grep -i "microcode\|vuln"', > 'lspci -nnk', > ], > }, ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux v2 0/2] update ZFS to 2.2.3 and include a fix for udev-links for partitions
changes from v1: * add a fix for #5288 after Fiona managed to reproduce it and we saw it was a known issue addressed by Fabian with a pull-request upstream * add a bit more detail to the submodule-update commit-message minimally tested in my virtual setup, additionally Fiona tested that the fix for 5288 indeed works with her reproducer Stoiko Ivanov (2): update zfs submodule to 2.2.3 and refresh patches fix #5288: cherry-pick fix for udev-partition links > 16 debian/patches/0005-Enable-zed-emails.patch | 2 +- ...hten-bounds-for-noalloc-stat-availab.patch | 4 +- ...do-not-truncate-shares-not-zfs-mount.patch | 131 -- ...rectly-handle-partition-16-and-later.patch | 52 +++ debian/patches/series | 2 +- upstream | 2 +- 6 files changed, 57 insertions(+), 136 deletions(-) delete mode 100644 debian/patches/0012-fix-mount-do-not-truncate-shares-not-zfs-mount.patch create mode 100644 debian/patches/0012-udev-correctly-handle-partition-16-and-later.patch -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux v2 2/2] fix #5288: cherry-pick fix for udev-partition links > 16
see: https://github.com/openzfs/zfs/pull/15970 https://github.com/openzfs/zfs/issues/15904 for some additional background. Signed-off-by: Stoiko Ivanov --- ...rectly-handle-partition-16-and-later.patch | 52 +++ debian/patches/series | 1 + 2 files changed, 53 insertions(+) create mode 100644 debian/patches/0012-udev-correctly-handle-partition-16-and-later.patch diff --git a/debian/patches/0012-udev-correctly-handle-partition-16-and-later.patch b/debian/patches/0012-udev-correctly-handle-partition-16-and-later.patch new file mode 100644 index ..578b74bd --- /dev/null +++ b/debian/patches/0012-udev-correctly-handle-partition-16-and-later.patch @@ -0,0 +1,52 @@ +From Mon Sep 17 00:00:00 2001 +From: =?UTF-8?q?Fabian=20Gr=C3=BCnbichler?= +Date: Wed, 6 Mar 2024 10:39:06 +0100 +Subject: [PATCH] udev: correctly handle partition #16 and later +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +If a zvol has more than 15 partitions, the minor device number exhausts +the slot count reserved for partitions next to the zvol itself. As a +result, the minor number cannot be used to determine the partition +number for the higher partition, and doing so results in wrong named +symlinks being generated by udev. + +Since the partition number is encoded in the block device name anyway, +let's just extract it from there instead. + +Fixes: #15904 + +Signed-off-by: Fabian Grünbichler +Signed-off-by: Stoiko Ivanov +--- + udev/zvol_id.c | 9 + + 1 file changed, 5 insertions(+), 4 deletions(-) + +diff --git a/udev/zvol_id.c b/udev/zvol_id.c +index 5960b9787..609349594 100644 +--- a/udev/zvol_id.c b/udev/zvol_id.c +@@ -51,7 +51,7 @@ const char *__asan_default_options(void) { + int + main(int argc, const char *const *argv) + { +- if (argc != 2) { ++ if (argc != 2 || strncmp(argv[1], "/dev/zd", 7) != 0) { + fprintf(stderr, "usage: %s /dev/zdX\n", argv[0]); + return (1); + } +@@ -72,9 +72,10 @@ main(int argc, const char *const *argv) + return (1); + } + +- unsigned int dev_part = minor(sb.st_rdev) % ZVOL_MINORS; +- if (dev_part != 0) +- sprintf(zvol_name + strlen(zvol_name), "-part%u", dev_part); ++ const char *dev_part = strrchr(dev_name, 'p'); ++ if (dev_part != NULL) { ++ sprintf(zvol_name + strlen(zvol_name), "-part%s", dev_part + 1); ++ } + + for (size_t i = 0; i < strlen(zvol_name); ++i) + if (isblank(zvol_name[i])) diff --git a/debian/patches/series b/debian/patches/series index 35f81d13..9eedf857 100644 --- a/debian/patches/series +++ b/debian/patches/series @@ -9,3 +9,4 @@ 0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch 0010-Fix-nfs_truncate_shares-without-etc-exports.d.patch 0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch +0012-udev-correctly-handle-partition-16-and-later.patch -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux v2 1/2] update zfs submodule to 2.2.3 and refresh patches
mostly support for newer kernel-versions, and fixes for the BRT bugs discovered with 2.2.0 (BRT remains disabled by default). The update contains a fix for CVE-2020-24370 in lua (which is present in ZFS for channel-programs, which we do not use) - see: https://github.com/openzfs/zfs/pull/15847 for more details. One patch from Stefan Lendl was backported and is now in the ZFS 2.2 branch. Signed-off-by: Stoiko Ivanov --- debian/patches/0005-Enable-zed-emails.patch | 2 +- ...hten-bounds-for-noalloc-stat-availab.patch | 4 +- ...do-not-truncate-shares-not-zfs-mount.patch | 131 -- debian/patches/series | 1 - upstream | 2 +- 5 files changed, 4 insertions(+), 136 deletions(-) delete mode 100644 debian/patches/0012-fix-mount-do-not-truncate-shares-not-zfs-mount.patch diff --git a/debian/patches/0005-Enable-zed-emails.patch b/debian/patches/0005-Enable-zed-emails.patch index 646d529c..af38f84e 100644 --- a/debian/patches/0005-Enable-zed-emails.patch +++ b/debian/patches/0005-Enable-zed-emails.patch @@ -13,7 +13,7 @@ Signed-off-by: Thomas Lamprecht 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmd/zed/zed.d/zed.rc b/cmd/zed/zed.d/zed.rc -index 78dc1afc7..41d5539ea 100644 +index bc269b155..e6d4b1703 100644 --- a/cmd/zed/zed.d/zed.rc +++ b/cmd/zed/zed.d/zed.rc @@ -41,7 +41,7 @@ ZED_EMAIL_ADDR="root" diff --git a/debian/patches/0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch b/debian/patches/0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch index f58c58e8..3c87b0cb 100644 --- a/debian/patches/0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch +++ b/debian/patches/0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch @@ -51,10 +51,10 @@ Signed-off-by: Thomas Lamprecht 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cmd/zpool/zpool_main.c b/cmd/zpool/zpool_main.c -index 5507f9d3f..98970abfe 100644 +index 69bf9649a..fd42ce7c1 100644 --- a/cmd/zpool/zpool_main.c +++ b/cmd/zpool/zpool_main.c -@@ -2478,7 +2478,8 @@ print_status_config(zpool_handle_t *zhp, status_cbdata_t *cb, const char *name, +@@ -2616,7 +2616,8 @@ print_status_config(zpool_handle_t *zhp, status_cbdata_t *cb, const char *name, if (vs->vs_scan_removing != 0) { (void) printf(gettext(" (removing)")); diff --git a/debian/patches/0012-fix-mount-do-not-truncate-shares-not-zfs-mount.patch b/debian/patches/0012-fix-mount-do-not-truncate-shares-not-zfs-mount.patch deleted file mode 100644 index ab01e623.. --- a/debian/patches/0012-fix-mount-do-not-truncate-shares-not-zfs-mount.patch +++ /dev/null @@ -1,131 +0,0 @@ -From Mon Sep 17 00:00:00 2001 -From: Stefan Lendl <1321542+s...@users.noreply.github.com> -Date: Fri, 12 Jan 2024 21:05:11 +0100 -Subject: [PATCH] fix(mount): do not truncate shares not zfs mount - -When running zfs share -a resetting the exports.d/zfs.exports makes -sense the get a clean state. -Truncating was also called with zfs mount which would not populate the -file again. -Add test to verify shares persist after mount -a. - -Reviewed-by: Brian Behlendorf -Signed-off-by: Stefan Lendl -Closes #15607 -Closes #15660 - cmd/zfs/zfs_main.c| 3 +- - tests/runfiles/common.run | 3 +- - tests/zfs-tests/tests/Makefile.am | 1 + - .../zfs_share/zfs_share_after_mount.ksh | 62 +++ - 4 files changed, 67 insertions(+), 2 deletions(-) - create mode 100755 tests/zfs-tests/tests/functional/cli_root/zfs_share/zfs_share_after_mount.ksh - -diff --git a/cmd/zfs/zfs_main.c b/cmd/zfs/zfs_main.c -index 9939f206a..f67f6114d 100644 a/cmd/zfs/zfs_main.c -+++ b/cmd/zfs/zfs_main.c -@@ -7234,7 +7234,8 @@ share_mount(int op, int argc, char **argv) - pthread_mutex_init(&share_mount_state.sm_lock, NULL); - - /* For a 'zfs share -a' operation start with a clean slate. */ -- zfs_truncate_shares(NULL); -+ if (op == OP_SHARE) -+ zfs_truncate_shares(NULL); - - /* -* libshare isn't mt-safe, so only do the operation in parallel -diff --git a/tests/runfiles/common.run b/tests/runfiles/common.run -index f6e5367f5..a600140ea 100644 a/tests/runfiles/common.run -+++ b/tests/runfiles/common.run -@@ -316,7 +316,8 @@ tags = ['functional', 'cli_root', 'zfs_set'] - [tests/functional/cli_root/zfs_share] - tests = ['zfs_share_001_pos', 'zfs_share_002_pos', 'zfs_share_003_pos', - 'zfs_share_004_pos', 'zfs_share_006_pos', 'zfs_share_008_neg', --'zfs_share_010_neg', 'zfs_share_011_pos', 'zfs_share_concurrent_shares'] -+'zfs_share_010_neg'
[pve-devel] [PATCH zfsonlinux] update zfs submodule to 2.2.3 and refresh patches
Signed-off-by: Stoiko Ivanov --- upstream changelog[0] is quite long this time, but nothing stood out too much during my (very minimal) testing and skimming through the changlog. Block cloning had a few bugs fixed, but remains disabled by default. ZFS-2.1.15 also got release [1], with a shorter changelog - I'll try to test this tomorrow and send a patch for the stable-7 branch. [0] https://github.com/openzfs/zfs/releases/tag/zfs-2.2.3 [1] https://github.com/openzfs/zfs/releases/tag/zfs-2.1.15 debian/patches/0005-Enable-zed-emails.patch | 2 +- ...hten-bounds-for-noalloc-stat-availab.patch | 4 +- ...do-not-truncate-shares-not-zfs-mount.patch | 131 -- debian/patches/series | 1 - upstream | 2 +- 5 files changed, 4 insertions(+), 136 deletions(-) delete mode 100644 debian/patches/0012-fix-mount-do-not-truncate-shares-not-zfs-mount.patch diff --git a/debian/patches/0005-Enable-zed-emails.patch b/debian/patches/0005-Enable-zed-emails.patch index 646d529c..af38f84e 100644 --- a/debian/patches/0005-Enable-zed-emails.patch +++ b/debian/patches/0005-Enable-zed-emails.patch @@ -13,7 +13,7 @@ Signed-off-by: Thomas Lamprecht 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmd/zed/zed.d/zed.rc b/cmd/zed/zed.d/zed.rc -index 78dc1afc7..41d5539ea 100644 +index bc269b155..e6d4b1703 100644 --- a/cmd/zed/zed.d/zed.rc +++ b/cmd/zed/zed.d/zed.rc @@ -41,7 +41,7 @@ ZED_EMAIL_ADDR="root" diff --git a/debian/patches/0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch b/debian/patches/0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch index f58c58e8..3c87b0cb 100644 --- a/debian/patches/0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch +++ b/debian/patches/0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch @@ -51,10 +51,10 @@ Signed-off-by: Thomas Lamprecht 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cmd/zpool/zpool_main.c b/cmd/zpool/zpool_main.c -index 5507f9d3f..98970abfe 100644 +index 69bf9649a..fd42ce7c1 100644 --- a/cmd/zpool/zpool_main.c +++ b/cmd/zpool/zpool_main.c -@@ -2478,7 +2478,8 @@ print_status_config(zpool_handle_t *zhp, status_cbdata_t *cb, const char *name, +@@ -2616,7 +2616,8 @@ print_status_config(zpool_handle_t *zhp, status_cbdata_t *cb, const char *name, if (vs->vs_scan_removing != 0) { (void) printf(gettext(" (removing)")); diff --git a/debian/patches/0012-fix-mount-do-not-truncate-shares-not-zfs-mount.patch b/debian/patches/0012-fix-mount-do-not-truncate-shares-not-zfs-mount.patch deleted file mode 100644 index ab01e623.. --- a/debian/patches/0012-fix-mount-do-not-truncate-shares-not-zfs-mount.patch +++ /dev/null @@ -1,131 +0,0 @@ -From Mon Sep 17 00:00:00 2001 -From: Stefan Lendl <1321542+s...@users.noreply.github.com> -Date: Fri, 12 Jan 2024 21:05:11 +0100 -Subject: [PATCH] fix(mount): do not truncate shares not zfs mount - -When running zfs share -a resetting the exports.d/zfs.exports makes -sense the get a clean state. -Truncating was also called with zfs mount which would not populate the -file again. -Add test to verify shares persist after mount -a. - -Reviewed-by: Brian Behlendorf -Signed-off-by: Stefan Lendl -Closes #15607 -Closes #15660 - cmd/zfs/zfs_main.c| 3 +- - tests/runfiles/common.run | 3 +- - tests/zfs-tests/tests/Makefile.am | 1 + - .../zfs_share/zfs_share_after_mount.ksh | 62 +++ - 4 files changed, 67 insertions(+), 2 deletions(-) - create mode 100755 tests/zfs-tests/tests/functional/cli_root/zfs_share/zfs_share_after_mount.ksh - -diff --git a/cmd/zfs/zfs_main.c b/cmd/zfs/zfs_main.c -index 9939f206a..f67f6114d 100644 a/cmd/zfs/zfs_main.c -+++ b/cmd/zfs/zfs_main.c -@@ -7234,7 +7234,8 @@ share_mount(int op, int argc, char **argv) - pthread_mutex_init(&share_mount_state.sm_lock, NULL); - - /* For a 'zfs share -a' operation start with a clean slate. */ -- zfs_truncate_shares(NULL); -+ if (op == OP_SHARE) -+ zfs_truncate_shares(NULL); - - /* -* libshare isn't mt-safe, so only do the operation in parallel -diff --git a/tests/runfiles/common.run b/tests/runfiles/common.run -index f6e5367f5..a600140ea 100644 a/tests/runfiles/common.run -+++ b/tests/runfiles/common.run -@@ -316,7 +316,8 @@ tags = ['functional', 'cli_root', 'zfs_set'] - [tests/functional/cli_root/zfs_share] - tests = ['zfs_share_001_pos', 'zfs_share_002_pos', 'zfs_share_003_pos', - 'zfs_share_004_pos', 'zfs_share_006_pos', 'zfs_share_008_neg', --'zfs_share_010_neg', 'zfs_share
[pve-devel] [PATCH dab-pve-appliances] fix #4858: install libsasl2-modules for pmg
the issue was already resolved for installations from ISO (short time after PMG 8.0 was released), but I forgot to adapt the container-template. Signed-off-by: Stoiko Ivanov --- quickly tested by building a template and checking dpkg -l debian-12-bookworm-pmg-8-64/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian-12-bookworm-pmg-8-64/Makefile b/debian-12-bookworm-pmg-8-64/Makefile index 2fb1ddb..ab590bc 100644 --- a/debian-12-bookworm-pmg-8-64/Makefile +++ b/debian-12-bookworm-pmg-8-64/Makefile @@ -9,7 +9,7 @@ all: info/init_ok ${CVD_FILES} cp systemd-presets ${BASEDIR}/etc/systemd/system-preset/00-pve-template.preset touch ${BASEDIR}/proxmox_install_mode dab install libdbi-perl perl-openssl-defaults libcgi-pm-perl proxmox-mailgateway-container gpg ifupdown2 - dab install antiword docx2txt odt2txt poppler-utils tesseract-ocr unrtf + dab install antiword docx2txt odt2txt poppler-utils tesseract-ocr unrtf libsasl2-modules rm ${BASEDIR}/proxmox_install_mode sed -i '/^deb.*\.proxmox\.com\/.*$$/d;$${/^$$/d;}' ${BASEDIR}/etc/apt/sources.list cp ${CVD_FILES} ${BASEDIR}/var/lib/clamav/ -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH dab-pve-appliances] pmg: update to 8.1
Signed-off-by: Stoiko Ivanov --- tested with the packages from our internal repository yesterday evening all looked ok. debian-12-bookworm-pmg-8-64/dab.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/debian-12-bookworm-pmg-8-64/dab.conf b/debian-12-bookworm-pmg-8-64/dab.conf index 166dbd4..fc34d9b 100644 --- a/debian-12-bookworm-pmg-8-64/dab.conf +++ b/debian-12-bookworm-pmg-8-64/dab.conf @@ -5,11 +5,11 @@ Source: http://ftp.debian.org/debian SUITE-updates main contrib Source: http://security.debian.org/debian-security SUITE-security main contrib Source: http://download.proxmox.com/debian/pmg/ SUITE pmg-no-subscription Architecture: amd64 -Name: proxmox-mailgateway-8.0-standard -Version: 8.0-1 +Name: proxmox-mailgateway-8.1-standard +Version: 8.1-1 Section: mail Maintainer: Proxmox Support Team Infopage: https://www.proxmox.com/en/proxmox-mail-gateway/overview -Description: Proxmox Mailgateway 8.0 +Description: Proxmox Mailgateway 8.1 A full featured mail proxy for spam and virus filtering, optimized for container environment. -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH pmg-docs] installation: fix codeblock rendering in zfs performance tips section
Thanks for the catch and fix! applied it! (nit: pmg-de...@list.proxmox.com is preferred for pmg-docs patches) On Wed, 28 Feb 2024 19:08:21 +0100 Christoph Heiss wrote: > That slipped through, asciidoc uses 4 not 3 dashes for that. > > Fixes: c8be3f0 ("installation: align zfs performance tip with PVE > documentation") > Signed-off-by: Christoph Heiss > --- > pmg-installation.adoc | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/pmg-installation.adoc b/pmg-installation.adoc > index 326fe5b..25d16a7 100644 > --- a/pmg-installation.adoc > +++ b/pmg-installation.adoc > @@ -284,9 +284,9 @@ ZFS can use a dedicated drive as write cache, called the > ZFS Intent Log (ZIL). > Use a fast drive (SSD) for it. It can be added after installation with the > following command: > > > + > # zpool add log > > + > > Adding the `nomodeset` Kernel Parameter > ~ ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 access-control] fix #5136: ldap: Decode non-ASCII characters in attributes
On Wed, 28 Feb 2024 16:00:48 +0100 Fiona Ebner wrote: > Am 28.02.24 um 15:41 schrieb Thomas Lamprecht: > > Am 09/01/2024 um 14:35 schrieb Filip Schauer: > >> UTF8 decode non-ASCII characters when syncing user attributes, since > >> those will be encoded later on. Without this fix the attributes were > >> encoded twice, resulting in cases such as 'ü' turning into 'ü'. > >> > >> Signed-off-by: Filip Schauer > >> --- > >> Changes since v1: > >> * Do not try to URI unescape the user attributes, since we do that later > >> in PVE::AccessControl::parse_user_config anyways. > >> > >> src/PVE/Auth/LDAP.pm | 2 +- > >> 1 file changed, 1 insertion(+), 1 deletion(-) > >> > >> diff --git a/src/PVE/Auth/LDAP.pm b/src/PVE/Auth/LDAP.pm > >> index b958f2b..06177db 100755 > >> --- a/src/PVE/Auth/LDAP.pm > >> +++ b/src/PVE/Auth/LDAP.pm > >> @@ -301,7 +301,7 @@ sub get_users { > >> > >>foreach my $attr (keys %$user_attributes) { > >>if (my $ours = $ldap_attribute_map->{$attr}) { > >> - $ret->{$username}->{$ours} = $user_attributes->{$attr}->[0]; > >> + $ret->{$username}->{$ours} = Encode::decode('utf8', > >> $user_attributes->{$attr}->[0]); > > Note: missing use Encode; at the beginning of the file. > > >>} > >>} > >> > > > > this would need a rebase, oh, and would be great if the original testers > > could reconfirm the v2 approach of doing utf-8 decoding only. > > > > Gave it a quick test and fixes issues with special characters for me. > Don't forget to also use the latest master of pve-cluster, otherwise > writing the user config will still do the wrong thing [0]! Both are > needed to fix the issue here. I'm just wondering if we are guaranteed > that the LDAP server sends UTF-8 encoded data? sadly (or luckily) not too much experience with validity of LDAP data out in the wild. Quickly searched online and went through the rfc-chain until there was not Link to "Obsoleted by" anymore (and then going through all RFC indexed there [0]: The (~18 year old) standard indicates that strings used should be UTF-8 encoded: https://datatracker.ietf.org/doc/html/rfc4511#section-4.1.2 (and pointed out the (by now probably not significant difference between unicode and ISO10646 - see [1]). However, probably with any protocol that has been around for 30+ years - guarantees are hard to come by: https://datatracker.ietf.org/doc/html/rfc4512#section-7.2 anyways - iiuc we can just skip the syncing of the attribute in this part? - if we add a warning to the log it sounds ok to me (but I only very quickly skimmed through what the code does) [0] https://datatracker.ietf.org/doc/html/rfc4510 [1] https://www.unicode.org/versions/Unicode15.0.0/appC.pdf > > [0]: > https://git.proxmox.com/?p=pve-cluster.git;a=commit;h=2e276ccd9beb2004ddd72396b2a9b72a288771d8 > > > ___ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] applied: [PATCH pmg_docs 1/1] Consistency of GB and GiB pmg
Thanks for the catch! pushed a fix-up On Tue, 6 Feb 2024 13:20:51 +0100 Thomas Lamprecht wrote: > Am 22/01/2024 um 18:58 schrieb Stoiko Ivanov: > > applied this one to pmg-docs - huge thanks! > > > > > > On Mon, Jul 10, 2023 at 03:49:49PM +0200, Noel Ullreich wrote: > >> Since the actual system-checks are done in GiB and to stay consistent > >> with the other docs, change all GB units to GiB > >> > >> Signed-off-by: Noel Ullreich > >> --- > >> pmg-planning-deployment.adoc | 8 > >> 1 file changed, 4 insertions(+), 4 deletions(-) > >> > >> diff --git a/pmg-planning-deployment.adoc b/pmg-planning-deployment.adoc > >> index 9287574..6a0083b 100644 > >> --- a/pmg-planning-deployment.adoc > >> +++ b/pmg-planning-deployment.adoc > >> @@ -110,13 +110,13 @@ Minimum System Requirements > >> > >> * CPU: 64bit (Intel EMT64 or AMD64) > >> > >> -* 2 GB RAM > >> +* 2 GiB RAM > >> > >> * Bootable CD-ROM-drive or USB boot support > >> > >> * Monitor with a minimum resolution of 1024x768 for the installation > >> > >> -* Hard disk with at least 8 GB of disk space > >> +* Hard disk with at least 8 GiB of disk space > > this is using the wrong unit though, we use the SI based GB for > disk space, as most storage vendors do, GiB is fine for memory though. > > >> > >> * Ethernet network interface card (NIC) > >> > >> @@ -127,7 +127,7 @@ Recommended System Requirements > >> * Multi-core CPU: 64bit (Intel EMT64 or AMD64), + > >> ** for use in a virtual machine, activate Intel VT/AMD-V CPU flag > >> > >> -* 4 GB RAM > >> +* 4 GiB RAM > >> > >> * Bootable CD-ROM-drive or USB boot support > >> > >> @@ -135,7 +135,7 @@ Recommended System Requirements > >> > >> * 1 Gbps Ethernet network interface card (NIC) > >> > >> -* Storage: at least 8 GB free disk space, best set up with redundancy, > >> +* Storage: at least 8 GiB free disk space, best set up with redundancy, > > same here > > >>using a hardware RAID controller with battery backed write cache > >> (``BBU'') or > >>ZFS. ZFS is not compatible with hardware RAID controllers. For best > >>performance, use enterprise-class SSDs with power loss protection. > >> -- > >> 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH pmg_docs 1/1] Consistency of GB and GiB pmg
applied this one to pmg-docs - huge thanks! On Mon, Jul 10, 2023 at 03:49:49PM +0200, Noel Ullreich wrote: > Since the actual system-checks are done in GiB and to stay consistent > with the other docs, change all GB units to GiB > > Signed-off-by: Noel Ullreich > --- > pmg-planning-deployment.adoc | 8 > 1 file changed, 4 insertions(+), 4 deletions(-) > > diff --git a/pmg-planning-deployment.adoc b/pmg-planning-deployment.adoc > index 9287574..6a0083b 100644 > --- a/pmg-planning-deployment.adoc > +++ b/pmg-planning-deployment.adoc > @@ -110,13 +110,13 @@ Minimum System Requirements > > * CPU: 64bit (Intel EMT64 or AMD64) > > -* 2 GB RAM > +* 2 GiB RAM > > * Bootable CD-ROM-drive or USB boot support > > * Monitor with a minimum resolution of 1024x768 for the installation > > -* Hard disk with at least 8 GB of disk space > +* Hard disk with at least 8 GiB of disk space > > * Ethernet network interface card (NIC) > > @@ -127,7 +127,7 @@ Recommended System Requirements > * Multi-core CPU: 64bit (Intel EMT64 or AMD64), + > ** for use in a virtual machine, activate Intel VT/AMD-V CPU flag > > -* 4 GB RAM > +* 4 GiB RAM > > * Bootable CD-ROM-drive or USB boot support > > @@ -135,7 +135,7 @@ Recommended System Requirements > > * 1 Gbps Ethernet network interface card (NIC) > > -* Storage: at least 8 GB free disk space, best set up with redundancy, > +* Storage: at least 8 GiB free disk space, best set up with redundancy, >using a hardware RAID controller with battery backed write cache (``BBU'') > or >ZFS. ZFS is not compatible with hardware RAID controllers. For best >performance, use enterprise-class SSDs with power loss protection. > -- > 2.39.2 > > > > ___ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH pve-docs; pmg-docs 0/6] Fix consistency of some words
Thanks for the patch @Noel, and thanks for the heads-up @Thomas! applied the pmg-docs parts as well (with one typo fixed up). On Fri, Nov 17, 2023 at 10:58:03AM +0100, Thomas Lamprecht wrote: > Am 03/07/2023 um 14:04 schrieb Noel Ullreich: > > In the docs for pve, pmg, and pbs, the consistency and grammatical > > correctness of some common words was not always good. > > I fixed the words: > > * USB flash drive > > * command line/command-line/commandline > > * motherboard/mainboard/main board > > * passhrough/pass-through > > > > Really, these are just some quick ones I found that (imo) were easy and > > necessary to fix, and, if possible, this should be automated somehow. > > Dietmar and I talked about this off-list and brainstormed some ideas, > > but nothing concrete so far. > > > > I also added some tables in our internal writing style guide, perhaps > > this should be made public? For those, that are interested in writing > > docs for pve/pbs/pmg? > > > > > > changes from v1: > > * the patches for proxmox-backup/docs have been applied > > * fixed spelling mistakes in the commit messages > > > > Noel Ullreich (4): > > Consistency of passthrough/pass-through > > Consistency: change mainboard to motherboard > > Consistency of USB flash drive spelling > > Fix spelling of command line/command-line > > > > certificate-management.adoc | 2 +- > > ha-manager.adoc | 10 +- > > index.adoc | 2 +- > > pct.adoc | 10 +- > > pve-admin-guide.adoc | 4 ++-- > > pve-faq.adoc | 2 +- > > pve-firewall.adoc| 2 +- > > pve-installation-media.adoc | 2 +- > > pve-intro.adoc | 6 +++--- > > pve-storage-lvmthin.adoc | 2 +- > > pve-storage-pbs.adoc | 2 +- > > pve-system-requirements.adoc | 2 +- > > pveceph.adoc | 6 +++--- > > pvecm.adoc | 4 ++-- > > pvesm.adoc | 4 ++-- > > pvesr.adoc | 6 +++--- > > pveum.adoc | 10 +- > > qm-pci-passthrough.adoc | 2 +- > > qm.adoc | 10 +- > > qm.conf.5-opts.adoc | 10 +- > > vzdump.adoc | 2 +- > > 21 files changed, 50 insertions(+), 50 deletions(-) > > > > Noel Ullreich (2): > > consistenct of USB flash drive spelling > > Fixed spelling of command line/command-line > > > > pmg-admin-guide.adoc| 4 ++-- > > pmg-administration.adoc | 2 +- > > pmg-installation-media.adoc | 2 +- > > pmg-intro.adoc | 4 ++-- > > pmg-ssl-certificate.adoc| 6 +++--- > > pmgbackup.adoc | 4 ++-- > > pmgcm.adoc | 6 +++--- > > pmgconfig.adoc | 4 ++-- > > pmgperf.adoc| 2 +- > > 9 files changed, 17 insertions(+), 17 deletions(-) > > > applied the PVE side with a few fixes to typo regressions... > > @stoiko, maybe check/apply the pmg side of this sometime (no pressure) ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux 1/2] update zfs submodule to 2.2.2 and refresh patches
the removed patches were cherry-picks, which are included in 2.2.2 Signed-off-by: Stoiko Ivanov --- ...-move-manpage-arcstat-1-to-arcstat-8.patch | 2 +- ...-guard-access-to-l2arc-MFU-MRU-stats.patch | 2 +- ...uncate_shares-without-etc-exports.d.patch} | 0 ...ol-Remove-broken-blk-mq-optimization.patch | 99 - ...evert-zvol-Temporally-disable-blk-mq.patch | 123 --- ...ten-bounds-for-noalloc-stat-availab.patch} | 0 ...und-UBSAN-errors-for-variable-arrays.patch | 72 --- ...g-between-unencrypted-and-encrypted-.patch | 44 ...Add-a-tunable-to-disable-BRT-support.patch | 201 -- ...2.1-Disable-block-cloning-by-default.patch | 42 ...heck-dnode-and-its-data-for-dirtines.patch | 97 - debian/patches/series | 11 +- upstream | 2 +- 13 files changed, 5 insertions(+), 690 deletions(-) rename debian/patches/{0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch => 0010-Fix-nfs_truncate_shares-without-etc-exports.d.patch} (100%) delete mode 100644 debian/patches/0010-zvol-Remove-broken-blk-mq-optimization.patch delete mode 100644 debian/patches/0011-Revert-zvol-Temporally-disable-blk-mq.patch rename debian/patches/{0014-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch => 0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch} (100%) delete mode 100644 debian/patches/0013-Workaround-UBSAN-errors-for-variable-arrays.patch delete mode 100644 debian/patches/0015-Fix-block-cloning-between-unencrypted-and-encrypted-.patch delete mode 100644 debian/patches/0016-Add-a-tunable-to-disable-BRT-support.patch delete mode 100644 debian/patches/0017-zfs-2.2.1-Disable-block-cloning-by-default.patch delete mode 100644 debian/patches/0018-dnode_is_dirty-check-dnode-and-its-data-for-dirtines.patch diff --git a/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch b/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch index b21a301f..c11c1ae8 100644 --- a/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch +++ b/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch @@ -15,7 +15,7 @@ Signed-off-by: Thomas Lamprecht rename man/{man1/arcstat.1 => man8/arcstat.8} (99%) diff --git a/man/Makefile.am b/man/Makefile.am -index 36c1aede1..94fd96e58 100644 +index 45156571e..3713e9371 100644 --- a/man/Makefile.am +++ b/man/Makefile.am @@ -2,7 +2,6 @@ dist_noinst_man_MANS = \ diff --git a/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch b/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch index fde0529a..f8cb3539 100644 --- a/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch +++ b/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch @@ -27,7 +27,7 @@ Signed-off-by: Thomas Lamprecht 2 files changed, 21 insertions(+), 21 deletions(-) diff --git a/cmd/arc_summary b/cmd/arc_summary -index 426e02070..9de198150 100755 +index 9c69ec4f8..edf94ea2a 100755 --- a/cmd/arc_summary +++ b/cmd/arc_summary @@ -655,13 +655,13 @@ def section_arc(kstats_dict): diff --git a/debian/patches/0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch b/debian/patches/0010-Fix-nfs_truncate_shares-without-etc-exports.d.patch similarity index 100% rename from debian/patches/0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch rename to debian/patches/0010-Fix-nfs_truncate_shares-without-etc-exports.d.patch diff --git a/debian/patches/0010-zvol-Remove-broken-blk-mq-optimization.patch b/debian/patches/0010-zvol-Remove-broken-blk-mq-optimization.patch deleted file mode 100644 index 178e68ee.. --- a/debian/patches/0010-zvol-Remove-broken-blk-mq-optimization.patch +++ /dev/null @@ -1,99 +0,0 @@ -From Mon Sep 17 00:00:00 2001 -From: Tony Hutter -Date: Mon, 23 Oct 2023 14:45:06 -0700 -Subject: [PATCH] zvol: Remove broken blk-mq optimization - -This fix removes a dubious optimization in zfs_uiomove_bvec_rq() -that saved the iterator contents of a rq_for_each_segment(). This -optimization allowed restoring the "saved state" from a previous -rq_for_each_segment() call on the same uio so that you wouldn't -need to iterate though each bvec on every zfs_uiomove_bvec_rq() call. -However, if the kernel is manipulating the requests/bios/bvecs under -the covers between zfs_uiomove_bvec_rq() calls, then it could result -in corruption from using the "saved state". This optimization -results in an unbootable system after installing an OS on a zvol -with blk-mq enabled. - -Reviewed-by: Brian Behlendorf -Signed-off-by: Tony Hutter -Closes #15351 -(cherry picked from commit 7c9b6fed16ed5034fd1cdfdaedfad93dc97b1557) -Signed-off-by: Stoiko Ivanov - include/os/linux/spl/sys/uio.h | 8 - module/os/linux/zfs/zfs_uio.c | 29
[pve-devel] [PATCH zfsonlinux 0/2] update to 2.2.2
Since ZFS 2.2.2 was released last week - and for now there were not regressions noticed with it - updating to 2.2.2 should help everyone to see that our version is not affected by the `dnode_is_dirty` issue anymore. The second patch was already present in my tests for 2.2.1: https://lists.proxmox.com/pipermail/pve-devel/2023-November/060751.html minimally tested on 2 VMs, with a few containers and storage-replication configured. Stoiko Ivanov (2): update zfs submodule to 2.2.2 and refresh patches d/zfsutils-linux.install: add zfs_prepare_disk and manpage ...-move-manpage-arcstat-1-to-arcstat-8.patch | 2 +- ...-guard-access-to-l2arc-MFU-MRU-stats.patch | 2 +- ...uncate_shares-without-etc-exports.d.patch} | 0 ...ol-Remove-broken-blk-mq-optimization.patch | 99 - ...evert-zvol-Temporally-disable-blk-mq.patch | 123 --- ...ten-bounds-for-noalloc-stat-availab.patch} | 0 ...und-UBSAN-errors-for-variable-arrays.patch | 72 --- ...g-between-unencrypted-and-encrypted-.patch | 44 ...Add-a-tunable-to-disable-BRT-support.patch | 201 -- ...2.1-Disable-block-cloning-by-default.patch | 42 ...heck-dnode-and-its-data-for-dirtines.patch | 97 - debian/patches/series | 11 +- debian/zfsutils-linux.install | 2 + upstream | 2 +- 14 files changed, 7 insertions(+), 690 deletions(-) rename debian/patches/{0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch => 0010-Fix-nfs_truncate_shares-without-etc-exports.d.patch} (100%) delete mode 100644 debian/patches/0010-zvol-Remove-broken-blk-mq-optimization.patch delete mode 100644 debian/patches/0011-Revert-zvol-Temporally-disable-blk-mq.patch rename debian/patches/{0014-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch => 0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch} (100%) delete mode 100644 debian/patches/0013-Workaround-UBSAN-errors-for-variable-arrays.patch delete mode 100644 debian/patches/0015-Fix-block-cloning-between-unencrypted-and-encrypted-.patch delete mode 100644 debian/patches/0016-Add-a-tunable-to-disable-BRT-support.patch delete mode 100644 debian/patches/0017-zfs-2.2.1-Disable-block-cloning-by-default.patch delete mode 100644 debian/patches/0018-dnode_is_dirty-check-dnode-and-its-data-for-dirtines.patch -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux 2/2] d/zfsutils-linux.install: add zfs_prepare_disk and manpage
Signed-off-by: Stoiko Ivanov --- debian/zfsutils-linux.install | 2 ++ 1 file changed, 2 insertions(+) diff --git a/debian/zfsutils-linux.install b/debian/zfsutils-linux.install index e788e27c..2fd7aefe 100644 --- a/debian/zfsutils-linux.install +++ b/debian/zfsutils-linux.install @@ -35,6 +35,7 @@ sbin/zstreamdump usr/bin/zvol_wait usr/bin/zilstat usr/lib/modules-load.d/ lib/ +usr/lib/zfs-linux/zfs_prepare_disk usr/lib/zfs-linux/zpool.d/ usr/lib/zfs-linux/zpool_influxdb usr/sbin/arc_summary @@ -72,6 +73,7 @@ usr/share/man/man8/zfs-list.8 usr/share/man/man8/zfs-load-key.8 usr/share/man/man8/zfs-mount-generator.8 usr/share/man/man8/zfs-mount.8 +usr/share/man/man8/zfs_prepare_disk.8 usr/share/man/man8/zfs-program.8 usr/share/man/man8/zfs-project.8 usr/share/man/man8/zfs-projectspace.8 -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH docs] fix #5089: sdn: fix s/quest/guest/ typo
additionally did a `git grep '\bquest'` for good measure Signed-off-by: Stoiko Ivanov --- pvesdn.adoc | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/pvesdn.adoc b/pvesdn.adoc index fdc6ea7..53293b1 100644 --- a/pvesdn.adoc +++ b/pvesdn.adoc @@ -381,7 +381,7 @@ Zone:: The associated zone for this VNet Tag:: The unique VLAN or VXLAN ID VLAN Aware:: Enables vlan-aware option on the interface, enabling configuration - in the quest. + in the guest. [[pvesdn_config_subnet]] @@ -715,17 +715,17 @@ details to enhance comprehension of the available configuration options. Simple Zone Example ~~~ -Simple zone networks create an isolated network for quests on a single host to +Simple zone networks create an isolated network for guests on a single host to connect to each other. -TIP: connection between quests are possible if all quests reside on a same host +TIP: connection between guests are possible if all guests reside on a same host but cannot be reached on other nodes. * Create a simple zone named `simple`. * Add a VNet names `vnet1`. * Create a Subnet with a gateway and the SNAT option enabled. * This creates a network bridge `vnet1` on the node. Assign this bridge to the - quests that shall join the network and configure an IP address. + guests that shall join the network and configure an IP address. The network interface configuration in two VMs may look like this which allows them to communicate via the 10.0.1.0/24 network. @@ -747,7 +747,7 @@ iface ens19 inet static Source NAT Example ~~ -If you want to allow outgoing connections for quests in the simple network zone +If you want to allow outgoing connections for guests in the simple network zone the simple zone offers a Source NAT (SNAT) option. Starting from the configuration xref:pvesdn_setup_example_simple[above], Add a @@ -759,10 +759,10 @@ Gateway: 172.16.0.1 SNAT: checked -In the quests configure the static IP address inside the subnet's IP range. +In the guests configure the static IP address inside the subnet's IP range. The node itself will join this network with the Gateway IP '172.16.0.1' and -function as the NAT gateway for quests within the subnet range. +function as the NAT gateway for guests within the subnet range. [[pvesdn_setup_example_vlan]] -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH] update kernel to 6.5.11 and ZFS to 2.2.1, refresh patches
* for the kernel-patch this includes a rename from 0003+0004 to 0001+0002 * for ZFS there was a change in upstream's autotools-setup - I referenced the commit in the actual patch-file minimally tested with a VM with a zfs-pool and an ext4 disk * restore of a directory on ext4 containing 160MB of debian packages as tar.zstd * restore of a small folder (/root in a debian container) on zfs both worked restoring files from a Windows guest - worked, however there is an independent issue with tpmstate not being found: `given image 'drive-tpmstate0-backup.img.fidx' not found (400)` directories with 10 million files also still cause the restore-shim to run into OOM (but this is independent of the restore-image) Signed-off-by: Stoiko Ivanov --- ...ch => 0001-vsock-reduce-packet-size.patch} | 9 +++-- ...estore-halt-machine-on-kernel-panic.patch} | 9 +++-- .../0001-remove-reference-to-libudev.patch| 19 +-- src/submodules/ubuntu-kernel | 2 +- src/submodules/zfsonlinux | 2 +- 5 files changed, 17 insertions(+), 24 deletions(-) rename src/patches/kernel/{0003-vsock-reduce-packet-size.patch => 0001-vsock-reduce-packet-size.patch} (86%) rename src/patches/kernel/{0004-PBS-restore-halt-machine-on-kernel-panic.patch => 0002-PBS-restore-halt-machine-on-kernel-panic.patch} (83%) diff --git a/src/patches/kernel/0003-vsock-reduce-packet-size.patch b/src/patches/kernel/0001-vsock-reduce-packet-size.patch similarity index 86% rename from src/patches/kernel/0003-vsock-reduce-packet-size.patch rename to src/patches/kernel/0001-vsock-reduce-packet-size.patch index 378da53..75b0e92 100644 --- a/src/patches/kernel/0003-vsock-reduce-packet-size.patch +++ b/src/patches/kernel/0001-vsock-reduce-packet-size.patch @@ -1,4 +1,4 @@ -From a437d428733881f408b5d42eb75812600083cb75 Mon Sep 17 00:00:00 2001 +From Mon Sep 17 00:00:00 2001 From: Stefan Reiter Date: Mon, 26 Apr 2021 14:08:36 +0200 Subject: [PATCH] vsock: reduce packet size @@ -19,10 +19,10 @@ Signed-off-by: Stefan Reiter 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h -index dc636b727179..18c09ff72929 100644 +index c58453699ee9..62a609444e12 100644 --- a/include/linux/virtio_vsock.h +++ b/include/linux/virtio_vsock.h -@@ -9,7 +9,7 @@ +@@ -112,7 +112,7 @@ static inline size_t virtio_vsock_skb_len(struct sk_buff *skb) #define VIRTIO_VSOCK_DEFAULT_RX_BUF_SIZE (1024 * 4) #define VIRTIO_VSOCK_MAX_BUF_SIZE 0xUL @@ -31,6 +31,3 @@ index dc636b727179..18c09ff72929 100644 enum { VSOCK_VQ_RX = 0, /* for host to guest data */ --- -2.20.1 - diff --git a/src/patches/kernel/0004-PBS-restore-halt-machine-on-kernel-panic.patch b/src/patches/kernel/0002-PBS-restore-halt-machine-on-kernel-panic.patch similarity index 83% rename from src/patches/kernel/0004-PBS-restore-halt-machine-on-kernel-panic.patch rename to src/patches/kernel/0002-PBS-restore-halt-machine-on-kernel-panic.patch index d79833f..8c2cabd 100644 --- a/src/patches/kernel/0004-PBS-restore-halt-machine-on-kernel-panic.patch +++ b/src/patches/kernel/0002-PBS-restore-halt-machine-on-kernel-panic.patch @@ -1,4 +1,4 @@ -From 7222e7424aab957f63b98853ea9fb30eec83666e Mon Sep 17 00:00:00 2001 +From Mon Sep 17 00:00:00 2001 From: Stefan Reiter Date: Mon, 3 May 2021 11:13:10 +0200 Subject: [PATCH] PBS-restore: halt machine on kernel panic @@ -14,10 +14,10 @@ Signed-off-by: Stefan Reiter 1 file changed, 3 insertions(+) diff --git a/kernel/panic.c b/kernel/panic.c -index 332736a72a58..56339ae5165c 100644 +index ea1c5fcb2d19..c317ca992a26 100644 --- a/kernel/panic.c +++ b/kernel/panic.c -@@ -325,6 +325,9 @@ void panic(const char *fmt, ...) +@@ -417,6 +417,9 @@ void panic(const char *fmt, ...) } } if (panic_timeout != 0) { @@ -27,6 +27,3 @@ index 332736a72a58..56339ae5165c 100644 /* * This will not be a clean reboot, with everything * shutting down. But if there is a chance of --- -2.20.1 - diff --git a/src/patches/zfs/0001-remove-reference-to-libudev.patch b/src/patches/zfs/0001-remove-reference-to-libudev.patch index 467d9b5..8fe9b31 100644 --- a/src/patches/zfs/0001-remove-reference-to-libudev.patch +++ b/src/patches/zfs/0001-remove-reference-to-libudev.patch @@ -6,6 +6,8 @@ Subject: [PATCH] remove reference to libudev since there's no command line flag I can see... Signed-off-by: Stefan Reiter +[ SI adapt to aebd94cc8541e0ec3b1de57edbd57c4280213089 ] +Signed-off-by: Stoiko Ivanov --- config/user-libudev.m4 | 17 - config/user.m4 | 1 - @@ -36,17 +38,14 @@ index 8c3c1d7e0..0 - ]) -]) diff --git a/config/user.m4 b/config/user.m4 -index c22067551..1b6d3a24e 100644 +inde
[pve-devel] [PATCH zfsonlinux 1/2] update submodule and patches for 2.2.1
Signed-off-by: Stoiko Ivanov --- ...-move-manpage-arcstat-1-to-arcstat-8.patch | 2 +- ...-guard-access-to-l2arc-MFU-MRU-stats.patch | 2 +- ...uncate_shares-without-etc-exports.d.patch} | 0 ...ol-Remove-broken-blk-mq-optimization.patch | 99 - ...evert-zvol-Temporally-disable-blk-mq.patch | 123 --- ...ten-bounds-for-noalloc-stat-availab.patch} | 0 ...und-UBSAN-errors-for-variable-arrays.patch | 72 --- ...g-between-unencrypted-and-encrypted-.patch | 44 ...Add-a-tunable-to-disable-BRT-support.patch | 201 -- ...2.1-Disable-block-cloning-by-default.patch | 42 debian/patches/series | 10 +- upstream | 2 +- 12 files changed, 5 insertions(+), 592 deletions(-) rename debian/patches/{0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch => 0010-Fix-nfs_truncate_shares-without-etc-exports.d.patch} (100%) delete mode 100644 debian/patches/0010-zvol-Remove-broken-blk-mq-optimization.patch delete mode 100644 debian/patches/0011-Revert-zvol-Temporally-disable-blk-mq.patch rename debian/patches/{0014-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch => 0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch} (100%) delete mode 100644 debian/patches/0013-Workaround-UBSAN-errors-for-variable-arrays.patch delete mode 100644 debian/patches/0015-Fix-block-cloning-between-unencrypted-and-encrypted-.patch delete mode 100644 debian/patches/0016-Add-a-tunable-to-disable-BRT-support.patch delete mode 100644 debian/patches/0017-zfs-2.2.1-Disable-block-cloning-by-default.patch diff --git a/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch b/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch index b21a301f..c11c1ae8 100644 --- a/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch +++ b/debian/patches/0008-Patch-move-manpage-arcstat-1-to-arcstat-8.patch @@ -15,7 +15,7 @@ Signed-off-by: Thomas Lamprecht rename man/{man1/arcstat.1 => man8/arcstat.8} (99%) diff --git a/man/Makefile.am b/man/Makefile.am -index 36c1aede1..94fd96e58 100644 +index 45156571e..3713e9371 100644 --- a/man/Makefile.am +++ b/man/Makefile.am @@ -2,7 +2,6 @@ dist_noinst_man_MANS = \ diff --git a/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch b/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch index fde0529a..f8cb3539 100644 --- a/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch +++ b/debian/patches/0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch @@ -27,7 +27,7 @@ Signed-off-by: Thomas Lamprecht 2 files changed, 21 insertions(+), 21 deletions(-) diff --git a/cmd/arc_summary b/cmd/arc_summary -index 426e02070..9de198150 100755 +index 9c69ec4f8..edf94ea2a 100755 --- a/cmd/arc_summary +++ b/cmd/arc_summary @@ -655,13 +655,13 @@ def section_arc(kstats_dict): diff --git a/debian/patches/0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch b/debian/patches/0010-Fix-nfs_truncate_shares-without-etc-exports.d.patch similarity index 100% rename from debian/patches/0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch rename to debian/patches/0010-Fix-nfs_truncate_shares-without-etc-exports.d.patch diff --git a/debian/patches/0010-zvol-Remove-broken-blk-mq-optimization.patch b/debian/patches/0010-zvol-Remove-broken-blk-mq-optimization.patch deleted file mode 100644 index 178e68ee.. --- a/debian/patches/0010-zvol-Remove-broken-blk-mq-optimization.patch +++ /dev/null @@ -1,99 +0,0 @@ -From Mon Sep 17 00:00:00 2001 -From: Tony Hutter -Date: Mon, 23 Oct 2023 14:45:06 -0700 -Subject: [PATCH] zvol: Remove broken blk-mq optimization - -This fix removes a dubious optimization in zfs_uiomove_bvec_rq() -that saved the iterator contents of a rq_for_each_segment(). This -optimization allowed restoring the "saved state" from a previous -rq_for_each_segment() call on the same uio so that you wouldn't -need to iterate though each bvec on every zfs_uiomove_bvec_rq() call. -However, if the kernel is manipulating the requests/bios/bvecs under -the covers between zfs_uiomove_bvec_rq() calls, then it could result -in corruption from using the "saved state". This optimization -results in an unbootable system after installing an OS on a zvol -with blk-mq enabled. - -Reviewed-by: Brian Behlendorf -Signed-off-by: Tony Hutter -Closes #15351 -(cherry picked from commit 7c9b6fed16ed5034fd1cdfdaedfad93dc97b1557) -Signed-off-by: Stoiko Ivanov - include/os/linux/spl/sys/uio.h | 8 - module/os/linux/zfs/zfs_uio.c | 29 - - 2 files changed, 37 deletions(-) - -diff --git a/include/os/linux/spl/sys/uio.h b/include/os/linux/spl/sys/uio.h -index cce097e16..a4b64 100644 a/include/os/linux/spl/sys/uio.h -++
[pve-devel] [PATCH zfsonlinux 0/2] update to 2.2.1
zfs-2.2.1 was released yesterday. We've already picked all critical patches (potentially involving dataloss) and a few cosmetic ones. quickly skimming through the changelog adds - nothing stands out that sounds particularly scary w.r.t. regressions, but also nothing that we missed until now. Some performance improvments are still missin in our current version. minimially tested on 2 VMs with a few containers with replication. Stoiko Ivanov (2): update submodule and patches for 2.2.1 d/zfsutils-linux.install: add zfs_prepare_disk and manpage ...-move-manpage-arcstat-1-to-arcstat-8.patch | 2 +- ...-guard-access-to-l2arc-MFU-MRU-stats.patch | 2 +- ...uncate_shares-without-etc-exports.d.patch} | 0 ...ol-Remove-broken-blk-mq-optimization.patch | 99 - ...evert-zvol-Temporally-disable-blk-mq.patch | 123 --- ...ten-bounds-for-noalloc-stat-availab.patch} | 0 ...und-UBSAN-errors-for-variable-arrays.patch | 72 --- ...g-between-unencrypted-and-encrypted-.patch | 44 ...Add-a-tunable-to-disable-BRT-support.patch | 201 -- ...2.1-Disable-block-cloning-by-default.patch | 42 debian/patches/series | 10 +- debian/zfsutils-linux.install | 2 + upstream | 2 +- 13 files changed, 7 insertions(+), 592 deletions(-) rename debian/patches/{0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch => 0010-Fix-nfs_truncate_shares-without-etc-exports.d.patch} (100%) delete mode 100644 debian/patches/0010-zvol-Remove-broken-blk-mq-optimization.patch delete mode 100644 debian/patches/0011-Revert-zvol-Temporally-disable-blk-mq.patch rename debian/patches/{0014-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch => 0011-zpool-status-tighten-bounds-for-noalloc-stat-availab.patch} (100%) delete mode 100644 debian/patches/0013-Workaround-UBSAN-errors-for-variable-arrays.patch delete mode 100644 debian/patches/0015-Fix-block-cloning-between-unencrypted-and-encrypted-.patch delete mode 100644 debian/patches/0016-Add-a-tunable-to-disable-BRT-support.patch delete mode 100644 debian/patches/0017-zfs-2.2.1-Disable-block-cloning-by-default.patch -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux 2/2] d/zfsutils-linux.install: add zfs_prepare_disk and manpage
Signed-off-by: Stoiko Ivanov --- debian/zfsutils-linux.install | 2 ++ 1 file changed, 2 insertions(+) diff --git a/debian/zfsutils-linux.install b/debian/zfsutils-linux.install index e788e27c..2fd7aefe 100644 --- a/debian/zfsutils-linux.install +++ b/debian/zfsutils-linux.install @@ -35,6 +35,7 @@ sbin/zstreamdump usr/bin/zvol_wait usr/bin/zilstat usr/lib/modules-load.d/ lib/ +usr/lib/zfs-linux/zfs_prepare_disk usr/lib/zfs-linux/zpool.d/ usr/lib/zfs-linux/zpool_influxdb usr/sbin/arc_summary @@ -72,6 +73,7 @@ usr/share/man/man8/zfs-list.8 usr/share/man/man8/zfs-load-key.8 usr/share/man/man8/zfs-mount-generator.8 usr/share/man/man8/zfs-mount.8 +usr/share/man/man8/zfs_prepare_disk.8 usr/share/man/man8/zfs-program.8 usr/share/man/man8/zfs-project.8 usr/share/man/man8/zfs-projectspace.8 -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH kernel-helper] add dedicated removable installation
seems adding `--removable` makes grub install ignore most other information - e.g. the bootloader-id (guessed based on [0]). add dedicated call with out `--removable` in addition Seems that's the reason why our installer also 'rolls its own removable' [1] minimally tested with an ISO with this installed and an install with ZFS on / (RAID1). [0] https://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-install.c#n1142 [1] https://git.proxmox.com/?p=pve-installer.git;a=blob;f=Proxmox/Install.pm;h=8825699416475b1934527531db4fbb40e9473ce8;hb=HEAD#l608 Signed-off-by: Stoiko Ivanov --- src/bin/proxmox-boot-tool | 8 1 file changed, 8 insertions(+) diff --git a/src/bin/proxmox-boot-tool b/src/bin/proxmox-boot-tool index e70850a..2903ff3 100755 --- a/src/bin/proxmox-boot-tool +++ b/src/bin/proxmox-boot-tool @@ -161,6 +161,14 @@ init_bootloader() { if [ -d /sys/firmware/efi ]; then if [ -n "$mode" ] && [ "$mode" = "grub" ]; then echo "Installing grub x86_64 target.." + grub-install.real \ + --boot-directory "${esp_mp}" \ + --target x86_64-efi \ + --no-floppy \ + --efi-directory "${esp_mp}" \ + --bootloader-id 'proxmox' \ + "/dev/$PKNAME" + echo "Installing grub x86_64 target (removable).." grub-install.real \ --boot-directory "${esp_mp}" \ --target x86_64-efi \ -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH kernel-helper] grub-install: provide --removable to grub-install
noticed while installing with secure-boot enabled on ZFS RAID1: The system has no entry to boot from in the efi-vars and the entry for the first disk simply does not boot (I assume OVMF tries the default bootx64.efi. Since `proxmox-boot-tool init` should only be done for ESPs, which are dedicated to proxmox products I don't think that this will cause many regressions For comparison - our installer has done the manual equivalent of the --removable option for installs on ext4 for quite a while. minimally tested on a VM during install. Reported-by: Thomas Lamprecht Signed-off-by: Stoiko Ivanov --- ftr: In Mira's tests a RAID1 install on ZFS booted fine as long as both disks were added to the boot-options (which I cannot explain) src/bin/proxmox-boot-tool | 1 + 1 file changed, 1 insertion(+) diff --git a/src/bin/proxmox-boot-tool b/src/bin/proxmox-boot-tool index befa2fb..e70850a 100755 --- a/src/bin/proxmox-boot-tool +++ b/src/bin/proxmox-boot-tool @@ -166,6 +166,7 @@ init_bootloader() { --target x86_64-efi \ --no-floppy \ --efi-directory "${esp_mp}" \ + --removable \ --bootloader-id 'proxmox' \ "/dev/$PKNAME" else -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v2] zfs: create dataset var-lib-vz for /var/lib/vz
Creating rpool/var/lib/vz and all intermediate datasets causes a service-failure of `var.mount` upon shutdown. creating the dataset for /var/lib/vz directly at the rpool and setting its mountpoint property seems the most robust way to address this. The alternative approach of setting `canmount=off` on the `var` dataset seems a bit dangerous (users setting a zfs property and suddenly hiding their /var contents). The only small downside to this approach is that the setting of the mountpoint happens quite a bit after extracting the data - but this would probably be better addressed with a refactoring of the lowlevel-installer code (setting the zfs-pool up under /target and getting rid of a few special cases) Fixes: dd19d40ceac179ba18652f1d6c3e4c23f246af00 Suggested-by: Fabian Grünbichler Signed-off-by: Stoiko Ivanov --- huge Thanks to Fabian - for the feedback and patiently walking me through the potential pitfalls! Proxmox/Install.pm | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm index 811db8c..1ed38c1 100644 --- a/Proxmox/Install.pm +++ b/Proxmox/Install.pm @@ -187,8 +187,8 @@ sub zfs_create_rpool { if ($iso_env->{product} eq 'pve') { syscmd("zfs create $pool_name/data") == 0 || die "unable to create zfs $pool_name/data volume\n"; - syscmd("zfs create -p $pool_name/ROOT/$root_volume_name/var/lib/vz") == 0 || - die "unable to create zfs $pool_name/ROOT/$root_volume_name/var/lib/vz volume\n"; + syscmd("zfs create -o mountpoint=/$pool_name/ROOT/$root_volume_name/var/lib/vz $pool_name/var-lib-vz") == 0 || + die "unable to create zfs $pool_name/var-lib-vz volume\n"; } # default to `relatime` on, fast enough for the installer and production @@ -1335,6 +1335,11 @@ _EOD syscmd("zfs set mountpoint=/ $zfs_pool_name/ROOT/$zfs_root_volume_name") == 0 || die "zfs set mountpoint failed\n"; + if ($iso_env->{product} eq 'pve') { + syscmd("zfs set mountpoint=/var/lib/vz $zfs_pool_name/var-lib-vz") == 0 || + die "zfs set mountpoint for var-lib-vz failed\n"; + } + syscmd("zpool set bootfs=$zfs_pool_name/ROOT/$zfs_root_volume_name $zfs_pool_name") == 0 || die "zpool set bootfs failed\n"; syscmd("zpool export $zfs_pool_name"); -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 1/2] zfs: set canmount=off on /var/lib
as explained in zfsprops(4) setting canmount to off is similar to setting mountpoint to none - except that you can still use the dataset for storing properties to be inherited to children (and we want /var/lib/vz to have a mountpoint set) Follows recommendations from upstreams ZFS on / guide: https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Bookworm%20Root%20on%20ZFS.html Fixes: dd19d40ceac179ba18652f1d6c3e4c23f246af00 Signed-off-by: Stoiko Ivanov --- Proxmox/Install.pm | 4 1 file changed, 4 insertions(+) diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm index 1a4ee93..fd9bf84 100644 --- a/Proxmox/Install.pm +++ b/Proxmox/Install.pm @@ -189,6 +189,10 @@ sub zfs_create_rpool { syscmd("zfs create $pool_name/data") == 0 || die "unable to create zfs $pool_name/data volume\n"; syscmd("zfs create -p $pool_name/ROOT/$root_volume_name/var/lib/vz") == 0 || die "unable to create zfs $pool_name/ROOT/$root_volume_name/var/lib/vz volume\n"; + syscmd("zfs set canmount=off $pool_name/ROOT/$root_volume_name/var/lib") == 0 || + die "unable to set canmount property on $pool_name/ROOT/$root_volume_name/var/lib\n"; + syscmd("zfs set canmount=off $pool_name/ROOT/$root_volume_name/var") == 0 || + die "unable to set canmount property on $pool_name/ROOT/$root_volume_name/var\n"; } # default to `relatime` on, fast enough for the installer and production -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 0/2] fix 2 small issues with recent installer patches
both issues are unrelated, and can be applied individually first is that with the recent creation of /var/lib/vz as dedicated dataset, systems yielded a failed service for var.mount upon shutdown. The fix simply sets the canmount property for the intermediate dataset var and var/lib to off - following upstream's guide the second one fixes that grub did not show up on the serial console after installation. it follows the recommendations from the archwiki on the topic: https://wiki.archlinux.org/title/working_with_the_serial_console#GRUB minimally tested with a VM here. Stoiko Ivanov (2): zfs: set canmount=off on /var/lib serial installer: add serial config for grub to target system Proxmox/Install.pm | 12 +++- 1 file changed, 11 insertions(+), 1 deletion(-) -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 2/2] serial installer: add serial config for grub to target system
Matching if a serial will be needed for grub is based on the target commandline - the speed is also read from there. The unit is based on the ttyS device - although I'd assume that this might not always match up. Signed-off-by: Stoiko Ivanov --- Proxmox/Install.pm | 8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm index fd9bf84..4293f96 100644 --- a/Proxmox/Install.pm +++ b/Proxmox/Install.pm @@ -1162,7 +1162,13 @@ _EOD update_progress(0.8, 0.95, 1, "make system bootable"); my $target_cmdline=''; if ($target_cmdline = Proxmox::Install::Config::get_target_cmdline()) { - my $target_cmdline_snippet = "GRUB_CMDLINE_LINUX=\"\$GRUB_CMDLINE_LINUX $target_cmdline\""; + my $target_cmdline_snippet = ''; + if ($target_cmdline =~ /console=ttyS(\d+),(\d+)/) { + $target_cmdline_snippet .= "GRUB_TERMINAL_INPUT=\"console serial\"\n"; + $target_cmdline_snippet .= "GRUB_TERMINAL_OUTPUT=\"gfxterm serial\"\n"; + $target_cmdline_snippet .= "GRUB_SERIAL_COMMAND=\"serial --unit=$1 --speed=$2\"\n"; + } + $target_cmdline_snippet .= "GRUB_CMDLINE_LINUX=\"\$GRUB_CMDLINE_LINUX $target_cmdline\""; file_write_all("$targetdir/etc/default/grub.d/installer.cfg", $target_cmdline_snippet); } -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer] tui: fix interface sort order
currently when multiple nics are present in a system the TUI sometimes selects the wrong interface (not the one that has the default gateway/dhcp lease) I assume this is due to HashMap's values yielding an iterator in arbitrary order Signed-off-by: Stoiko Ivanov --- sadly a bit difficult to test due to the randomnes - but at least the 3 tests on a VM were consistent. proxmox-tui-installer/src/main.rs | 13 ++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/proxmox-tui-installer/src/main.rs b/proxmox-tui-installer/src/main.rs index 4c14482..85b6811 100644 --- a/proxmox-tui-installer/src/main.rs +++ b/proxmox-tui-installer/src/main.rs @@ -488,16 +488,23 @@ fn network_dialog(siv: &mut Cursive) -> InstallerView { let state = siv.user_data::().unwrap(); let options = &state.options.network; let ifaces = state.runtime_info.network.interfaces.values(); -let ifnames = ifaces +let ifname_entries = ifaces .clone() .map(|iface| (iface.render(), iface.name.clone())); -let mut ifaces_selection = SelectView::new().popup().with_all(ifnames.clone()); +let mut ifaces_selection = SelectView::new().popup().with_all(ifname_entries.clone()); + +let mut ifnames = ifaces +.clone() +.map(|iface| iface.name.clone()) +.collect::>(); +ifnames.sort(); ifaces_selection.sort(); ifaces_selection.set_selection( ifnames +.iter() .clone() -.position(|iface| iface.1 == options.ifname) +.position(|iface| *iface == options.ifname) .unwrap_or(ifaces.len() - 1), ); -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux] pick bug-fixes staged for 2.2.1
ZFS 2.2.1 is currently being prepared, but the 3 patches added here seem quite relevant, as the might cause dataloss/panics on setups which run `zpool upgrade`. See upstreams discussion for 2.2.1: https://github.com/openzfs/zfs/pull/15498/ and the most critical issue: https://github.com/openzfs/zfs/pull/15529 finally: https://github.com/openzfs/zfs/commit/459c99ff2339a4a514abcf2255f9b3e5324ef09e should not hurt either the change to the UBSAN patch (0013) is unrelate, cosmetic only and happened by running export-patchqueue. Signed-off-by: Stoiko Ivanov --- minimally tested by building our current kernel with this and booting it in 2 VMs - the tunable (module parameter) is present and set to 0 ...und-UBSAN-errors-for-variable-arrays.patch | 5 +- ...g-between-unencrypted-and-encrypted-.patch | 44 ...Add-a-tunable-to-disable-BRT-support.patch | 201 ++ ...2.1-Disable-block-cloning-by-default.patch | 42 debian/patches/series | 3 + 5 files changed, 291 insertions(+), 4 deletions(-) create mode 100644 debian/patches/0015-Fix-block-cloning-between-unencrypted-and-encrypted-.patch create mode 100644 debian/patches/0016-Add-a-tunable-to-disable-BRT-support.patch create mode 100644 debian/patches/0017-zfs-2.2.1-Disable-block-cloning-by-default.patch diff --git a/debian/patches/0013-Workaround-UBSAN-errors-for-variable-arrays.patch b/debian/patches/0013-Workaround-UBSAN-errors-for-variable-arrays.patch index 02815311..0b98c42a 100644 --- a/debian/patches/0013-Workaround-UBSAN-errors-for-variable-arrays.patch +++ b/debian/patches/0013-Workaround-UBSAN-errors-for-variable-arrays.patch @@ -1,4 +1,4 @@ -From 28be24aefc13b11e4c96e172cf2685994e03150d Mon Sep 17 00:00:00 2001 +From Mon Sep 17 00:00:00 2001 From: Tony Hutter Date: Thu, 9 Nov 2023 16:43:35 -0800 Subject: [PATCH] Workaround UBSAN errors for variable arrays @@ -70,6 +70,3 @@ index c13217159..b9c284a24 100644 # Suppress incorrect warnings from versions of objtool which are not # aware of x86 EVEX prefix instructions used for AVX512. OBJECT_FILES_NON_STANDARD_vdev_raidz_math_avx512bw.o := y --- -2.39.2 - diff --git a/debian/patches/0015-Fix-block-cloning-between-unencrypted-and-encrypted-.patch b/debian/patches/0015-Fix-block-cloning-between-unencrypted-and-encrypted-.patch new file mode 100644 index ..c2fc506e --- /dev/null +++ b/debian/patches/0015-Fix-block-cloning-between-unencrypted-and-encrypted-.patch @@ -0,0 +1,44 @@ +From Mon Sep 17 00:00:00 2001 +From: =?UTF-8?q?Martin=20Matu=C5=A1ka?= +Date: Tue, 31 Oct 2023 21:49:41 +0100 +Subject: [PATCH] Fix block cloning between unencrypted and encrypted datasets + +Block cloning from an encrypted dataset into an unencrypted dataset +and vice versa is not possible. The current code did allow cloning +unencrypted files into an encrypted dataset causing a panic when +these were accessed. Block cloning between encrypted and encrypted +is currently supported on the same filesystem only. + +Reviewed-by: Alexander Motin +Reviewed-by: Kay Pedersen +Reviewed-by: Rob N +Reviewed-by: Brian Behlendorf +Signed-off-by: Martin Matuska +Closes #15464 +Closes #15465 +(cherry picked from commit 459c99ff2339a4a514abcf2255f9b3e5324ef09e) +Signed-off-by: Stoiko Ivanov +--- + module/zfs/zfs_vnops.c | 9 + + 1 file changed, 9 insertions(+) + +diff --git a/module/zfs/zfs_vnops.c b/module/zfs/zfs_vnops.c +index 40d6c87a7..84e6b10ef 100644 +--- a/module/zfs/zfs_vnops.c b/module/zfs/zfs_vnops.c +@@ -1094,6 +1094,15 @@ zfs_clone_range(znode_t *inzp, uint64_t *inoffp, znode_t *outzp, + + ASSERT(!outzfsvfs->z_replay); + ++ /* ++ * Block cloning from an unencrypted dataset into an encrypted ++ * dataset and vice versa is not supported. ++ */ ++ if (inos->os_encrypted != outos->os_encrypted) { ++ zfs_exit_two(inzfsvfs, outzfsvfs, FTAG); ++ return (SET_ERROR(EXDEV)); ++ } ++ + error = zfs_verify_zp(inzp); + if (error == 0) + error = zfs_verify_zp(outzp); diff --git a/debian/patches/0016-Add-a-tunable-to-disable-BRT-support.patch b/debian/patches/0016-Add-a-tunable-to-disable-BRT-support.patch new file mode 100644 index ..53977479 --- /dev/null +++ b/debian/patches/0016-Add-a-tunable-to-disable-BRT-support.patch @@ -0,0 +1,201 @@ +From Mon Sep 17 00:00:00 2001 +From: Rich Ercolani <214141+rincebr...@users.noreply.github.com> +Date: Thu, 16 Nov 2023 14:35:22 -0500 +Subject: [PATCH] Add a tunable to disable BRT support. + +Copy the disable parameter that FreeBSD implemented, and extend it to +work on Linux as well, until we're sure this is stable. + +Reviewed-by: Alexander Motin +Reviewed-by: Brian Behlendorf +Signed-off-by: Rich Ercolani +Closes #15529 +(cherr
[pve-devel] [PATCH installer] run env: do not store emtpy hostname
without this patch the hostname ends up as the empty string in run-env-info.json, which results in a parse-error in the TUI code (an empty string is not None, but still too short as hostname) Minimally tested on a VM. Fixes: bda1cdf699a3fcfc1cf3cfa446b1493689fc8eb8 Signed-off-by: Stoiko Ivanov --- Proxmox/Install/RunEnv.pm | 4 +++- Proxmox/Sys/Net.pm| 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Proxmox/Install/RunEnv.pm b/Proxmox/Install/RunEnv.pm index 5f68d82..2d91401 100644 --- a/Proxmox/Install/RunEnv.pm +++ b/Proxmox/Install/RunEnv.pm @@ -268,7 +268,9 @@ sub query_installation_environment : prototype() { }; # Cannot be put directly in the above hash as it might return undef .. -$output->{network}->{hostname} = Proxmox::Sys::Net::get_dhcp_hostname(); +if ( my $hostname = Proxmox::Sys::Net::get_dhcp_hostname()) { + $output->{network}->{hostname} = $hostname; +} # FIXME: move whatever makes sense over to Proxmox::Sys::Net:: and keep that as single source, # it can then use some different structure just fine (after adapting the GTK GUI to that) but diff --git a/Proxmox/Sys/Net.pm b/Proxmox/Sys/Net.pm index 35d2abd..7415bf9 100644 --- a/Proxmox/Sys/Net.pm +++ b/Proxmox/Sys/Net.pm @@ -211,7 +211,7 @@ sub get_dhcp_hostname : prototype() { } close($fh); -return $1 if defined($name) && $name =~ m/^([^\.]+)(?:\.(?:\S+))?$/; +return $name if defined($name) && $name =~ m/^([^\.]+)(?:\.(?:\S+))?$/; } 1; -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer] zfs: create /var/lib/vz only for PVE
the recent patch to create /var/lib/vz as dedicated dataset, did so for all our products - but this is only needed/wanted for PVE moved the creation of the root-dataset above the creation of rpool/data, so that the pve-specifics can remain in one if block. Fixes: dd19d40ceac179ba18652f1d6c3e4c23f246af00 Signed-off-by: Stoiko Ivanov --- sending as separate patch in place of the v2 for the installer-glitches, because I forgot to pull before sending the v2 Proxmox/Install.pm | 11 +-- 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm index 4045a97..1a4ee93 100644 --- a/Proxmox/Install.pm +++ b/Proxmox/Install.pm @@ -182,15 +182,14 @@ sub zfs_create_rpool { syscmd("zfs create $pool_name/ROOT") == 0 || die "unable to create zfs $pool_name/ROOT volume\n"; -if ($iso_env->{product} eq 'pve') { - syscmd("zfs create $pool_name/data") == 0 || die "unable to create zfs $pool_name/data volume\n"; -} - syscmd("zfs create $pool_name/ROOT/$root_volume_name") == 0 || die "unable to create zfs $pool_name/ROOT/$root_volume_name volume\n"; -syscmd("zfs create -p $pool_name/ROOT/$root_volume_name/var/lib/vz") == 0 || - die "unable to create zfs $pool_name/ROOT/$root_volume_name/var/lib/vz volume\n"; +if ($iso_env->{product} eq 'pve') { + syscmd("zfs create $pool_name/data") == 0 || die "unable to create zfs $pool_name/data volume\n"; + syscmd("zfs create -p $pool_name/ROOT/$root_volume_name/var/lib/vz") == 0 || + die "unable to create zfs $pool_name/ROOT/$root_volume_name/var/lib/vz volume\n"; +} # default to `relatime` on, fast enough for the installer and production syscmd("zfs set atime=on relatime=on $pool_name") == 0 || die "unable to set zfs properties\n"; -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v2 1/4] fix #4747: pass kernel cmdline parameters to target system
Parameters needed for booting during installation are best preserved in the target cmdline as well - e.g. if you need a particular cmdline switch for your system to boot at all - not having to add it for the first boot of the installed system and manually adding it to the bootloader config is an improvement. This additionally enables us to drop the console parameter handling for serial consoles (it is just one of the parameters to pass along). Finally it fixes the regular expressions for the installer settings we read from the cmdline (swapsize, maxroot,...) which were broken if added as last entry. Signed-off-by: Stoiko Ivanov --- Proxmox/Install.pm| 11 +-- Proxmox/Install/Config.pm | 25 + 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm index 66adb2d..c868992 100644 --- a/Proxmox/Install.pm +++ b/Proxmox/Install.pm @@ -1152,11 +1152,10 @@ _EOD } update_progress(0.8, 0.95, 1, "make system bootable"); - my $console_param=''; - if (my $console = Proxmox::Install::Config::get_console()) { - $console_param="console=$console"; - my $console_snippet = "GRUB_CMDLINE_LINUX=\"\$GRUB_CMDLINE_LINUX $console_param\""; - file_write_all("$targetdir/etc/default/grub.d/console.cfg", $console_snippet); + my $target_cmdline=''; + if ($target_cmdline = Proxmox::Install::Config::get_target_cmdline()) { + my $target_cmdline_snippet = "GRUB_CMDLINE_LINUX=\"\$GRUB_CMDLINE_LINUX $target_cmdline\""; + file_write_all("$targetdir/etc/default/grub.d/installer.cfg", $target_cmdline_snippet); } if ($use_zfs) { @@ -1164,7 +1163,7 @@ _EOD my $zfs_snippet = "GRUB_CMDLINE_LINUX=\"\$GRUB_CMDLINE_LINUX root=ZFS=$zfs_pool_name/ROOT/$zfs_root_volume_name boot=zfs\""; file_write_all("$targetdir/etc/default/grub.d/zfs.cfg", $zfs_snippet); - file_write_all("$targetdir/etc/kernel/cmdline", "root=ZFS=$zfs_pool_name/ROOT/$zfs_root_volume_name boot=zfs $console_param\n"); + file_write_all("$targetdir/etc/kernel/cmdline", "root=ZFS=$zfs_pool_name/ROOT/$zfs_root_volume_name boot=zfs $target_cmdline\n"); zfs_setup_module_conf($targetdir); } diff --git a/Proxmox/Install/Config.pm b/Proxmox/Install/Config.pm index 5e80255..b1acebc 100644 --- a/Proxmox/Install/Config.pm +++ b/Proxmox/Install/Config.pm @@ -16,36 +16,37 @@ my sub parse_kernel_cmdline { my $cmdline = Proxmox::Install::RunEnv::get('kernel_cmdline'); -if ($cmdline =~ m/\s(ext4|xfs)(\s.*)?$/) { +if ($cmdline =~ s/\b(ext4|xfs)\s?//i) { $cfg->{filesys} = $1; } -if ($cmdline =~ m/hdsize=(\d+(\.\d+)?)[\s\n]/i) { +if ($cmdline =~ s/\bhdsize=(\d+(\.\d+)?)\s?//i) { $cfg->{hdsize} = $1; } -if ($cmdline =~ m/swapsize=(\d+(\.\d+)?)[\s\n]/i) { +if ($cmdline =~ s/\bswapsize=(\d+(\.\d+)?)\s?//i) { $cfg->{swapsize} = $1; } -if ($cmdline =~ m/maxroot=(\d+(\.\d+)?)[\s\n]/i) { +if ($cmdline =~ s/\bmaxroot=(\d+(\.\d+)?)\s?//i) { $cfg->{maxroot} = $1; } -if ($cmdline =~ m/minfree=(\d+(\.\d+)?)[\s\n]/i) { +if ($cmdline =~ s/\bminfree=(\d+(\.\d+)?)\s?//i) { $cfg->{minfree} = $1; } my $iso_env = Proxmox::Install::ISOEnv::get(); if ($iso_env->{product} eq 'pve') { - if ($cmdline =~ m/maxvz=(\d+(\.\d+)?)[\s\n]/i) { + if ($cmdline =~ s/\bmaxvz=(\d+(\.\d+)?)\s?//i) { $cfg->{maxvz} = $1; } } -if ($cmdline =~ m/console=(\S+)[\s\n]?/i) { - $cfg->{console} = $1; -} +$cmdline =~ s/(?:BOOT_IMAGE|root|ramdisk_size|splash|vga)=\S+\s?//gi; +$cmdline =~ s/ro|rw|quiet|proxdebug|proxtui|nomodeset//gi; + +$cfg->{target_cmdline}= $cmdline; return $cfg; } @@ -101,7 +102,7 @@ my sub init_cfg { cidr => undef, gateway => undef, dns => undef, - console => undef, + target_cmdline => undef, }; $initial = parse_kernel_cmdline($initial); @@ -235,8 +236,8 @@ sub get_gateway { return get('gateway'); } sub set_dns { set_key('dns', $_[0]); } sub get_dns { return get('dns'); } -sub set_console { set_key('console', $_[0]); } -sub get_console { return get('console'); } +sub set_target_cmdline { set_key('target_cmdline', $_[0]); } +sub get_target_cmdline { return get('target_cmdline'); } 1; -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v2 0/4] adaptation to kernel cmdline handling and 2 small ZFS related improvements
v1->v2: * adapted the creation of /var/lib/vz as separate dataset to be only done for PVE and not for our other products original cover-letter for v1: This patch-series contains mostly unrelated patches, which still can be tested well together: * patch 1/4 fixes the parsing of our kernel cmdline parameters during the install (they were ignored if being the last on the line). additionally everything not used by our installer now end up in the target system's kernel cmdline - so if you need to add something for the system to boot - you do not need to do so 3 times (for the install, for the first boot, for the boot-loader config) * patch 2/4 is a minor glitch I introduced, which caused me to debug the wrong thing for too long * patch 3/4 has been a long-standing and quite sensible request * patch 4/4 fixes a minor inconvenience on machines with local users and ZFS on root (dmesg gets filled with journald-messages upon journal-rotation and user-logins) Tested the changes a bit (4/4 on my machine, 3/4 was straight-forward, 2/4 with a quick look, 1/4 with a few installer-options and net.ifnames=0) Stoiko Ivanov (4): fix #4747: pass kernel cmdline parameters to target system unconfigured.sh: set serial to a number to prevent warning fix #1410: zfs: create /var/lib/vz as separate dataset zfs: set acltype=posix for root-dataset Proxmox/Install.pm| 21 - Proxmox/Install/Config.pm | 25 + unconfigured.sh | 1 + 3 files changed, 26 insertions(+), 21 deletions(-) -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v2 3/4] fix #1410: zfs: create /var/lib/vz as separate dataset
this enables the users to set reservations on / separate from /var/lib/vz - where backups, ISOs, and other data might fill the complete pool. moved the creation of the root-dataset above the creation of rpool/data, so that the pve-specifics can remain in one if block. Signed-off-by: Stoiko Ivanov --- Proxmox/Install.pm | 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm index c868992..48c157a 100644 --- a/Proxmox/Install.pm +++ b/Proxmox/Install.pm @@ -182,13 +182,15 @@ sub zfs_create_rpool { syscmd("zfs create $pool_name/ROOT") == 0 || die "unable to create zfs $pool_name/ROOT volume\n"; +syscmd("zfs create $pool_name/ROOT/$root_volume_name") == 0 || + die "unable to create zfs $pool_name/ROOT/$root_volume_name volume\n"; + if ($iso_env->{product} eq 'pve') { syscmd("zfs create $pool_name/data") == 0 || die "unable to create zfs $pool_name/data volume\n"; + syscmd("zfs create -p $pool_name/ROOT/$root_volume_name/var/lib/vz") == 0 || + die "unable to create zfs $pool_name/ROOT/$root_volume_name/var/lib/vz volume\n"; } -syscmd("zfs create $pool_name/ROOT/$root_volume_name") == 0 || - die "unable to create zfs $pool_name/ROOT/$root_volume_name volume\n"; - # default to `relatime` on, fast enough for the installer and production syscmd("zfs set atime=on relatime=on $pool_name") == 0 || die "unable to set zfs properties\n"; -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v2 4/4] zfs: set acltype=posix for root-dataset
journald as a core component tries setting a ACL on the journal files for (non-root) users and fails on our ZFS installs. Resulting in dmesg being spammed with messages from journald upon each journal-rotation for each user upon their first login. This is also suggested by OpenZFS in their Debian guide for root on ZFS: https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Bookworm%20Root%20on%20ZFS.html Tested by setting this on a machine of mine, where this has been bugging for quite a while. Signed-off-by: Stoiko Ivanov --- Proxmox/Install.pm | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm index 48c157a..1a4ee93 100644 --- a/Proxmox/Install.pm +++ b/Proxmox/Install.pm @@ -202,6 +202,8 @@ sub zfs_create_rpool { $value = $zfs_opts->{copies} // 1; syscmd("zfs set copies=$value $pool_name") if defined($value) && $value != 1; + +syscmd("zfs set acltype=posix $pool_name/ROOT/$root_volume_name"); } my $get_raid_devlist = sub { -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v2 2/4] unconfigured.sh: set serial to a number to prevent warning
caught me off-guard while debugging other things - the message: ` [: : integer expression expected` made me look a bit too long at the wrong place. Fixes: a31259b1597447a0b431cd5c81a6db2bc80f1ddf Signed-off-by: Stoiko Ivanov --- unconfigured.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/unconfigured.sh b/unconfigured.sh index a361a20..6b3a8bf 100755 --- a/unconfigured.sh +++ b/unconfigured.sh @@ -7,6 +7,7 @@ trap "err_reboot" ERR parse_cmdline() { proxdebug=0 proxtui=0 +serial=0 # shellcheck disable=SC2013 # per word splitting is wanted here for par in $(cat /proc/cmdline); do case $par in -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH installer 3/4] fix #1410: zfs: create /var/lib/vz as separate dataset
this is wrong! the dataset is only needed for pve I'll resend a v2 sorry for the noise! On Thu, 16 Nov 2023 16:00:40 +0100 Stoiko Ivanov wrote: > this enables the users to set reservations on / separate from > /var/lib/vz - where backups, ISOs, and other data might fill the > complete pool. > > Signed-off-by: Stoiko Ivanov > --- > Proxmox/Install.pm | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm > index c868992..a96249e 100644 > --- a/Proxmox/Install.pm > +++ b/Proxmox/Install.pm > @@ -189,6 +189,9 @@ sub zfs_create_rpool { > syscmd("zfs create $pool_name/ROOT/$root_volume_name") == 0 || > die "unable to create zfs $pool_name/ROOT/$root_volume_name volume\n"; > > +syscmd("zfs create -p $pool_name/ROOT/$root_volume_name/var/lib/vz") == > 0 || > + die "unable to create zfs $pool_name/ROOT/$root_volume_name/var/lib/vz > volume\n"; > + > # default to `relatime` on, fast enough for the installer and production > syscmd("zfs set atime=on relatime=on $pool_name") == 0 || die "unable to > set zfs properties\n"; > ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 2/4] unconfigured.sh: set serial to a number to prevent warning
caught me off-guard while debugging other things - the message: ` [: : integer expression expected` made me look a bit too long at the wrong place. Fixes: a31259b1597447a0b431cd5c81a6db2bc80f1ddf Signed-off-by: Stoiko Ivanov --- unconfigured.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/unconfigured.sh b/unconfigured.sh index a361a20..6b3a8bf 100755 --- a/unconfigured.sh +++ b/unconfigured.sh @@ -7,6 +7,7 @@ trap "err_reboot" ERR parse_cmdline() { proxdebug=0 proxtui=0 +serial=0 # shellcheck disable=SC2013 # per word splitting is wanted here for par in $(cat /proc/cmdline); do case $par in -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 4/4] zfs: set acltype=posix for root-dataset
journald as a core component tries setting a ACL on the journal files for (non-root) users and fails on our ZFS installs. Resulting in dmesg being spammed with messages from journald upon each journal-rotation for each user upon their first login. This is also suggested by OpenZFS in their Debian guide for root on ZFS: https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Bookworm%20Root%20on%20ZFS.html Tested by setting this on a machine of mine, where this has been bugging for quite a while. Signed-off-by: Stoiko Ivanov --- Proxmox/Install.pm | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm index a96249e..4045a97 100644 --- a/Proxmox/Install.pm +++ b/Proxmox/Install.pm @@ -203,6 +203,8 @@ sub zfs_create_rpool { $value = $zfs_opts->{copies} // 1; syscmd("zfs set copies=$value $pool_name") if defined($value) && $value != 1; + +syscmd("zfs set acltype=posix $pool_name/ROOT/$root_volume_name"); } my $get_raid_devlist = sub { -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 0/4] adaptation to kernel cmdline handling and 2 small ZFS related improvements
This patch-series contains mostly unrelated patches, which still can be tested well together: * patch 1/4 fixes the parsing of our kernel cmdline parameters during the install (they were ignored if being the last on the line). additionally everything not used by our installer now end up in the target system's kernel cmdline - so if you need to add something for the system to boot - you do not need to do so 3 times (for the install, for the first boot, for the boot-loader config) * patch 2/4 is a minor glitch I introduced, which caused me to debug the wrong thing for too long * patch 3/4 has been a long-standing and quite sensible request * patch 4/4 fixes a minor inconvenience on machines with local users and ZFS on root (dmesg gets filled with journald-messages upon journal-rotation and user-logins) Tested the changes a bit (4/4 on my machine, 3/4 was straight-forward, 2/4 with a quick look, 1/4 with a few installer-options and net.ifnames=0) Stoiko Ivanov (4): fix #4747: pass kernel cmdline parameters to target system unconfigured.sh: set serial to a number to prevent warning fix #1410: zfs: create /var/lib/vz as separate dataset zfs: set acltype=posix for root-dataset Proxmox/Install.pm| 16 ++-- Proxmox/Install/Config.pm | 25 + unconfigured.sh | 1 + 3 files changed, 24 insertions(+), 18 deletions(-) -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 1/4] fix #4747: pass kernel cmdline parameters to target system
Parameters needed for booting during installation are best preserved in the target cmdline as well - e.g. if you need a particular cmdline switch for your system to boot at all - not having to add it for the first boot of the installed system and manually adding it to the bootloader config is an improvement. This additionally enables us to drop the console parameter handling for serial consoles (it is just one of the parameters to pass along). Finally it fixes the regular expressions for the installer settings we read from the cmdline (swapsize, maxroot,...) which were broken if added as last entry. Signed-off-by: Stoiko Ivanov --- Proxmox/Install.pm| 11 +-- Proxmox/Install/Config.pm | 25 + 2 files changed, 18 insertions(+), 18 deletions(-) diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm index 66adb2d..c868992 100644 --- a/Proxmox/Install.pm +++ b/Proxmox/Install.pm @@ -1152,11 +1152,10 @@ _EOD } update_progress(0.8, 0.95, 1, "make system bootable"); - my $console_param=''; - if (my $console = Proxmox::Install::Config::get_console()) { - $console_param="console=$console"; - my $console_snippet = "GRUB_CMDLINE_LINUX=\"\$GRUB_CMDLINE_LINUX $console_param\""; - file_write_all("$targetdir/etc/default/grub.d/console.cfg", $console_snippet); + my $target_cmdline=''; + if ($target_cmdline = Proxmox::Install::Config::get_target_cmdline()) { + my $target_cmdline_snippet = "GRUB_CMDLINE_LINUX=\"\$GRUB_CMDLINE_LINUX $target_cmdline\""; + file_write_all("$targetdir/etc/default/grub.d/installer.cfg", $target_cmdline_snippet); } if ($use_zfs) { @@ -1164,7 +1163,7 @@ _EOD my $zfs_snippet = "GRUB_CMDLINE_LINUX=\"\$GRUB_CMDLINE_LINUX root=ZFS=$zfs_pool_name/ROOT/$zfs_root_volume_name boot=zfs\""; file_write_all("$targetdir/etc/default/grub.d/zfs.cfg", $zfs_snippet); - file_write_all("$targetdir/etc/kernel/cmdline", "root=ZFS=$zfs_pool_name/ROOT/$zfs_root_volume_name boot=zfs $console_param\n"); + file_write_all("$targetdir/etc/kernel/cmdline", "root=ZFS=$zfs_pool_name/ROOT/$zfs_root_volume_name boot=zfs $target_cmdline\n"); zfs_setup_module_conf($targetdir); } diff --git a/Proxmox/Install/Config.pm b/Proxmox/Install/Config.pm index 5e80255..b1acebc 100644 --- a/Proxmox/Install/Config.pm +++ b/Proxmox/Install/Config.pm @@ -16,36 +16,37 @@ my sub parse_kernel_cmdline { my $cmdline = Proxmox::Install::RunEnv::get('kernel_cmdline'); -if ($cmdline =~ m/\s(ext4|xfs)(\s.*)?$/) { +if ($cmdline =~ s/\b(ext4|xfs)\s?//i) { $cfg->{filesys} = $1; } -if ($cmdline =~ m/hdsize=(\d+(\.\d+)?)[\s\n]/i) { +if ($cmdline =~ s/\bhdsize=(\d+(\.\d+)?)\s?//i) { $cfg->{hdsize} = $1; } -if ($cmdline =~ m/swapsize=(\d+(\.\d+)?)[\s\n]/i) { +if ($cmdline =~ s/\bswapsize=(\d+(\.\d+)?)\s?//i) { $cfg->{swapsize} = $1; } -if ($cmdline =~ m/maxroot=(\d+(\.\d+)?)[\s\n]/i) { +if ($cmdline =~ s/\bmaxroot=(\d+(\.\d+)?)\s?//i) { $cfg->{maxroot} = $1; } -if ($cmdline =~ m/minfree=(\d+(\.\d+)?)[\s\n]/i) { +if ($cmdline =~ s/\bminfree=(\d+(\.\d+)?)\s?//i) { $cfg->{minfree} = $1; } my $iso_env = Proxmox::Install::ISOEnv::get(); if ($iso_env->{product} eq 'pve') { - if ($cmdline =~ m/maxvz=(\d+(\.\d+)?)[\s\n]/i) { + if ($cmdline =~ s/\bmaxvz=(\d+(\.\d+)?)\s?//i) { $cfg->{maxvz} = $1; } } -if ($cmdline =~ m/console=(\S+)[\s\n]?/i) { - $cfg->{console} = $1; -} +$cmdline =~ s/(?:BOOT_IMAGE|root|ramdisk_size|splash|vga)=\S+\s?//gi; +$cmdline =~ s/ro|rw|quiet|proxdebug|proxtui|nomodeset//gi; + +$cfg->{target_cmdline}= $cmdline; return $cfg; } @@ -101,7 +102,7 @@ my sub init_cfg { cidr => undef, gateway => undef, dns => undef, - console => undef, + target_cmdline => undef, }; $initial = parse_kernel_cmdline($initial); @@ -235,8 +236,8 @@ sub get_gateway { return get('gateway'); } sub set_dns { set_key('dns', $_[0]); } sub get_dns { return get('dns'); } -sub set_console { set_key('console', $_[0]); } -sub get_console { return get('console'); } +sub set_target_cmdline { set_key('target_cmdline', $_[0]); } +sub get_target_cmdline { return get('target_cmdline'); } 1; -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 3/4] fix #1410: zfs: create /var/lib/vz as separate dataset
this enables the users to set reservations on / separate from /var/lib/vz - where backups, ISOs, and other data might fill the complete pool. Signed-off-by: Stoiko Ivanov --- Proxmox/Install.pm | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm index c868992..a96249e 100644 --- a/Proxmox/Install.pm +++ b/Proxmox/Install.pm @@ -189,6 +189,9 @@ sub zfs_create_rpool { syscmd("zfs create $pool_name/ROOT/$root_volume_name") == 0 || die "unable to create zfs $pool_name/ROOT/$root_volume_name volume\n"; +syscmd("zfs create -p $pool_name/ROOT/$root_volume_name/var/lib/vz") == 0 || + die "unable to create zfs $pool_name/ROOT/$root_volume_name/var/lib/vz volume\n"; + # default to `relatime` on, fast enough for the installer and production syscmd("zfs set atime=on relatime=on $pool_name") == 0 || die "unable to set zfs properties\n"; -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 2/2] serial install: fix console parameter parsing
The regex matching in Proxmox::Install::Config was blindly copied from above - so the other parameters are also likely to not get recognized if they are the last on the cmdline Signed-off-by: Stoiko Ivanov --- Proxmox/Install.pm| 2 +- Proxmox/Install/Config.pm | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm index a289037..66adb2d 100644 --- a/Proxmox/Install.pm +++ b/Proxmox/Install.pm @@ -1154,7 +1154,7 @@ _EOD update_progress(0.8, 0.95, 1, "make system bootable"); my $console_param=''; if (my $console = Proxmox::Install::Config::get_console()) { - my $console_param="console=$console";; + $console_param="console=$console"; my $console_snippet = "GRUB_CMDLINE_LINUX=\"\$GRUB_CMDLINE_LINUX $console_param\""; file_write_all("$targetdir/etc/default/grub.d/console.cfg", $console_snippet); } diff --git a/Proxmox/Install/Config.pm b/Proxmox/Install/Config.pm index 55e53c7..5e80255 100644 --- a/Proxmox/Install/Config.pm +++ b/Proxmox/Install/Config.pm @@ -43,7 +43,7 @@ my sub parse_kernel_cmdline { } } -if ($cmdline =~ m/console=(\S+)[\s\n]/i) { +if ($cmdline =~ m/console=(\S+)[\s\n]?/i) { $cfg->{console} = $1; } -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 0/2] minimal changes for a working serial installation
patch 1/2 is a tiny issue I ran into while building the installer patch 2/2 adds the correct console parameter for serial installations patch 2/2 needs some further work (the other kernel cmdline parameter matchings are probably wrong as well, additionally #4747 would be a better and more general solution for what to add to the cmdline from the install environment) - I'll try to send a follow-up tomorrow Stoiko Ivanov (2): d/control: add libgtk3-perl to B-D serial install: fix console parameter parsing Proxmox/Install.pm| 2 +- Proxmox/Install/Config.pm | 2 +- debian/control| 1 + 3 files changed, 3 insertions(+), 2 deletions(-) -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer 1/2] d/control: add libgtk3-perl to B-D
With the first tests entering the installer in 9a0d66cb36d395a1186904132aed1d5dc33a0937 we now need libgtk3-perl during package-building with `make deb` Signed-off-by: Stoiko Ivanov --- debian/control | 1 + 1 file changed, 1 insertion(+) diff --git a/debian/control b/debian/control index d77b12a..9057f59 100644 --- a/debian/control +++ b/debian/control @@ -5,6 +5,7 @@ Maintainer: Proxmox Support Team Build-Depends: cargo:native, debhelper-compat (= 12), iso-codes, + libgtk3-perl, libpve-common-perl, librsvg2-bin, librust-cursive+termion-backend-dev (>= 0.20.0), -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH zfsonlinux] add patch for spurious warning on `zfs mount -a`
reported in our community forum: https://forum.proxmox.com/threads/.135635/post-60036 the small fix was merged upstream: https://github.com/openzfs/zfs/pull/15468 minimally tested by building with this patch and running `zfs mount -a` on an affected system. Signed-off-by: Stoiko Ivanov --- this patch fixes a cosmetic issue, but might help keep support requests down a bit. also quickly skimmed through the other patches in upstream/master and salsa.debian.org - but currently don't think anything needs to be pulled in urgently ...runcate_shares-without-etc-exports.d.patch | 76 +++ debian/patches/series | 1 + 2 files changed, 77 insertions(+) create mode 100644 debian/patches/0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch diff --git a/debian/patches/0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch b/debian/patches/0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch new file mode 100644 index ..7eb9721d --- /dev/null +++ b/debian/patches/0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch @@ -0,0 +1,76 @@ +From Mon Sep 17 00:00:00 2001 +From: siv0 +Date: Tue, 31 Oct 2023 21:57:54 +0100 +Subject: [PATCH] Fix nfs_truncate_shares without /etc/exports.d + +Calling nfs_reset_shares on Linux prints a warning: +`failed to lock /etc/exports.d/zfs.exports.lock: No such file or +directory` +when /etc/exports.d does not exist. The directory gets created, when a +filesystem is actually exported through nfs_toggle_share and +nfs_init_share. The truncation of /etc/exports.d/zfs.exports happens +unconditionally when calling `zfs mount -a` (via zfs_do_mount and +share_mount in `cmd/zfs/zfs_main.c`). + +Fixing the issue only in the Linux part, since the exports file on +freebsd is in `/etc/zfs/`, which seems present on 2 FreeBSD systems I +have access to (through `/etc/zfs/compatibility.d/`), while a Debian +box does not have the directory even if `/usr/sbin/exportfs` is +present through the `nfs-kernel-server` package. + +The code for exports_available is copied from nfs_available above. + +Fixes: ede037cda73675f42b1452187e8dd3438fafc220 +("Make zfs-share service resilient to stale exports") + +Reviewed-by: Brian Atkinson +Reviewed-by: Brian Behlendorf +Signed-off-by: Stoiko Ivanov +Closes #15369 +Closes #15468 +(cherry picked from commit 41e55b476bcfc90f1ad81c02c5375367fdace9e9) +Signed-off-by: Stoiko Ivanov +--- + lib/libshare/os/linux/nfs.c | 18 ++ + 1 file changed, 18 insertions(+) + +diff --git a/lib/libshare/os/linux/nfs.c b/lib/libshare/os/linux/nfs.c +index 004946b0c..3dce81840 100644 +--- a/lib/libshare/os/linux/nfs.c b/lib/libshare/os/linux/nfs.c +@@ -47,6 +47,7 @@ + + + static boolean_t nfs_available(void); ++static boolean_t exports_available(void); + + typedef int (*nfs_shareopt_callback_t)(const char *opt, const char *value, + void *cookie); +@@ -539,6 +540,8 @@ nfs_commit_shares(void) + static void + nfs_truncate_shares(void) + { ++ if (!exports_available()) ++ return; + nfs_reset_shares(ZFS_EXPORTS_LOCK, ZFS_EXPORTS_FILE); + } + +@@ -566,3 +569,18 @@ nfs_available(void) + + return (avail == 1); + } ++ ++static boolean_t ++exports_available(void) ++{ ++ static int avail; ++ ++ if (!avail) { ++ if (access(ZFS_EXPORTS_DIR, F_OK) != 0) ++ avail = -1; ++ else ++ avail = 1; ++ } ++ ++ return (avail == 1); ++} diff --git a/debian/patches/series b/debian/patches/series index 710cbfbe..6a5ab10f 100644 --- a/debian/patches/series +++ b/debian/patches/series @@ -9,3 +9,4 @@ 0009-arc-stat-summary-guard-access-to-l2arc-MFU-MRU-stats.patch 0010-zvol-Remove-broken-blk-mq-optimization.patch 0011-Revert-zvol-Temporally-disable-blk-mq.patch +0012-Fix-nfs_truncate_shares-without-etc-exports.d.patch -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH kernel-helper/manager v3] check for fitting grub-meta package on uefi systems
v2->v3: * adapted Friedrich's feedback (huge thanks for the patience and attention to semantically important details!!) - so that the pve7to8 warning is actually understandable v1->v2: * adapted Friedrich's feedback (huge thanks!) ** fixed the wrongly negated check for installed grub-efi-amd64 in the boot-tool hook. ** Rephrased the error-message in pve7to8 to 2 sentences. I tried adding a newline as well, however this results in the message not being printed in the warning color anymore (most likely due to [0]) - and I felt this to be more important than having it on a separate line. [0] https://perldoc.perl.org/Term::ANSIColor#RESTRICTIONS original cover-letter for v1: The following patchset is a followup to the one for the installer: https://lists.proxmox.com/pipermail/pve-devel/2023-September/059270.html As suggested by Thomas - adding the check to proxmox-kernel-helper seems like a good idea. While adding it to d/postinst I thought that this might not be the best place - and that getting the warning upon every kernel-upgrade would be better vs. upon every upgrade of proxmox-kernel-helper (which are far less often). (Can gladly send the version with d/postinst as well) If the pve-manager patch gets applied - I'd push the equivalent change to pmg and provide one for pbs. Tested on legacy and uefi VMs installed with pve-8.0 iso and grub-efi-amd64 (and systemd-boot) removed vs. installed. proxmox-kernel-helper Stoiko Ivanov (2): proxmox-boot-tool: do not exit early in kernel-hook proxmox-boot-tool: check if correct grub metapackage is installed src/proxmox-boot/zz-proxmox-boot | 21 - 1 file changed, 20 insertions(+), 1 deletion(-) pve-manager: Stoiko Ivanov (1): pve7to8: check for proper grub meta-package for bootmode PVE/CLI/pve7to8.pm | 39 +++ 1 file changed, 23 insertions(+), 16 deletions(-) -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH manager v3 1/1] pve7to8: check for proper grub meta-package for bootmode
This should catch installations from our ISO on non-ZFS in uefi mode, which won't get the updated grub efi binary installed upon upgrade, because grub-pc is installed instead of grub-efi-amd64. Adding this to pve7to8 should make this even more visible, than the corresponding patch for promxox-kernel-helper (warnings printed during regular package upgrades might be overlooked more easily than a yellow line in the major upgrade checkscript) The if/else order was chosen to limit the nesting level of the long messages. Signed-off-by: Stoiko Ivanov --- PVE/CLI/pve7to8.pm | 39 +++ 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/PVE/CLI/pve7to8.pm b/PVE/CLI/pve7to8.pm index d1a71eff..b34c8362 100644 --- a/PVE/CLI/pve7to8.pm +++ b/PVE/CLI/pve7to8.pm @@ -1302,29 +1302,36 @@ sub check_time_sync { sub check_bootloader { log_info("Checking bootloader configuration..."); -if (!$upgraded) { - log_skip("not yet upgraded, no need to check the presence of systemd-boot"); - return; -} -if (! -f "/etc/kernel/proxmox-boot-uuids") { - log_skip("proxmox-boot-tool not used for bootloader configuration"); +if (! -d '/sys/firmware/efi') { + log_skip("System booted in legacy-mode - no need for additional packages"); return; } -if (! -d "/sys/firmware/efi") { - log_skip("System booted in legacy-mode - no need for systemd-boot"); - return; -} - -if ( -f "/usr/share/doc/systemd-boot/changelog.Debian.gz") { - log_pass("systemd-boot is installed"); -} else { +if ( -f "/etc/kernel/proxmox-boot-uuids") { + if (!$upgraded) { + log_skip("not yet upgraded, no need to check the presence of systemd-boot"); + return; + } + if ( -f "/usr/share/doc/systemd-boot/changelog.Debian.gz") { + log_pass("bootloader packages installed correctly"); + return; + } log_warn( "proxmox-boot-tool is used for bootloader configuration in uefi mode" - . "but the separate systemd-boot package, existing in Debian Bookworm is not installed" - . "initializing new ESPs will not work until the package is installed" + . " but the separate systemd-boot package is not installed," + . " initializing new ESPs will not work until the package is installed" + ); + return; +} elsif ( ! -f "/usr/share/doc/grub-efi-amd64/changelog.Debian.gz" ) { + log_warn( + "System booted in uefi mode but grub-efi-amd64 meta-package not installed," + . " new grub versions will not be installed to /boot/efi!" + . " Install grub-efi-amd64." ); + return; +} else { + log_pass("bootloader packages installed correctly"); } } -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH kernel-helper v3 1/2] proxmox-boot-tool: do not exit early in kernel-hook
update_esps is called first in the actual execution below - exiting early does not work for systems that don't use proxmox-boot-tool if a check added later needs to work there too. Signed-off-by: Stoiko Ivanov --- src/proxmox-boot/zz-proxmox-boot | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/proxmox-boot/zz-proxmox-boot b/src/proxmox-boot/zz-proxmox-boot index 793882b..1adc1b1 100755 --- a/src/proxmox-boot/zz-proxmox-boot +++ b/src/proxmox-boot/zz-proxmox-boot @@ -44,7 +44,7 @@ fi update_esps() { if [ ! -f "${ESP_LIST}" ]; then warn "No ${ESP_LIST} found, skipping ESP sync." - exit 0 + return fi if [ -f /etc/kernel/cmdline ]; then # we can have cmdline files with multiple or no new line at all, handle both! -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH kernel-helper v3 2/2] proxmox-boot-tool: check if correct grub metapackage is installed
this part of the hook applies only to systems not using pbt for bootmangement. Currently our ISO installs grub-pc unconditionally - and never the conflicting grub-efi-amd64. Both packages are responsible for running grub-install (for the appropriate disks) upon an upgrade of grub. This results in grub currently not getting updated on uefi-booted systems (which do not use proxmox-boot-tool). The patch causes a warning to be printed to notify the user. Also considered putting the check+warning in d/postinst - but this way it will get triggered more often (upon every kernel-upgrade/update-initramfs, instead of only on proxmox-kernel-helper updates, which are less often), increasing the chances of being noticed. checking for the changelog-presence was chosen, over `dpkg-query` for the status, for consistency with the similar patch for pve7to8 (and potentially a small speed-gain). Suggested-by: Thomas Lamprecht Signed-off-by: Stoiko Ivanov --- src/proxmox-boot/zz-proxmox-boot | 19 +++ 1 file changed, 19 insertions(+) diff --git a/src/proxmox-boot/zz-proxmox-boot b/src/proxmox-boot/zz-proxmox-boot index 1adc1b1..4dfa765 100755 --- a/src/proxmox-boot/zz-proxmox-boot +++ b/src/proxmox-boot/zz-proxmox-boot @@ -215,6 +215,23 @@ disable_systemd_boot_hook() { } +check_grub_efi_package() { + + if [ -f "${ESP_LIST}" ]; then + return + fi + + if [ ! -d /sys/firmware/efi ]; then + return + fi + + if [ -f /usr/share/doc/grub-efi-amd64/changelog.Debian.gz ]; then + return + fi + warn "uefi-booted system, without grub-efi-amd64 package - /boot/efi will not be updated" + +} + set -- $DEB_MAINT_PARAMS mode="${1#\'}" mode="${mode%\'}" @@ -228,6 +245,7 @@ case $0:$mode in BOOT_KVERS="$(boot_kernel_list "$@")" update_esps disable_systemd_boot_hook + check_grub_efi_package ;; */postrm.d/*:|*/postrm.d/*:remove) reexec_in_mountns "$@" @@ -235,6 +253,7 @@ case $0:$mode in BOOT_KVERS="$(boot_kernel_list)" update_esps disable_systemd_boot_hook + check_grub_efi_package ;; esac -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH kernel-helper v2 2/2] proxmox-boot-tool: check if correct grub metapackage is installed
this part of the hook applies only to systems not using pbt for bootmangement. Currently our ISO installs grub-pc unconditionally - and never the conflicting grub-efi-amd64. Both packages are responsible for running grub-install (for the appropriate disks) upon an upgrade of grub. This results in grub currently not getting updated on uefi-booted systems (which do not use proxmox-boot-tool). The patch causes a warning to be printed to notify the user. Also considered putting the check+warning in d/postinst - but this way it will get triggered more often (upon every kernel-upgrade/update-initramfs, instead of only on proxmox-kernel-helper updates, which are less often), increasing the chances of being noticed. checking for the changelog-presence was chosen, over `dpkg-query` for the status, for consistency with the similar patch for pve7to8 (and potentially a small speed-gain). Suggested-by: Thomas Lamprecht Signed-off-by: Stoiko Ivanov --- src/proxmox-boot/zz-proxmox-boot | 19 +++ 1 file changed, 19 insertions(+) diff --git a/src/proxmox-boot/zz-proxmox-boot b/src/proxmox-boot/zz-proxmox-boot index 1adc1b1..4dfa765 100755 --- a/src/proxmox-boot/zz-proxmox-boot +++ b/src/proxmox-boot/zz-proxmox-boot @@ -215,6 +215,23 @@ disable_systemd_boot_hook() { } +check_grub_efi_package() { + + if [ -f "${ESP_LIST}" ]; then + return + fi + + if [ ! -d /sys/firmware/efi ]; then + return + fi + + if [ -f /usr/share/doc/grub-efi-amd64/changelog.Debian.gz ]; then + return + fi + warn "uefi-booted system, without grub-efi-amd64 package - /boot/efi will not be updated" + +} + set -- $DEB_MAINT_PARAMS mode="${1#\'}" mode="${mode%\'}" @@ -228,6 +245,7 @@ case $0:$mode in BOOT_KVERS="$(boot_kernel_list "$@")" update_esps disable_systemd_boot_hook + check_grub_efi_package ;; */postrm.d/*:|*/postrm.d/*:remove) reexec_in_mountns "$@" @@ -235,6 +253,7 @@ case $0:$mode in BOOT_KVERS="$(boot_kernel_list)" update_esps disable_systemd_boot_hook + check_grub_efi_package ;; esac -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH kernel-helper v2 1/2] proxmox-boot-tool: do not exit early in kernel-hook
update_esps is called first in the actual execution below - exiting early does not work for systems that don't use proxmox-boot-tool if a check added later needs to work there too. Signed-off-by: Stoiko Ivanov --- src/proxmox-boot/zz-proxmox-boot | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/proxmox-boot/zz-proxmox-boot b/src/proxmox-boot/zz-proxmox-boot index 793882b..1adc1b1 100755 --- a/src/proxmox-boot/zz-proxmox-boot +++ b/src/proxmox-boot/zz-proxmox-boot @@ -44,7 +44,7 @@ fi update_esps() { if [ ! -f "${ESP_LIST}" ]; then warn "No ${ESP_LIST} found, skipping ESP sync." - exit 0 + return fi if [ -f /etc/kernel/cmdline ]; then # we can have cmdline files with multiple or no new line at all, handle both! -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH kernel-helper/manager v2] check for fitting grub-meta package on uefi systems
v1->v2: * adapted Friedrich's feedback (huge thanks!) ** fixed the wrongly negated check for installed grub-efi-amd64 in the boot-tool hook. ** Rephrased the error-message in pve7to8 to 2 sentences. I tried adding a newline as well, however this results in the message not being printed in the warning color anymore (most likely due to [0]) - and I felt this to be more important than having it on a separate line. [0] https://perldoc.perl.org/Term::ANSIColor#RESTRICTIONS original cover-letter for v1: The following patchset is a followup to the one for the installer: https://lists.proxmox.com/pipermail/pve-devel/2023-September/059270.html As suggested by Thomas - adding the check to proxmox-kernel-helper seems like a good idea. While adding it to d/postinst I thought that this might not be the best place - and that getting the warning upon every kernel-upgrade would be better vs. upon every upgrade of proxmox-kernel-helper (which are far less often). (Can gladly send the version with d/postinst as well) If the pve-manager patch gets applied - I'd push the equivalent change to pmg and provide one for pbs. Tested on legacy and uefi VMs installed with pve-8.0 iso and grub-efi-amd64 (and systemd-boot) removed vs. installed. proxmox-kernel-helper: Stoiko Ivanov (2): proxmox-boot-tool: do not exit early in kernel-hook proxmox-boot-tool: check if correct grub metapackage is installed src/proxmox-boot/zz-proxmox-boot | 21 - 1 file changed, 20 insertions(+), 1 deletion(-) pve-manager: Stoiko Ivanov (1): pve7to8: check for proper grub meta-package for bootmode PVE/CLI/pve7to8.pm | 39 +++ 1 file changed, 23 insertions(+), 16 deletions(-) -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH manager v2 1/1] pve7to8: check for proper grub meta-package for bootmode
This should catch installations from our ISO on non-ZFS in uefi mode, which won't get the updated grub efi binary installed upon upgrade, because grub-pc is installed instead of grub-efi-amd64. Adding this to pve7to8 should make this even more visible, than the corresponding patch for promxox-kernel-helper (warnings printed during regular package upgrades might be overlooked more easily than a yellow line in the major upgrade checkscript) The if/else order was chosen to limit the nesting level of the long messages. Signed-off-by: Stoiko Ivanov --- PVE/CLI/pve7to8.pm | 39 +++ 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/PVE/CLI/pve7to8.pm b/PVE/CLI/pve7to8.pm index d1a71eff..ff7825b3 100644 --- a/PVE/CLI/pve7to8.pm +++ b/PVE/CLI/pve7to8.pm @@ -1302,29 +1302,36 @@ sub check_time_sync { sub check_bootloader { log_info("Checking bootloader configuration..."); -if (!$upgraded) { - log_skip("not yet upgraded, no need to check the presence of systemd-boot"); - return; -} -if (! -f "/etc/kernel/proxmox-boot-uuids") { - log_skip("proxmox-boot-tool not used for bootloader configuration"); +if (! -d '/sys/firmware/efi') { + log_skip("System booted in legacy-mode - no need for additional packages"); return; } -if (! -d "/sys/firmware/efi") { - log_skip("System booted in legacy-mode - no need for systemd-boot"); - return; -} - -if ( -f "/usr/share/doc/systemd-boot/changelog.Debian.gz") { - log_pass("systemd-boot is installed"); -} else { +if ( -f "/etc/kernel/proxmox-boot-uuids") { + if (!$upgraded) { + log_skip("not yet upgraded, no need to check the presence of systemd-boot"); + return; + } + if ( -f "/usr/share/doc/systemd-boot/changelog.Debian.gz") { + log_pass("bootloader packages installed correctly"); + return; + } log_warn( "proxmox-boot-tool is used for bootloader configuration in uefi mode" - . "but the separate systemd-boot package, existing in Debian Bookworm is not installed" - . "initializing new ESPs will not work until the package is installed" + . " but the separate systemd-boot package, existing in Debian Bookworm is not installed" + . " initializing new ESPs will not work until the package is installed" + ); + return; +} elsif ( ! -f "/usr/share/doc/grub-efi-amd64/changelog.Debian.gz" ) { + log_warn( + "System booted in uefi mode but grub-efi-amd64 meta-package not installed" + . " new grub versions will not be installed to /boot/efi!" + . " Install grub-efi-amd64." ); + return; +} else { + log_pass("bootloader packages installed correctly"); } } -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH 1/1] pve7to8: check for proper grub meta-package for bootmode
This should catch installations from our ISO on non-ZFS in uefi mode, which won't get the updated grub efi binary installed upon upgrade, because grub-pc is installed instead of grub-efi-amd64. Adding this to pve7to8 should make this even more visible, than the corresponding patch for promxox-kernel-helper (warnings printed during regular package upgrades might be overlooked more easily than a yellow line in the major upgrade checkscript) The if/else order was chosen to limit the nesting level of the long messages. Signed-off-by: Stoiko Ivanov --- PVE/CLI/pve7to8.pm | 39 +++ 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/PVE/CLI/pve7to8.pm b/PVE/CLI/pve7to8.pm index d1a71eff..2e2304ea 100644 --- a/PVE/CLI/pve7to8.pm +++ b/PVE/CLI/pve7to8.pm @@ -1302,29 +1302,36 @@ sub check_time_sync { sub check_bootloader { log_info("Checking bootloader configuration..."); -if (!$upgraded) { - log_skip("not yet upgraded, no need to check the presence of systemd-boot"); - return; -} -if (! -f "/etc/kernel/proxmox-boot-uuids") { - log_skip("proxmox-boot-tool not used for bootloader configuration"); +if (! -d '/sys/firmware/efi') { + log_skip("System booted in legacy-mode - no need for additional packages"); return; } -if (! -d "/sys/firmware/efi") { - log_skip("System booted in legacy-mode - no need for systemd-boot"); - return; -} - -if ( -f "/usr/share/doc/systemd-boot/changelog.Debian.gz") { - log_pass("systemd-boot is installed"); -} else { +if ( -f "/etc/kernel/proxmox-boot-uuids") { + if (!$upgraded) { + log_skip("not yet upgraded, no need to check the presence of systemd-boot"); + return; + } + if ( -f "/usr/share/doc/systemd-boot/changelog.Debian.gz") { + log_pass("bootloader packages installed correctly"); + return; + } log_warn( "proxmox-boot-tool is used for bootloader configuration in uefi mode" - . "but the separate systemd-boot package, existing in Debian Bookworm is not installed" - . "initializing new ESPs will not work until the package is installed" + . " but the separate systemd-boot package, existing in Debian Bookworm is not installed" + . " initializing new ESPs will not work until the package is installed" + ); + return; +} elsif ( ! -f "/usr/share/doc/grub-efi-amd64/changelog.Debian.gz" ) { + log_warn( + "System booted in uefi mode but grub-efi-amd64 meta-package not installed" + . " new grub versions will not be installed to /boot/efi -" + . " install grub-efi-amd64" ); + return; +} else { + log_pass("bootloader packages installed correctly"); } } -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH 2/2] proxmox-boot-tool: check if correct grub metapackage is installed
this part of the hook applies only to systems not using pbt for bootmangement. Currently our ISO installs grub-pc unconditionally - and never the conflicting grub-efi-amd64. Both packages are responsible for running grub-install (for the appropriate disks) upon an upgrade of grub. This results in grub currently not getting updated on uefi-booted systems (which do not use proxmox-boot-tool). The patch causes a warning to be printed to notify the user. Also considered putting the check+warning in d/postinst - but this way it will get triggered more often (upon every kernel-upgrade/update-initramfs, instead of only on proxmox-kernel-helper updates, which are less often), increasing the chances of being noticed. checking for the changelog-presence was chosen, over `dpkg-query` for the status, for consistency with the similar patch for pve7to8 (and potentially a small speed-gain). Suggested-by: Thomas Lamprecht Signed-off-by: Stoiko Ivanov --- src/proxmox-boot/zz-proxmox-boot | 19 +++ 1 file changed, 19 insertions(+) diff --git a/src/proxmox-boot/zz-proxmox-boot b/src/proxmox-boot/zz-proxmox-boot index 1adc1b1..0d08dbf 100755 --- a/src/proxmox-boot/zz-proxmox-boot +++ b/src/proxmox-boot/zz-proxmox-boot @@ -215,6 +215,23 @@ disable_systemd_boot_hook() { } +check_grub_efi_package() { + + if [ -f "${ESP_LIST}" ]; then + return + fi + + if [ ! -d /sys/firmware/efi ]; then + return + fi + + if [ ! -f /usr/share/doc/grub-efi-amd64/changelog.Debian.gz ]; then + return + fi + warn "uefi-booted system, without grub-efi-amd64 package - /boot/efi will not be updated" + +} + set -- $DEB_MAINT_PARAMS mode="${1#\'}" mode="${mode%\'}" @@ -228,6 +245,7 @@ case $0:$mode in BOOT_KVERS="$(boot_kernel_list "$@")" update_esps disable_systemd_boot_hook + check_grub_efi_package ;; */postrm.d/*:|*/postrm.d/*:remove) reexec_in_mountns "$@" @@ -235,6 +253,7 @@ case $0:$mode in BOOT_KVERS="$(boot_kernel_list)" update_esps disable_systemd_boot_hook + check_grub_efi_package ;; esac -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH 1/2] proxmox-boot-tool: do not exit early in kernel-hook
update_esps is called first in the actual execution below - exiting early does not work for systems that don't use proxmox-boot-tool if a check added later needs to work there too. Signed-off-by: Stoiko Ivanov --- src/proxmox-boot/zz-proxmox-boot | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/proxmox-boot/zz-proxmox-boot b/src/proxmox-boot/zz-proxmox-boot index 793882b..1adc1b1 100755 --- a/src/proxmox-boot/zz-proxmox-boot +++ b/src/proxmox-boot/zz-proxmox-boot @@ -44,7 +44,7 @@ fi update_esps() { if [ ! -f "${ESP_LIST}" ]; then warn "No ${ESP_LIST} found, skipping ESP sync." - exit 0 + return fi if [ -f /etc/kernel/cmdline ]; then # we can have cmdline files with multiple or no new line at all, handle both! -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH kernel-helper/manager] check for fitting grub-meta package on uefi systems
The following patchset is a followup to the one for the installer: https://lists.proxmox.com/pipermail/pve-devel/2023-September/059270.html As suggested by Thomas - adding the check to proxmox-kernel-helper seems like a good idea. While adding it to d/postinst I thought that this might not be the best place - and that getting the warning upon every kernel-upgrade would be better vs. upon every upgrade of proxmox-kernel-helper (which are far less often). (Can gladly send the version with d/postinst as well) If the pve-manager patch gets applied - I'd push the equivalent change to pmg and provide one for pbs. Tested on legacy and uefi VMs installed with pve-8.0 iso and grub-efi-amd64 (and systemd-boot) removed vs. installed. proxmox-kernel-helper: Stoiko Ivanov (2): proxmox-boot-tool: do not exit early in kernel-hook proxmox-boot-tool: check if correct grub metapackage is installed src/proxmox-boot/zz-proxmox-boot | 21 - 1 file changed, 20 insertions(+), 1 deletion(-) pve-manager Stoiko Ivanov (1): pve7to8: check for proper grub meta-package for bootmode PVE/CLI/pve7to8.pm | 39 +++ 1 file changed, 23 insertions(+), 16 deletions(-) -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH installer] install: install correct grub metapackage for the current boot-mode
just realized while talking with Friedrich off-list - if this gets applied it probably would make sense to include it in the pve7to8 (same for pbs and pmg) checks (and also in the upgrade guides) (mostly meant as a note to myself) On Thu, 28 Sep 2023 16:05:33 +0200 Stoiko Ivanov wrote: > grub packages in debian split between: > * meta-packages, which handles (among other things) the reinstalling > grub to the actual device/ESP in case of a version upgrade (grub-pc, > grub-efi-amd64) > * bin-packages, which contain the actual boot-loaders > The bin-packages can coexist on a system, but the meta-package > conflict with each other (didn't check why, but I don't see a hard > conflict on a quick glance) > > Currently our ISO installs grub-pc unconditionally (and both bin > packages, since we install the legacy bootloader also on uefi-booted > systems). This results in uefi-systems not getting a new grub > installed automatically upon upgrade. > > Reported in our community-forum from users who upgraded to PVE 8.0, > and still run into an issue fixed in grub for bookworm: > https://forum.proxmox.com/threads/.123512/ > > Reproduced and analyzed by Friedrich. > > This patch changes the installer, to install the meta-package fitting > for the boot-mode. > > We do not set the debconf variable install_devices, because in my > tests a plain debian installed in uefi mode has this set, and a > `grep -ri install_devices /var/lib/dpkg/info` yields only results with > grub-pc. > > Reported-by: Friedrich Weber > Signed-off-by: Stoiko Ivanov > --- > quickly tested by building an ISO (with the necessary modifications to > ship both packages as .deb) and installing in legacy mode and uefi mode > once. > Proxmox/Install.pm | 6 ++ > 1 file changed, 6 insertions(+) > > diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm > index 1117fc4..d775ac0 100644 > --- a/Proxmox/Install.pm > +++ b/Proxmox/Install.pm > @@ -1057,6 +1057,12 @@ _EOD > chomp; > my $path = $_; > my ($deb) = $path =~ m/${proxmox_pkgdir}\/(.*\.deb)/; > + > + # the grub-pc/grub-efi-amd64 packages (w/o -bin) are the ones > actually updating grub > + # upon upgrade - and conflict with each other - install the fitting > one only > + next if ($deb =~ /grub-pc_/ && $run_env->{boot_type} ne 'bios'); > + next if ($deb =~ /grub-efi-amd64_/ && $run_env->{boot_type} ne > 'efi'); > + > update_progress($count/$pkg_count, 0.5, 0.75, "extracting $deb"); > print STDERR "extracting: $deb\n"; > syscmd("chroot $targetdir dpkg $dpkg_opts --force-depends > --no-triggers --unpack /tmp/pkg/$deb") == 0 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel