Re: [pve-devel] [PATCH storage 3/9] plugin: dir: handle ova files for import
> Dominik Csapak hat am 17.04.2024 16:07 CEST
> geschrieben:
> On 4/17/24 15:52, Fabian Grünbichler wrote:
> > On April 17, 2024 3:10 pm, Dominik Csapak wrote:
> >> On 4/17/24 14:45, Fabian Grünbichler wrote:
> >>> On April 16, 2024 3:18 pm, Dominik Csapak wrote:
> +sub cleanup_extracted_image {
> >>>
> >>> same for this?
> >>>
> +my ($source) = @_;
> +
> +if ($source =~ m|^(/.+/\.tmp_[0-9]+_[0-9]+)/[^/]+$|) {
> +my $tmpdir = $1;
> +
> +unlink $source or $! == ENOENT or die "removing image $source
> failed: $!\n";
> +rmdir $tmpdir or $! == ENOENT or die "removing tmpdir $tmpdir
> failed: $!\n";
> +} else {
> +die "invalid extraced image path '$source'\n";
> >>>
> >>> nit: typo
> >>>
> >>> these are also not discoverable if the error handling in qemu-server
> >>> failed for some reason.. might be a source of unwanted space
> >>> consumption..
> >>
> >> any suggestions for better handling that cleanup?
> >> we could put it at the beginning of each cleanup step, that should
> >> at least make sure we cleaned up the temporary images
> >
> > we could extract them into images/XXX/vm-XXX-disk-.. directly (or
> > rename/move them there after extraction), that way at least they could
> > be cleaned up via the storage API or rescan + delete (and via a regular
> > vdisk_free in qemu-server, instead of requiring a special helper).
> >
> > other than that, I don't think we have an easy way of
> > - exposing them in list & free_image
> > - while ensuring nobody deletes them while the import is still going on
> >(the target VM ownership checks ensure that at least via the UI if we
> >make it an owned volume)
> >
> > it would also allow skipping the conversion if the storage+format
> > already match the target spec as well..
>
> mhmm that could work, but what if the storage does not have
> the 'images' content type enabled? should we simply fail then?
right, that would make it a bit limiting. we could clear the tmpdir on reboots?
;)
it might also be nice (as a follow-up?) to make the tmpdir configurable and/or
see what limitations direct streaming actually has (other than live-import not
working) - because if the OVA is on NFS/.. right now we incur a lot of back and
forth copying..
I wonder if live-import even makes much sense here - if I have to copy/extract
the disks anyway before starting the live-import (which then does another
copy), I can just as well do a regular import and start the VM afterwards,
especially if that saves me one copy action?
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH storage/qemu-server/pve-manager] implement ova/ovf import for directory type storages
> Dominik Csapak hat am 17.04.2024 15:19 CEST > geschrieben: > On 4/17/24 15:11, Fabian Grünbichler wrote: > > On April 16, 2024 3:18 pm, Dominik Csapak wrote: > >> This series enables importing ova/ovf from directory based storages, > >> inclusive upload/download via the webui (ova only). > >> > >> It also improves the ovf importer by parsing the ostype, nics, bootorder > >> (and firmware from vmware exported files). > >> > >> I currently opted to move the OVF.pm to pve-storage, since there is no > >> real other place where we could put it. Building a seperate package > >> from qemu-servers git repo would also not be ideal, since we still > >> have a cyclic dev dependency then > >> (If someone has a better idea how to handle that, please do tell, and > >> i can do that in a v2) > >> > >> There are surely some wrinkles left i did not think of, but all in all, > >> it should be pretty usable. E.g. i downloaded some ovas, uploaded them > >> on my cephfs in my virtual cluster, and successfully imported that with > >> live-import. > >> > >> The biggest caveat when importing from ovas is that we have to > >> temporarily extract the disk images. I opted for doing that into the > >> import storage, but if we have a better idea where to put that, i can > >> implement it in a v2 (or as a follow up). For example, we could add a > >> new 'tmpdir' parameter to the create call and use that for extractig. > > > > something is wrong with the permissions, since the import images are not > > added to check_volume_access, I can now upload an OVA, but not see it > > afterwards ;) > > > > I guess if a user has upload rights for improt images > > (Datastore.AllocateTemplate), they should also be able to see and use > > (and remove) import images? > > > > ah yes, i forgot to add it there. > > but FWICS isos can have the same problem? > upload only requires 'Datastore.AllocateTemplate' but seeing them requires > 'Datastore.AllocateSpace' or 'Datastore.Audit' > > is that a mistake? that's a slightly less problematic variant of a similar issue, yes. Datastore.AllocateSpace and Datastore.Audit are the "weaker cousins" of Datastore.AllocateTemplate, in most configurations if you have the latter you'll also have (one of) the former. IMHO it wouldn't hurt to allow Datastore.AllocateTemplate users access to iso files (and container templates), since they can upload them the behaviour is weird as it is. for the OVA files right now you'd need Datastore.Allocate, which is a higher privilege than those others. I guess treating OVA files like iso and templates w.r.t. ACLs kind of makes sense, even if they have a slightly bigger attack surface behind the scenes. this would also allow just giving trusted admins the option to upload new OVAs, while allowing users to create VMs based on that trusted set. ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH manager v2 0/2] fix #5093 add custom directory and eab to ui
Am 17/04/2024 um 17:55 schrieb Folke Gleumes: > This patch series adds the option to set a custom directory for ACME and > enables the user to use external account binding, which is required by > some providers. > > manager: > > Folke Gleumes (2): > fix #5093: webui: acme: custom directory option > webui: acme: add eab fields > > www/manager6/node/ACME.js | 167 ++ > 1 file changed, 135 insertions(+), 32 deletions(-) > > > Summary over all repositories: > 1 files changed, 135 insertions(+), 32 deletions(-) > applied series, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH manager v3 4/5] ui: fix typo to make pve-cluster-tasks store globally available
Am 12/04/2024 um 16:15 schrieb Friedrich Weber: > This way, it can be used to retrieve the current list of tasks. > > Signed-off-by: Friedrich Weber > --- > > Notes: > changes v2 -> v3: > * no changes > > new in v2: > * moved fix for pve-cluster-tasks store into its own patch > > www/manager6/dc/Tasks.js | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH qemu-server v3 3/5] fix #4474: qemu api: add overrule-shutdown parameter to stop endpoint
Am 12/04/2024 um 16:15 schrieb Friedrich Weber: > The new `overrule-shutdown` parameter is boolean and defaults to 0. If > it is 1, all active `qmshutdown` tasks for the same VM (which are > visible to the user/token) are aborted before attempting to stop the > VM. > > Passing `overrule-shutdown=1` is forbidden for HA resources. > > Signed-off-by: Friedrich Weber > --- > > Notes: > changes v2 -> v3: > - broke over-long lines > - avoid printing empty list of overruled tasks > - removed Wolfgang's Acked-by because patch changed > - rephrased parameter description/commit to reflect changed logic > - adapt to rename to `abort_guest_tasks` > > no changes v1 -> v2 > > PVE/API2/Qemu.pm | 19 ++- > 1 file changed, 18 insertions(+), 1 deletion(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH guest-common v3 1/5] guest helpers: add helper to abort active guest tasks of a certain type
Am 12/04/2024 um 16:15 schrieb Friedrich Weber: > Given a `(type, user, vmid)` tuple, the helper aborts all tasks of the > given `type` for guest `vmid` that `user` is allowed to abort: > > - If `user` has `Sys.Modify` on the node, they can abort any task > - If `user` is an API token, it can abort any task it started itself > - If `user` is a user, they can abort any task started by themselves > or one of their API tokens. > > The helper is used to overrule any active qmshutdown/vzshutdown tasks > when attempting to stop a VM/CT (if requested). > > Signed-off-by: Friedrich Weber > --- > > Notes: > As the computation of `$can_abort_task` essentially duplicates logic > from PVE/API2/Tasks.pm, I considered reusing that, but this would have > required moving it to one of the dependencies of pve-guest-common > (Thomas suggested pve-access-control off-list). Seeing that the logic > boils down to 4 lines in `abort_guest_tasks`, I didn't consider it > worth the trouble in the end. Happy to reconsider, though. > > changes v2 -> v3: > - improved readability: renamed subroutine to describe what it does, > renamed return value, added comment, clarified commit message (thx > Thomas) > - better align logic with current permission model for stopping tasks: > - allow users with Sys.Modify to abort *any* task (thx Thomas) > - allow users to abort tasks of their tokens > > no changes v1 -> v2 > > src/PVE/GuestHelpers.pm | 35 +++ > 1 file changed, 35 insertions(+) > > applied, with some (very) tiny efficiency improvement as follow-up, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH container v3 2/5] fix #4474: lxc api: add overrule-shutdown parameter to stop endpoint
Am 12/04/2024 um 16:15 schrieb Friedrich Weber: > The new `overrule-shutdown` parameter is boolean and defaults to 0. If > it is 1, all active `vzshutdown` tasks for the same CT (which are > visible to the user/token) are aborted before attempting to stop the > CT. > > Passing `overrule-shutdown=1` is forbidden for HA resources. > > Signed-off-by: Friedrich Weber > --- > > Notes: > changes v2 -> v3: > - avoid printing empty list of overruled tasks > - rephrased parameter description/commit to reflect changed logic > - adapt to rename to `abort_guest_tasks` > > changes v1 -> v2: > - move overrule code into worker, as suggested by Wolfgang > - print to worker stdout instead of syslog > > src/PVE/API2/LXC/Status.pm | 19 +++ > 1 file changed, 19 insertions(+) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH manager v2 0/2] fix #5093 add custom directory and eab to ui
This patch series adds the option to set a custom directory for ACME and enables the user to use external account binding, which is required by some providers. manager: Folke Gleumes (2): fix #5093: webui: acme: custom directory option webui: acme: add eab fields www/manager6/node/ACME.js | 167 ++ 1 file changed, 135 insertions(+), 32 deletions(-) Summary over all repositories: 1 files changed, 135 insertions(+), 32 deletions(-) -- Generated by git-murpp 0.7.1 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH manager v2 2/2] webui: acme: add eab fields
Adds fields for eab credentials. By default eab is optional, but if the
directory should report that eab is required, the eab credential fields
are marked as mandatory and prevent the form from being submittable
until credentials are provided.
Signed-off-by: Folke Gleumes
---
www/manager6/node/ACME.js | 28
1 file changed, 28 insertions(+)
diff --git a/www/manager6/node/ACME.js b/www/manager6/node/ACME.js
index d2863a7c..a0db51a6 100644
--- a/www/manager6/node/ACME.js
+++ b/www/manager6/node/ACME.js
@@ -16,6 +16,12 @@ Ext.define('PVE.node.ACMEAccountCreate', {
viewModel: {
data: {
customDirectory: false,
+ eabRequired: false,
+ },
+ formulas: {
+ eabEmptyText: function(get) {
+ return get('eabRequired') ? gettext("required") :
gettext("optional");
+ },
},
},
@@ -124,6 +130,7 @@ Ext.define('PVE.node.ACMEAccountCreate', {
let me = this;
let w = me.up('window');
+ let vm = w.getViewModel();
let disp = w.down('#tos_url_display');
let field = w.down('#tos_url');
let checkbox = w.down('#tos_checkbox');
@@ -151,6 +158,7 @@ Ext.define('PVE.node.ACMEAccountCreate', {
checkbox.setValue(false);
disp.setValue("No terms of service
agreement required");
}
+ vm.set('eabRequired',
!!response.result.data.externalAccountRequired);
},
failure: function(response, opt) {
disp.setValue(undefined);
@@ -185,6 +193,26 @@ Ext.define('PVE.node.ACMEAccountCreate', {
return false;
},
},
+ {
+ xtype: 'proxmoxtextfield',
+ name: 'eab-kid',
+ fieldLabel: gettext('EAB Key ID'),
+ bind: {
+ hidden: '{!customDirectory}',
+ allowBlank: '{!eabRequired}',
+ emptyText: '{eabEmptyText}',
+ },
+ },
+ {
+ xtype: 'proxmoxtextfield',
+ name: 'eab-hmac-key',
+ fieldLabel: gettext('EAB Key'),
+ bind: {
+ hidden: '{!customDirectory}',
+ allowBlank: '{!eabRequired}',
+ emptyText: '{eabEmptyText}',
+ },
+ },
],
clearToSFields: function() {
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH manager v2 1/2] fix #5093: webui: acme: custom directory option
This patch allows the user to set a custom ACME directory by providing
a 'Custom' option in the directory dropdown. This in turn reveals an
input for the url. When using a custom directory the directory has to
be manually queried via button press to prevent from spamming the
directory on every input.
Signed-off-by: Folke Gleumes
---
changes since v1:
* re-add 'allowBlank: false' to disable the clear trigger
www/manager6/node/ACME.js | 139 +-
1 file changed, 107 insertions(+), 32 deletions(-)
diff --git a/www/manager6/node/ACME.js b/www/manager6/node/ACME.js
index 21137b1a..d2863a7c 100644
--- a/www/manager6/node/ACME.js
+++ b/www/manager6/node/ACME.js
@@ -10,6 +10,14 @@ Ext.define('PVE.node.ACMEAccountCreate', {
url: '/cluster/acme/account',
showTaskViewer: true,
defaultExists: false,
+referenceHolder: true,
+onlineHelp: "sysadmin_certs_acme_account",
+
+viewModel: {
+ data: {
+ customDirectory: false,
+ },
+},
items: [
{
@@ -30,12 +38,18 @@ Ext.define('PVE.node.ACMEAccountCreate', {
},
{
xtype: 'proxmoxComboGrid',
- name: 'directory',
+ notFoundIsValid: true,
+ isFormField: false,
allowBlank: false,
valueField: 'url',
displayField: 'name',
fieldLabel: gettext('ACME Directory'),
store: {
+ listeners: {
+ 'load': function() {
+ this.add({ name: gettext("Custom"), url: '' });
+ },
+ },
autoLoad: true,
fields: ['name', 'url'],
idProperty: ['name'],
@@ -43,10 +57,6 @@ Ext.define('PVE.node.ACMEAccountCreate', {
type: 'proxmox',
url: '/api2/json/cluster/acme/directories',
},
- sorters: {
- property: 'name',
- direction: 'ASC',
- },
},
listConfig: {
columns: [
@@ -64,41 +74,93 @@ Ext.define('PVE.node.ACMEAccountCreate', {
},
listeners: {
change: function(combogrid, value) {
- var me = this;
- if (!value) {
- return;
- }
+ let me = this;
+
+ let vm = me.up('window').getViewModel();
+ let dirField =
me.up('window').lookupReference('directoryInput');
+ let tosButton = me.up('window').lookupReference('queryTos');
- var disp = me.up('window').down('#tos_url_display');
- var field = me.up('window').down('#tos_url');
- var checkbox = me.up('window').down('#tos_checkbox');
+ let isCustom = combogrid.getSelection().get('name') ===
gettext("Custom");
+ vm.set('customDirectory', isCustom);
- disp.setValue(gettext('Loading'));
- field.setValue(undefined);
- checkbox.setValue(undefined);
- checkbox.setHidden(true);
+ dirField.setValue(value);
- Proxmox.Utils.API2Request({
- url: '/cluster/acme/meta',
- method: 'GET',
- params: {
- directory: value,
+ if (!isCustom) {
+ tosButton.click();
+ } else {
+ me.up('window').clearToSFields();
+ }
+ },
+ },
+ },
+ {
+ xtype: 'fieldcontainer',
+ layout: 'hbox',
+ fieldLabel: gettext('URL'),
+ bind: {
+ hidden: '{!customDirectory}',
+ },
+ items: [
+ {
+ xtype: 'proxmoxtextfield',
+ name: 'directory',
+ reference: 'directoryInput',
+ flex: 1,
+ allowBlank: false,
+ listeners: {
+ change: function(textbox, value) {
+ let me = this;
+ me.up('window').clearToSFields();
},
- success: function(response, opt) {
- if (response.result.data.termsOfService) {
-
field.setValue(response.result.data.termsOfService);
-
disp.setValue(response.result.data.termsOfService);
- checkbox.setHidden(false);
+ },
+ },
+ {
+ xtype: 'proxmoxButton',
+ margin: '0 0 0 5',
+ reference: 'queryTos',
+ text: gettext('Query URL'),
+ listener
Re: [pve-devel] [PATCH ksm-control-daemon] ksmtuned: use PSS instead of RSZ for caluculating memory usage
Am 11/04/2024 um 12:04 schrieb Roland: > where arcsize is not taken into account > > https://bugzilla.proxmox.com/show_bug.cgi?id=3859 I think this bug should be split, as those are two completely different things implementation wise. The existing one could be kept for RRD, and a new one added for ksmtuned. For the latter it might be simpler to fix, as we do not have to care about upgrading some RRD schema in a cluster, which has a few orders of complexity more compared to checking the ARC on-demand in ksmtuned. ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH ha-manager] d/postinst: make deb-systemd-invoke non-fatal
Am 11/04/2024 um 12:10 schrieb Fabian Grünbichler: > else this can break an upgrade for unrelated reasons. > > this also mimics debhelper behaviour more (which we only not use here because > of lack of reload support) - restructured the snippet to be more similar with > an explicit `if` as well. > > Signed-off-by: Fabian Grünbichler > --- > debian/pve-ha-manager.postinst | 10 ++ > 1 file changed, 6 insertions(+), 4 deletions(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH manager] d/postinst: make deb-systemd-invoke non-fatal
Am 11/04/2024 um 12:10 schrieb Fabian Grünbichler: > else this can break an upgrade for unrelated reasons (regular debhelper also > constructs the restart invocations like this, it even redirects output to > /dev/null) > > Signed-off-by: Fabian Grünbichler > --- > debian/postinst | 12 ++-- > 1 file changed, 6 insertions(+), 6 deletions(-) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH storage] plugin: move definition for 'port' option to base plugin
Am 15/04/2024 um 14:48 schrieb Fiona Ebner:
> Commit 7020491 ("esxi: add 'port' config parameter") started using
> the 'port' option in a second plugin, but the definition stayed in the
> PBS plugin. Avoid the hidden dependency and move the definition to the
> base plugin instead.
>
> It is necessary to mark it as optional or it would be required always.
>
> Signed-off-by: Fiona Ebner
> ---
> src/PVE/Storage/PBSPlugin.pm | 6 --
> src/PVE/Storage/Plugin.pm| 8
> 2 files changed, 8 insertions(+), 6 deletions(-)
>
> diff --git a/src/PVE/Storage/PBSPlugin.pm b/src/PVE/Storage/PBSPlugin.pm
> index 08ceb88..0808bcc 100644
> --- a/src/PVE/Storage/PBSPlugin.pm
> +++ b/src/PVE/Storage/PBSPlugin.pm
> @@ -49,12 +49,6 @@ sub properties {
> description => "Base64-encoded, PEM-formatted public RSA key. Used
> to encrypt a copy of the encryption-key which will be added to each encrypted
> backup.",
> type => 'string',
> },
> - port => {
> - description => "For non default port.",
> - type => 'integer',
> - minimum => 1,
> - maximum => 65535,
> - },
> };
> }
>
> diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
> index 22a9729..5f49830 100644
> --- a/src/PVE/Storage/Plugin.pm
> +++ b/src/PVE/Storage/Plugin.pm
> @@ -205,6 +205,14 @@ my $defaultData = {
> format => 'pve-storage-options',
> optional => 1,
> },
> + port => {
> + description => "For PBS/ESXi, use this port to connect to the
> storage instead of the"
I'd probably avoid hard-coding "PBS/ESXi" here, it would work as good if that
part would be omitted:
"Use this port to connect to the storage instead of the default one."
In the long run we should switch to a per-plugin schema, like the (IIRC)
mappings have, as then we could correctly define defaults and descriptions
without having to be overly general to fit a common denominator.
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH docs] storage: pbs: document port option
Am 15/04/2024 um 14:48 schrieb Fiona Ebner: > Signed-off-by: Fiona Ebner > --- > pve-storage-pbs.adoc | 4 > 1 file changed, 4 insertions(+) > > applied, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH kernel 1/1] cherry-pick improved erratum 1386 workaround
Am 15/04/2024 um 14:56 schrieb Folke Gleumes: > The original fix disabled the xsaves feature for zen1/2. The issue has > since been fixed in the cpus microcode and this patch keeps the feature > enabled > if the microcode version is recent enough to contain the fix. > > Signed-off-by: Folke Gleumes > --- > > Tested this on an AMD Epyc 7302P v2. > > ...-improve-the-erratum-1386-workaround.patch | 82 +++ > 1 file changed, 82 insertions(+) > create mode 100644 > patches/kernel/0013-improve-the-erratum-1386-workaround.patch > > applied both patches, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 container] fix #4846: Avoid the outdated noacl mount option on ext4
On 11/04/2024 15:44, Fabian Grünbichler wrote: I am not sure this is correct.. or rather, wouldn't it be simpler to say if $storage && $format eq 'raw' => no noacl ? if we get complains that somebody did something non-standard (i.e., manually formatted a raw volume using a different filesystem), we can always think about adding support for that (e.g., via some "fs=XX" property on the mountpoint that allows us to handle it here, although I am not even sure if we*want* to support that ;)). yeah, I simplified it in patch v3: https://lists.proxmox.com/pipermail/pve-devel/2024-April/063227.html ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v3 container] fix #4846: Avoid the outdated noacl mount option on ext4
Do not use the 'noacl' mount option when mounting a container disk with
an ext4 file system. The option was removed from the kernel in commit
2d544ec923db
Signed-off-by: Filip Schauer
---
Changes since v3:
* Simplify ext4 detection
* Do not add noacl if $acl is undefined
src/PVE/LXC.pm | 16 ++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm
index e688ea6..394ffb8 100644
--- a/src/PVE/LXC.pm
+++ b/src/PVE/LXC.pm
@@ -1825,8 +1825,20 @@ sub __mountpoint_mount {
}
my $acl = $mountpoint->{acl};
-if (defined($acl)) {
- push @$optlist, ($acl ? 'acl' : 'noacl');
+
+if ($acl) {
+ push @$optlist, 'acl';
+} elsif (defined($acl)) {
+ my $noacl = 1;
+
+ if ($storage) {
+ my (undef, undef, undef, undef, undef, undef, $format) =
+ PVE::Storage::parse_volname($storage_cfg, $volid);
+
+ $noacl = 0 if $format eq 'raw';
+ }
+
+ push @$optlist, 'noacl' if $noacl;
}
my $optstring = join(',', @$optlist);
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager 1/2] fix #5093: webui: acme: custom directory option
On 1/16/24 15:33, Folke Gleumes wrote:
> This patch allows the user to set a custom ACME directory by providing
> a 'Custom' option in the directory dropdown. This in turn reveals an
> input for the url. When using a custom directory the directory has to
> be manually queried via button press to prevent from spamming the
> directory on every input.
>
> Signed-off-by: Folke Gleumes
> ---
> www/manager6/node/ACME.js | 140 +-
> 1 file changed, 107 insertions(+), 33 deletions(-)
>
> diff --git a/www/manager6/node/ACME.js b/www/manager6/node/ACME.js
> index 21137b1a..5b71778a 100644
> --- a/www/manager6/node/ACME.js
> +++ b/www/manager6/node/ACME.js
> @@ -10,6 +10,14 @@ Ext.define('PVE.node.ACMEAccountCreate', {
> url: '/cluster/acme/account',
> showTaskViewer: true,
> defaultExists: false,
> +referenceHolder: true,
> +onlineHelp: "sysadmin_certs_acme_account",
> +
> +viewModel: {
> + data: {
> + customDirectory: false,
> + },
> +},
>
> items: [
> {
> @@ -30,12 +38,17 @@ Ext.define('PVE.node.ACMEAccountCreate', {
> },
> {
> xtype: 'proxmoxComboGrid',
> - name: 'directory',
> - allowBlank: false,
> + notFoundIsValid: true,
> + isFormField: false,
> valueField: 'url',
> displayField: 'name',
> fieldLabel: gettext('ACME Directory'),
> store: {
> + listeners: {
> + 'load': function() {
> + this.add({ name: gettext("Custom"), url: '' });
> + },
> + },
> autoLoad: true,
> fields: ['name', 'url'],
> idProperty: ['name'],
> @@ -43,10 +56,6 @@ Ext.define('PVE.node.ACMEAccountCreate', {
> type: 'proxmox',
> url: '/api2/json/cluster/acme/directories',
> },
> - sorters: {
> - property: 'name',
> - direction: 'ASC',
> - },
> },
> listConfig: {
> columns: [
> @@ -64,41 +73,93 @@ Ext.define('PVE.node.ACMEAccountCreate', {
> },
> listeners: {
> change: function(combogrid, value) {
> - var me = this;
> - if (!value) {
> - return;
> - }
> + let me = this;
>
> - var disp = me.up('window').down('#tos_url_display');
> - var field = me.up('window').down('#tos_url');
> - var checkbox = me.up('window').down('#tos_checkbox');
> + let vm = me.up('window').getViewModel();
> + let dirField =
> me.up('window').lookupReference('directoryInput');
> + let tosButton = me.up('window').lookupReference('queryTos');
>
> - disp.setValue(gettext('Loading'));
> - field.setValue(undefined);
> - checkbox.setValue(undefined);
> - checkbox.setHidden(true);
> + let isCustom = combogrid.getSelection().get('name') ===
> gettext("Custom");
> + vm.set('customDirectory', isCustom);
>
> - Proxmox.Utils.API2Request({
> - url: '/cluster/acme/meta',
> - method: 'GET',
> - params: {
> - directory: value,
> + dirField.setValue(value);
> +
> + if (!isCustom) {
> + tosButton.click();
> + } else {
> + me.up('window').clearToSFields();
> + }
> + },
> + },
> + },
> + {
> + xtype: 'fieldcontainer',
> + layout: 'hbox',
> + fieldLabel: gettext('URL'),
> + bind: {
> + hidden: '{!customDirectory}',
> + },
> + items: [
> + {
> + xtype: 'proxmoxtextfield',
> + name: 'directory',
> + reference: 'directoryInput',
> + flex: 1,
> + allowBlank: false,
> + listeners: {
> + change: function(textbox, value) {
> + let me = this;
> + me.up('window').clearToSFields();
> },
> - success: function(response, opt) {
> - if (response.result.data.termsOfService) {
> -
> field.setValue(response.result.data.termsOfService);
> -
> disp.setValue(response.result.data.termsOfService);
> - checkbox.setHidden(false);
> + },
> + },
> + {
> + xtype: 'proxmoxButton',
> + margin: '0 0 0 5',
> + reference: 'queryTos',
> + text: gettext('Qu
Re: [pve-devel] [PATCH storage 3/9] plugin: dir: handle ova files for import
On 4/17/24 15:52, Fabian Grünbichler wrote:
On April 17, 2024 3:10 pm, Dominik Csapak wrote:
On 4/17/24 14:45, Fabian Grünbichler wrote:
On April 16, 2024 3:18 pm, Dominik Csapak wrote:
+sub cleanup_extracted_image {
same for this?
+my ($source) = @_;
+
+if ($source =~ m|^(/.+/\.tmp_[0-9]+_[0-9]+)/[^/]+$|) {
+ my $tmpdir = $1;
+
+ unlink $source or $! == ENOENT or die "removing image $source failed:
$!\n";
+ rmdir $tmpdir or $! == ENOENT or die "removing tmpdir $tmpdir failed:
$!\n";
+} else {
+ die "invalid extraced image path '$source'\n";
nit: typo
these are also not discoverable if the error handling in qemu-server
failed for some reason.. might be a source of unwanted space
consumption..
any suggestions for better handling that cleanup?
we could put it at the beginning of each cleanup step, that should
at least make sure we cleaned up the temporary images
we could extract them into images/XXX/vm-XXX-disk-.. directly (or
rename/move them there after extraction), that way at least they could
be cleaned up via the storage API or rescan + delete (and via a regular
vdisk_free in qemu-server, instead of requiring a special helper).
other than that, I don't think we have an easy way of
- exposing them in list & free_image
- while ensuring nobody deletes them while the import is still going on
(the target VM ownership checks ensure that at least via the UI if we
make it an owned volume)
it would also allow skipping the conversion if the storage+format
already match the target spec as well..
mhmm that could work, but what if the storage does not have
the 'images' content type enabled? should we simply fail then?
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied-series: [PATCH-SERIES v4 manager/docs] close #4513: add advanced tab for backup jobs and improve performance fallback/default
Am 16/04/2024 um 14:09 schrieb Fiona Ebner: > Changes in v4 (Thanks to Thomas for feedback!): > * rename tab from 'Performance' to 'Advanced' > * move repeat-missed setting there too > * update docs to clarify that those settings can be found in the > advanced tab > > Changes in v3 (Thanks to Thomas for feedback!): > * new patch to actually honor default values for performance > format/schema > * new patch to switch to per-property fallback for performance > properties > * also handle new (came in after v2) pbs-entries-max in UI (make > sure to preserve value, but don't expose it) > * drop already applied patch > > Improve fallback for the 'performance' sub-properties by using a > per-property fallback and honor schema defaults. > > Expose commonly used advanced properties in the backup job UI under a > new tab. > > > manager: > > Fiona Ebner (5): > vzdump: actually honor schema defaults for performance > vzdump: use per-property fallback for performance settings > close #4513: ui: backup job: add tab for advanced options > ui: backup job: disable zstd thread count field when zstd isn't used > ui: backup job: move repeat-missed option to advanced tab > > PVE/VZDump.pm | 33 +++- > www/manager6/Makefile | 1 + > www/manager6/dc/Backup.js | 42 +++-- > www/manager6/panel/BackupAdvancedOptions.js | 169 > 4 files changed, 228 insertions(+), 17 deletions(-) > create mode 100644 www/manager6/panel/BackupAdvancedOptions.js > > > docs: > > Fiona Ebner (2): > backup: update information about performance settings > backup: clarify where repeat-missed option can be found now > > vzdump.adoc | 15 --- > 1 file changed, 8 insertions(+), 7 deletions(-) > applied series, with some slight rework of the empty texts and hints done as follow-up, thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH pve-container v2 36/39] firewall: add handling for new nft firewall
When the nftables firewall is enabled, we do not need to create
firewall bridges.
Signed-off-by: Stefan Hanreich
---
src/PVE/LXC.pm | 5 +
1 file changed, 5 insertions(+)
diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm
index e688ea6..85800ea 100644
--- a/src/PVE/LXC.pm
+++ b/src/PVE/LXC.pm
@@ -18,6 +18,7 @@ use PVE::AccessControl;
use PVE::CGroup;
use PVE::CpuSet;
use PVE::Exception qw(raise_perm_exc);
+use PVE::Firewall;
use PVE::GuestHelpers qw(check_vnet_access safe_string_ne safe_num_ne
safe_boolean_ne);
use PVE::INotify;
use PVE::JSONSchema qw(get_standard_option);
@@ -949,6 +950,10 @@ sub net_tap_plug : prototype($$) {
my ($bridge, $tag, $firewall, $trunks, $rate, $hwaddr) =
$net->@{'bridge', 'tag', 'firewall', 'trunks', 'rate', 'hwaddr'};
+my $cluster_fw_conf = PVE::Firewall::load_clusterfw_conf();
+my $host_fw_conf = PVE::Firewall::load_hostfw_conf($cluster_fw_conf);
+$firewall = $net->{firewall} && !($host_fw_conf->{options}->{nftables} //
0);
+
if ($have_sdn) {
PVE::Network::SDN::Zones::tap_plug($iface, $bridge, $tag, $firewall,
$trunks, $rate);
PVE::Network::SDN::Zones::add_bridge_fdb($iface, $hwaddr, $bridge);
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-firewall v2 16/39] config: firewall: add conntrack helper types
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/resources/ct_helper.json | 52 +
proxmox-ve-config/src/firewall/ct_helper.rs | 115
proxmox-ve-config/src/firewall/mod.rs | 1 +
3 files changed, 168 insertions(+)
create mode 100644 proxmox-ve-config/resources/ct_helper.json
create mode 100644 proxmox-ve-config/src/firewall/ct_helper.rs
diff --git a/proxmox-ve-config/resources/ct_helper.json
b/proxmox-ve-config/resources/ct_helper.json
new file mode 100644
index 000..5e70a3a
--- /dev/null
+++ b/proxmox-ve-config/resources/ct_helper.json
@@ -0,0 +1,52 @@
+[
+ {
+"name": "amanda",
+"v4": true,
+"v6": true,
+"udp": 10080
+ },
+ {
+"name": "ftp",
+"v4": true,
+"v6": true,
+"tcp": 21
+ } ,
+ {
+"name": "irc",
+"v4": true,
+"tcp": 6667
+ },
+ {
+"name": "netbios-ns",
+"v4": true,
+"udp": 137
+ },
+ {
+"name": "pptp",
+"v4": true,
+"tcp": 1723
+ },
+ {
+"name": "sane",
+"v4": true,
+"v6": true,
+"tcp": 6566
+ },
+ {
+"name": "sip",
+"v4": true,
+"v6": true,
+"udp": 5060
+ },
+ {
+"name": "snmp",
+"v4": true,
+"udp": 161
+ },
+ {
+"name": "tftp",
+"v4": true,
+"v6": true,
+"udp": 69
+ }
+]
diff --git a/proxmox-ve-config/src/firewall/ct_helper.rs
b/proxmox-ve-config/src/firewall/ct_helper.rs
new file mode 100644
index 000..40e4fee
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/ct_helper.rs
@@ -0,0 +1,115 @@
+use anyhow::{bail, Error};
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::sync::OnceLock;
+
+use crate::firewall::types::address::Family;
+use crate::firewall::types::rule_match::{Ports, Protocol, Tcp, Udp};
+
+#[derive(Clone, Debug, Deserialize)]
+pub struct CtHelperMacroJson {
+pub v4: Option,
+pub v6: Option,
+pub name: String,
+pub tcp: Option,
+pub udp: Option,
+}
+
+impl TryFrom for CtHelperMacro {
+type Error = Error;
+
+fn try_from(value: CtHelperMacroJson) -> Result {
+if value.tcp.is_none() && value.udp.is_none() {
+bail!("Neither TCP nor UDP port set in CT helper!");
+}
+
+let family = match (value.v4, value.v6) {
+(Some(true), Some(true)) => None,
+(Some(true), _) => Some(Family::V4),
+(_, Some(true)) => Some(Family::V6),
+_ => bail!("Neither v4 nor v6 set in CT Helper Macro!"),
+};
+
+let mut ct_helper = CtHelperMacro {
+family,
+name: value.name,
+tcp: None,
+udp: None,
+};
+
+if let Some(dport) = value.tcp {
+let ports = Ports::from_u16(None, dport);
+ct_helper.tcp = Some(Tcp::new(ports).into());
+}
+
+if let Some(dport) = value.udp {
+let ports = Ports::from_u16(None, dport);
+ct_helper.udp = Some(Udp::new(ports).into());
+}
+
+Ok(ct_helper)
+}
+}
+
+#[derive(Clone, Debug, Deserialize)]
+#[serde(try_from = "CtHelperMacroJson")]
+pub struct CtHelperMacro {
+family: Option,
+name: String,
+tcp: Option,
+udp: Option,
+}
+
+impl CtHelperMacro {
+fn helper_name(&self, protocol: &str) -> String {
+format!("helper-{}-{protocol}", self.name)
+}
+
+pub fn tcp_helper_name(&self) -> String {
+self.helper_name("tcp")
+}
+
+pub fn udp_helper_name(&self) -> String {
+self.helper_name("udp")
+}
+
+pub fn family(&self) -> Option {
+self.family
+}
+
+pub fn name(&self) -> &str {
+self.name.as_ref()
+}
+
+pub fn tcp(&self) -> Option<&Protocol> {
+self.tcp.as_ref()
+}
+
+pub fn udp(&self) -> Option<&Protocol> {
+self.udp.as_ref()
+}
+}
+
+fn hashmap() -> &'static HashMap {
+const MACROS: &str = include_str!("../../resources/ct_helper.json");
+static HASHMAP: OnceLock> = OnceLock::new();
+
+HASHMAP.get_or_init(|| {
+let macro_data: Vec = match
serde_json::from_str(MACROS) {
+Ok(data) => data,
+Err(err) => {
+log::error!("could not load data for ct helpers: {err}");
+Vec::new()
+}
+};
+
+macro_data
+.into_iter()
+.map(|elem| (elem.name.clone(), elem))
+.collect()
+})
+}
+
+pub fn get_cthelper(name: &str) -> Option<&'static CtHelperMacro> {
+hashmap().get(name)
+}
diff --git a/proxmox-ve-config/src/firewall/mod.rs
b/proxmox-ve-config/src/firewall/mod.rs
index 0f438ca..2cf57e2 100644
--- a/proxmox-ve-config/src/firewall/mod.rs
+++ b/proxmox-ve-config/src/firewall/mod.rs
@@ -1,5 +1,6 @@
pub mod cluster;
pub mod common;
+pub mod ct_helper;
pub mod fw_macros;
pub mod guest;
pub mod host;
--
2.39.2
___
pve-devel mailin
[pve-devel] [PATCH proxmox-firewall v2 33/39] firewall: add files for debian packaging
Suggested-By: Fabian Grünbichler
Signed-off-by: Stefan Hanreich
---
.gitignore | 3 ++
Makefile| 70 +
debian/changelog| 5 +++
debian/control | 38 ++
debian/copyright| 16
debian/postrm | 14 +++
debian/proxmox-firewall.install | 1 +
debian/proxmox-firewall.service | 9 +
debian/proxmox-firewall.timer | 13 ++
debian/rules| 32 +++
debian/source/format| 1 +
defines.mk | 13 ++
12 files changed, 215 insertions(+)
create mode 100644 Makefile
create mode 100644 debian/changelog
create mode 100644 debian/control
create mode 100644 debian/copyright
create mode 100755 debian/postrm
create mode 100644 debian/proxmox-firewall.install
create mode 100644 debian/proxmox-firewall.service
create mode 100644 debian/proxmox-firewall.timer
create mode 100755 debian/rules
create mode 100644 debian/source/format
create mode 100644 defines.mk
diff --git a/.gitignore b/.gitignore
index 3cb8114..90749ee 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,5 +2,8 @@
/Cargo.lock
proxmox-firewall-*/
*.deb
+*.dsc
+*.tar*
+*.build
*.buildinfo
*.changes
diff --git a/Makefile b/Makefile
new file mode 100644
index 000..c235b93
--- /dev/null
+++ b/Makefile
@@ -0,0 +1,70 @@
+include /usr/share/dpkg/pkg-info.mk
+include /usr/share/dpkg/architecture.mk
+include defines.mk
+
+PACKAGE=proxmox-firewall
+BUILDDIR ?= $(PACKAGE)-$(DEB_VERSION_UPSTREAM)
+CARGO ?= cargo
+
+DEB=$(PACKAGE)_$(DEB_VERSION_UPSTREAM_REVISION)_$(DEB_HOST_ARCH).deb
+DBG_DEB=$(PACKAGE)-dbgsym_$(DEB_VERSION_UPSTREAM_REVISION)_$(DEB_HOST_ARCH).deb
+DSC=rust-$(PACKAGE)_$(DEB_VERSION_UPSTREAM_REVISION).dsc
+
+DEBS = $(DEB) $(DBG_DEB)
+
+ifeq ($(BUILD_MODE), release)
+CARGO_BUILD_ARGS += --release
+COMPILEDIR := target/release
+else
+COMPILEDIR := target/debug
+endif
+
+
+all: cargo-build
+
+.PHONY: cargo-build
+cargo-build:
+ $(CARGO) build $(CARGO_BUILD_ARGS)
+
+.PHONY: build
+build: $(BUILDDIR)
+$(BUILDDIR):
+ rm -rf $@ $@.tmp; mkdir $@.tmp
+ cp -a proxmox-firewall proxmox-nftables proxmox-ve-config debian
Cargo.toml Makefile defines.mk $@.tmp/
+ mv $@.tmp $@
+
+.PHONY: deb
+deb: $(DEB)
+$(HELPER_DEB) $(DBG_DEB) $(HELPER_DBG_DEB) $(DOC_DEB): $(DEB)
+$(DEB): $(BUILDDIR)
+ cd $(BUILDDIR); dpkg-buildpackage -b -us -uc --no-pre-clean
+ lintian $(DEB) $(DOC_DEB) $(HELPER_DEB)
+
+.PHONY: test
+test:
+ $(CARGO) test
+
+.PHONY: dsc
+dsc:
+ rm -rf $(BUILDDIR) $(DSC)
+ $(MAKE) $(DSC)
+ lintian $(DSC)
+$(DSC): $(BUILDDIR)
+ cd $(BUILDDIR); dpkg-buildpackage -S -us -uc -d -nc
+
+sbuild: $(DSC)
+ sbuild $<
+
+.PHONY: dinstall
+dinstall: $(DEB)
+ dpkg -i $(DEB) $(DBG_DEB) $(DOC_DEB)
+
+.PHONY: distclean
+distclean: clean
+
+.PHONY: clean
+clean:
+ $(CARGO) clean
+ rm -f *.deb *.build *.buildinfo *.changes *.dsc rust-$(PACKAGE)*.tar*
+ rm -rf $(PACKAGE)-[0-9]*/
+ find . -name '*~' -exec rm {} ';'
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 000..3ca5833
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+rust-proxmox-firewall (0.1) UNRELEASED; urgency=medium
+
+ * Initial release.
+
+ -- Stefan Hanreich Thu, 07 Mar 2024 10:15:10 +0100
diff --git a/debian/control b/debian/control
new file mode 100644
index 000..fe9467b
--- /dev/null
+++ b/debian/control
@@ -0,0 +1,38 @@
+Source: rust-proxmox-firewall
+Section: admin
+Priority: optional
+Maintainer: Proxmox Support Team
+Build-Depends: cargo:native,
+ debhelper-compat (= 13),
+ libnftables-dev,
+ librust-anyhow-1+default-dev,
+ librust-env-logger-0.10+default-dev,
+ librust-log-0.4+default-dev (>= 0.4.17-~~),
+ librust-nix-0.26+default-dev (>= 0.26.1-~~),
+ librust-proxmox-sys-dev,
+ librust-proxmox-sortable-macro-dev,
+ librust-serde-1+default-dev,
+ librust-serde-1+derive-dev,
+ librust-serde-json-1+default-dev,
+ librust-serde-plain-1+default-dev,
+ librust-serde-plain-1+default-dev,
+ librust-serde-with+default-dev,
+ librust-libc-0.2+default-dev,
+ librust-proxmox-schema-3+default-dev,
+ libstd-rust-dev,
+ netbase,
+ python3,
+ rustc:native,
+Standards-Version: 4.6.2
+Homepage: https://www.proxmox.com
+
+Package: proxmox-firewall
+Architecture: any
+Conflicts: ulogd,
+Depends: ${misc:Depends}, ${shlibs:Depends},
+ pve-firewall,
+ nftables,
+ netbase,
+Description: Proxmox nftables firewall
+ This package contains a nftables-based implementation of the Proxmox VE
+ Firewall
diff --git a/debian/copyright b/debian/copyrig
[pve-devel] [PATCH proxmox-firewall v2 19/39] nftables: expression: add types
Adds an enum containing most of the expressions defined in the
nftables-json schema [1].
[1]
https://manpages.debian.org/bookworm/libnftables1/libnftables-json.5.en.html#EXPRESSIONS
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-nftables/Cargo.toml| 2 +-
proxmox-nftables/src/expression.rs | 268 +
proxmox-nftables/src/lib.rs| 4 +
proxmox-nftables/src/types.rs | 53 ++
4 files changed, 326 insertions(+), 1 deletion(-)
create mode 100644 proxmox-nftables/src/expression.rs
create mode 100644 proxmox-nftables/src/types.rs
diff --git a/proxmox-nftables/Cargo.toml b/proxmox-nftables/Cargo.toml
index ebece9d..909869b 100644
--- a/proxmox-nftables/Cargo.toml
+++ b/proxmox-nftables/Cargo.toml
@@ -17,4 +17,4 @@ serde = { version = "1", features = [ "derive" ] }
serde_json = "1"
serde_plain = "1"
-proxmox-ve-config = { path = "../proxmox-ve-config", optional = true }
+proxmox-ve-config = { path = "../proxmox-ve-config" }
diff --git a/proxmox-nftables/src/expression.rs
b/proxmox-nftables/src/expression.rs
new file mode 100644
index 000..5478291
--- /dev/null
+++ b/proxmox-nftables/src/expression.rs
@@ -0,0 +1,268 @@
+use crate::types::{ElemConfig, Verdict};
+use serde::{Deserialize, Serialize};
+use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
+
+use crate::helper::NfVec;
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum Expression {
+Concat(Vec),
+Set(Vec),
+Range(Box<(Expression, Expression)>),
+Map(Box),
+Prefix(Prefix),
+Payload(Payload),
+Meta(Meta),
+Ct(Ct),
+Elem(Box),
+
+#[serde(rename = "|")]
+Or(Box<(Expression, Expression)>),
+#[serde(rename = "&")]
+And(Box<(Expression, Expression)>),
+#[serde(rename = "^")]
+Xor(Box<(Expression, Expression)>),
+#[serde(rename = "<<")]
+ShiftLeft(Box<(Expression, Expression)>),
+#[serde(rename = ">>")]
+ShiftRight(Box<(Expression, Expression)>),
+
+#[serde(untagged)]
+List(Vec),
+
+#[serde(untagged)]
+Verdict(Verdict),
+
+#[serde(untagged)]
+Bool(bool),
+#[serde(untagged)]
+Number(i64),
+#[serde(untagged)]
+String(String),
+}
+
+impl Expression {
+pub fn set(expressions: impl IntoIterator) -> Self {
+Expression::Set(Vec::from_iter(expressions))
+}
+
+pub fn concat(expressions: impl IntoIterator) -> Self {
+Expression::Concat(Vec::from_iter(expressions))
+}
+}
+
+impl From for Expression {
+#[inline]
+fn from(v: bool) -> Self {
+Expression::Bool(v)
+}
+}
+
+impl From for Expression {
+#[inline]
+fn from(v: i64) -> Self {
+Expression::Number(v)
+}
+}
+
+impl From for Expression {
+#[inline]
+fn from(v: u16) -> Self {
+Expression::Number(v.into())
+}
+}
+
+impl From for Expression {
+#[inline]
+fn from(v: u8) -> Self {
+Expression::Number(v.into())
+}
+}
+
+impl From<&str> for Expression {
+#[inline]
+fn from(v: &str) -> Self {
+Expression::String(v.to_string())
+}
+}
+
+impl From for Expression {
+#[inline]
+fn from(v: String) -> Self {
+Expression::String(v)
+}
+}
+
+impl From for Expression {
+#[inline]
+fn from(meta: Meta) -> Self {
+Expression::Meta(meta)
+}
+}
+
+impl From for Expression {
+#[inline]
+fn from(ct: Ct) -> Self {
+Expression::Ct(ct)
+}
+}
+
+impl From for Expression {
+#[inline]
+fn from(payload: Payload) -> Self {
+Expression::Payload(payload)
+}
+}
+
+impl From for Expression {
+#[inline]
+fn from(prefix: Prefix) -> Self {
+Expression::Prefix(prefix)
+}
+}
+
+impl From for Expression {
+#[inline]
+fn from(value: Verdict) -> Self {
+Expression::Verdict(value)
+}
+}
+
+impl From<&IpAddr> for Expression {
+fn from(value: &IpAddr) -> Self {
+Expression::String(value.to_string())
+}
+}
+
+impl From<&Ipv6Addr> for Expression {
+fn from(address: &Ipv6Addr) -> Self {
+Expression::String(address.to_string())
+}
+}
+
+impl From<&Ipv4Addr> for Expression {
+fn from(address: &Ipv4Addr) -> Self {
+Expression::String(address.to_string())
+}
+}
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum IpFamily {
+Ip,
+Ip6,
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub struct Meta {
+key: String,
+}
+
+impl Meta {
+pub fn new(key: impl Into) -> Self {
+Self { key: key.into() }
+}
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub struct Map {
+key: Expression,
+data: Expression,
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub struct Ct {
+key: String,
+#[serde(skip_serializing_if = "Option::is_none")]
+family: Option,
+#[
[pve-devel] [PATCH proxmox-firewall v2 29/39] firewall: add rule generation logic
ToNftRules is basically a conversion trait for firewall config structs
to convert them into the respective nftables statements.
We are passing a list of rules to the method, which then modifies the
list of rules such that all relevant rules in the list have statements
appended that apply the configured constraints from the firewall
config.
This is particularly relevant for the rule generation logic for
ipsets. Due to how sets work in nftables we need to generate two rules
for every ipset: a rule for the v4 ipset and a rule for the v6 ipset.
This is because sets can only contain either v4 or v6 addresses. By
passing a list of all generated rules we can duplicate all rules and
then add a statement for the v4 or v6 set respectively.
This also enables us to start with multiple rules, which is required
for using log statements in conjunction with limit statements.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-firewall/src/main.rs | 1 +
proxmox-firewall/src/rule.rs | 761 +
proxmox-nftables/src/expression.rs | 4 +
3 files changed, 766 insertions(+)
create mode 100644 proxmox-firewall/src/rule.rs
diff --git a/proxmox-firewall/src/main.rs b/proxmox-firewall/src/main.rs
index 656ac15..ae832e3 100644
--- a/proxmox-firewall/src/main.rs
+++ b/proxmox-firewall/src/main.rs
@@ -1,6 +1,7 @@
use anyhow::Error;
mod config;
+mod rule;
fn main() -> Result<(), Error> {
env_logger::init();
diff --git a/proxmox-firewall/src/rule.rs b/proxmox-firewall/src/rule.rs
new file mode 100644
index 000..c8099d0
--- /dev/null
+++ b/proxmox-firewall/src/rule.rs
@@ -0,0 +1,761 @@
+use std::ops::{Deref, DerefMut};
+
+use anyhow::{format_err, Error};
+use proxmox_nftables::{
+expression::{Ct, IpFamily, Meta, Payload, Prefix},
+statement::{Log, LogLevel, Match, Operator},
+types::{AddRule, ChainPart, SetName},
+Expression, Statement,
+};
+use proxmox_ve_config::{
+firewall::{
+ct_helper::CtHelperMacro,
+fw_macros::{get_macro, FwMacro},
+types::{
+address::Family,
+alias::AliasName,
+ipset::{Ipfilter, IpsetName},
+log::LogRateLimit,
+rule::{Direction, Kind, RuleGroup},
+rule_match::{
+Icmp, Icmpv6, IpAddrMatch, IpMatch, Ports, Protocol,
RuleMatch, Sctp, Tcp, Udp,
+},
+Alias, Rule,
+},
+},
+guest::types::Vmid,
+};
+
+use crate::config::FirewallConfig;
+
+#[derive(Debug, Clone)]
+pub(crate) struct NftRule {
+family: Option,
+statements: Vec,
+terminal_statements: Vec,
+}
+
+impl NftRule {
+pub fn from_terminal_statements(terminal_statements: Vec) ->
Self {
+Self {
+family: None,
+statements: Vec::new(),
+terminal_statements,
+}
+}
+
+pub fn new(terminal_statement: Statement) -> Self {
+Self {
+family: None,
+statements: Vec::new(),
+terminal_statements: vec![terminal_statement],
+}
+}
+
+pub fn from_config_rule(rule: &Rule, env: &NftRuleEnv) ->
Result, Error> {
+let mut rules = Vec::new();
+
+if rule.disabled() {
+return Ok(rules);
+}
+
+rule.to_nft_rules(&mut rules, env)?;
+
+Ok(rules)
+}
+
+pub fn from_ct_helper(
+ct_helper: &CtHelperMacro,
+env: &NftRuleEnv,
+) -> Result, Error> {
+let mut rules = Vec::new();
+ct_helper.to_nft_rules(&mut rules, env)?;
+Ok(rules)
+}
+
+pub fn from_ipfilter(ipfilter: &Ipfilter, env: &NftRuleEnv) ->
Result, Error> {
+let mut rules = Vec::new();
+ipfilter.to_nft_rules(&mut rules, env)?;
+Ok(rules)
+}
+}
+
+impl Deref for NftRule {
+type Target = Vec;
+
+fn deref(&self) -> &Self::Target {
+&self.statements
+}
+}
+
+impl DerefMut for NftRule {
+fn deref_mut(&mut self) -> &mut Self::Target {
+&mut self.statements
+}
+}
+
+impl NftRule {
+pub fn into_add_rule(self, chain: ChainPart) -> AddRule {
+let statements =
self.statements.into_iter().chain(self.terminal_statements);
+
+AddRule::from_statements(chain, statements)
+}
+
+pub fn family(&self) -> Option {
+self.family
+}
+
+pub fn set_family(&mut self, family: Family) {
+self.family = Some(family);
+}
+}
+
+pub(crate) struct NftRuleEnv<'a> {
+pub(crate) chain: ChainPart,
+pub(crate) direction: Direction,
+pub(crate) firewall_config: &'a FirewallConfig,
+pub(crate) vmid: Option,
+}
+
+impl NftRuleEnv<'_> {
+fn alias(&self, name: &AliasName) -> Option<&Alias> {
+self.firewall_config.alias(name, self.vmid)
+}
+
+fn iface_name(&self, rule_iface: &str) -> String {
+match &self.vmid {
+Some(vmid) => {
+if let Some(c
[pve-devel] [PATCH proxmox-firewall v2 28/39] firewall: add config loader
We load the firewall configuration from the default paths, as well as
only the guest configurations that are local to the node itself. In
the future we could change this to use pmxcfs directly instead.
We also load information from nftables directly about dynamically
created chains (mostly chains for the guest firewall).
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-firewall/Cargo.toml| 2 +
proxmox-firewall/src/config.rs | 281 +
proxmox-firewall/src/main.rs | 3 +
3 files changed, 286 insertions(+)
create mode 100644 proxmox-firewall/src/config.rs
diff --git a/proxmox-firewall/Cargo.toml b/proxmox-firewall/Cargo.toml
index b59d973..431e71a 100644
--- a/proxmox-firewall/Cargo.toml
+++ b/proxmox-firewall/Cargo.toml
@@ -11,6 +11,8 @@ description = "Proxmox VE nftables firewall implementation"
license = "AGPL-3"
[dependencies]
+log = "0.4"
+env_logger = "0.10"
anyhow = "1"
proxmox-nftables = { path = "../proxmox-nftables", features = ["config-ext"] }
diff --git a/proxmox-firewall/src/config.rs b/proxmox-firewall/src/config.rs
new file mode 100644
index 000..f5df20f
--- /dev/null
+++ b/proxmox-firewall/src/config.rs
@@ -0,0 +1,281 @@
+use std::collections::BTreeMap;
+use std::default::Default;
+use std::fs::File;
+use std::io::{self, BufReader};
+use std::sync::OnceLock;
+
+use anyhow::Error;
+
+use proxmox_ve_config::firewall::cluster::Config as ClusterConfig;
+use proxmox_ve_config::firewall::guest::Config as GuestConfig;
+use proxmox_ve_config::firewall::host::Config as HostConfig;
+use proxmox_ve_config::firewall::types::alias::{Alias, AliasName, AliasScope};
+
+use proxmox_ve_config::guest::types::Vmid;
+use proxmox_ve_config::guest::{GuestEntry, GuestMap};
+
+use proxmox_nftables::command::{CommandOutput, Commands, List, ListOutput};
+use proxmox_nftables::types::ListChain;
+use proxmox_nftables::NftCtx;
+
+pub trait FirewallConfigLoader {
+fn cluster(&self) -> Option>;
+fn host(&self) -> Option>;
+fn guest_list(&self) -> GuestMap;
+fn guest_config(&self, vmid: &Vmid, guest: &GuestEntry) -> Option>;
+fn guest_firewall_config(&self, vmid: &Vmid) -> Option>;
+}
+
+#[derive(Default)]
+struct PveFirewallConfigLoader {}
+
+impl PveFirewallConfigLoader {
+pub fn new() -> Self {
+Default::default()
+}
+}
+
+/// opens a configuration file
+///
+/// It returns a file handle to the file or [`None`] if it doesn't exist.
+fn open_config_file(path: &str) -> Result, Error> {
+match File::open(path) {
+Ok(data) => Ok(Some(data)),
+Err(err) if err.kind() == io::ErrorKind::NotFound => {
+log::info!("config file does not exist: {path}");
+Ok(None)
+}
+Err(err) => {
+let context = format!("unable to open configuration file at
{path}");
+Err(anyhow::Error::new(err).context(context))
+}
+}
+}
+
+const CLUSTER_CONFIG_PATH: &str = "/etc/pve/firewall/cluster.fw";
+const HOST_CONFIG_PATH: &str = "/etc/pve/local/host.fw";
+
+impl FirewallConfigLoader for PveFirewallConfigLoader {
+fn cluster(&self) -> Option> {
+log::info!("loading cluster config");
+
+let fd =
+open_config_file(CLUSTER_CONFIG_PATH).expect("able to read cluster
firewall config");
+
+if let Some(file) = fd {
+let buf_reader = Box::new(BufReader::new(file)) as Box;
+return Some(buf_reader);
+}
+
+None
+}
+
+fn host(&self) -> Option> {
+log::info!("loading host config");
+
+let fd = open_config_file(HOST_CONFIG_PATH).expect("able to read host
firewall config");
+
+if let Some(file) = fd {
+let buf_reader = Box::new(BufReader::new(file)) as Box;
+return Some(buf_reader);
+}
+
+None
+}
+
+fn guest_list(&self) -> GuestMap {
+log::info!("loading vmlist");
+GuestMap::new().expect("able to read vmlist")
+}
+
+fn guest_config(&self, vmid: &Vmid, entry: &GuestEntry) -> Option> {
+log::info!("loading guest #{vmid} config");
+
+let fd = open_config_file(&GuestMap::config_path(vmid, entry))
+.expect("able to read guest config");
+
+if let Some(file) = fd {
+let buf_reader = Box::new(BufReader::new(file)) as Box;
+return Some(buf_reader);
+}
+
+None
+}
+
+fn guest_firewall_config(&self, vmid: &Vmid) -> Option> {
+log::info!("loading guest #{vmid} firewall config");
+
+let fd = open_config_file(&GuestMap::firewall_config_path(vmid))
+.expect("able to read guest firewall config");
+
+if let Some(file) = fd {
+let buf_reader = Box::new(BufReader::new(file)) as Box;
+return Some(buf_reader);
+}
+
+None
+}
+}
+
+pub trait NftConfigLoader {
+fn chains(&self) -> C
[pve-devel] [PATCH pve-docs v2 39/39] firewall: add documentation for proxmox-firewall
Add a section that explains how to use the new nftables-based
proxmox-firewall.
Signed-off-by: Stefan Hanreich
---
pve-firewall.adoc | 162 ++
1 file changed, 162 insertions(+)
diff --git a/pve-firewall.adoc b/pve-firewall.adoc
index a5e40f9..ac3d9ba 100644
--- a/pve-firewall.adoc
+++ b/pve-firewall.adoc
@@ -379,6 +379,7 @@ discovery protocol to work.
+[[pve_firewall_services_commands]]
Services and Commands
-
@@ -637,6 +638,167 @@ Ports used by {pve}
* corosync cluster traffic: 5405-5412 UDP
* live migration (VM memory and local-disk data): 6-60050 (TCP)
+
+nftables
+
+
+As an alternative to `pve-firewall` we offer `proxmox-firewall`, which is an
+implementation of the Proxmox VE firewall based on the newer
+https://wiki.nftables.org/wiki-nftables/index.php/What_is_nftables%3F[nftables]
+rather than iptables.
+
+WARNING: `proxmox-firewall` is currently in tech preview. There might be bugs
or
+incompatibilies with the original firewall. It is currently not suited for
+production use.
+
+This implementation uses the same configuration files and configuration format,
+so you can use your old configuration when switching. It provides the exact
same
+functionality with a few exceptions:
+
+* REJECT is currently not possible for guest traffic (traffic will instead be
+ dropped).
+* Using the `NDP`, `Router Advertisement` or `DHCP` options will *always*
create
+ firewall rules, irregardless of your default policy.
+* firewall rules for guests are evaluated even for connections that have
+ conntrack table entries.
+
+
+Installation and Usage
+~~
+
+Install the `proxmox-firewall` package:
+
+
+apt install proxmox-firewall
+
+
+Enable the nftables backend via the Web UI on your hosts (Host > Firewall >
+Options > nftables), or by enabling it in the configuration file for your hosts
+(`/etc/pve/nodes//host.fw`):
+
+
+[OPTIONS]
+
+nftables: 1
+
+
+WARNING: If you enable nftables without installing the `proxmox-firewall`
+package, then *no* firewall rules will be generated and your host and guests
are
+left unprotected.
+
+Additionally, all running VMs and containers need to be restarted for the new
+firewall to work.
+
+After setting the `nftables` configuration key, the new `proxmox-firewall`
+service will take over. You can check if the new service is working by
examining
+the generated ruleset. You can find more information about this in the section
+xref:pve_firewall_nft_helpful_commands[Helpful Commands]. You should also check
+whether `pve-firewall` is no longer generating iptables rules, you can find the
+respective commands in the
+xref:pve_firewall_services_commands[Services and Commands] section.
+
+Switching back to the old firewall can be done by simply setting the
+configuration value to "no" / 0.
+
+Usage
+~
+
+`proxmox-firewall` will create two tables that are managed by the
+`proxmox-firewall` service: `proxmox-firewall` and `proxmox-firewall-guests`.
If
+you want to create custom rules that live outside the Proxmox VE firewall
+configuration you can create your own tables to manage your custom firewall
+rules. `proxmox-firewall` will only touch the tables it generates, so you can
+easily extend and modify the behavior of the `proxmox-firewall` by adding your
+own tables.
+
+Instead of using the `pve-firewall` command, the nftables-based firewall uses
+`proxmox-firewall`. It is a systemd service that is triggered regularly via a
+timer, so you can start and stop it via `systemctl`:
+
+
+systemctl start proxmox-firewall.timer
+systemctl stop proxmox-firewall.timer
+
+
+To query the status of the firewall, you can query the status of the service:
+
+
+systemctl status proxmox-firewall
+
+
+
+[[pve_firewall_nft_helpful_commands]]
+Helpful Commands
+
+You can check the generated ruleset via the following command:
+
+
+nft list ruleset
+
+
+If you want to debug `proxmox-firewall` you can simply run the binary once with
+the `RUST_LOG` environment variable set to `trace`. This should provide you
with
+detailed debugging output as well as an error message in case something goes
+wrong.
+
+
+RUST_LOG=trace proxmox-firewall
+
+
+This writes the log to STDERR, you can redirect it with the following command
+(e.g. for submitting logs to the community forum):
+
+
+RUST_LOG=trace proxmox-firewall 2> firewall_log_$(hostname).txt
+
+
+Other, less verbose, log levels are `info` and `debug`.
+
+It can be helpful to trace packet flow through the different chains in order to
+debug firewall rules. This can be achieved by setting `nftrace` to 1 for
packets
+that you want to track. It is advisable that you do not set this flag for *all*
+packets, in the example below we only examine ICMP packets.
+
+
+#!/usr/sbin/nft -f
+table bridge tracebridge
+delete table bridge tracebridge
+
+table bridge tracebridge {
+chai
[pve-devel] [PATCH proxmox-firewall v2 24/39] nftables: types: add conversion traits
Some parts of the firewall config map directly to nftables objects, so
we introduce conversion traits for convenient conversion into the
respective nftables objects / types.
They are guarded behind a feature, so the nftables crate can be used
standalone without depending on the proxmox-ve-config crate.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-nftables/src/types.rs | 80 ++-
1 file changed, 79 insertions(+), 1 deletion(-)
diff --git a/proxmox-nftables/src/types.rs b/proxmox-nftables/src/types.rs
index 90d3466..a83e958 100644
--- a/proxmox-nftables/src/types.rs
+++ b/proxmox-nftables/src/types.rs
@@ -7,6 +7,12 @@ use crate::{Expression, Statement};
use serde::{Deserialize, Serialize};
+#[cfg(feature = "config-ext")]
+use proxmox_ve_config::firewall::types::address::Family;
+
+#[cfg(feature = "config-ext")]
+use proxmox_ve_config::firewall::types::ipset::IpsetName;
+
#[cfg(feature = "config-ext")]
use proxmox_ve_config::guest::types::Vmid;
@@ -33,6 +39,15 @@ impl TableFamily {
_ => vec![IpFamily::Ip, IpFamily::Ip6],
}
}
+
+#[cfg(feature = "config-ext")]
+pub fn families(&self) -> Vec {
+match self {
+TableFamily::Ip => vec![Family::V4],
+TableFamily::Ip6 => vec![Family::V6],
+_ => vec![Family::V4, Family::V6],
+}
+}
}
#[derive(Clone, Copy, Debug, Eq, PartialEq, Deserialize, Serialize)]
@@ -157,6 +172,21 @@ pub enum RateTimescale {
Day,
}
+#[cfg(feature = "config-ext")]
+use proxmox_ve_config::firewall::types::log::LogRateLimitTimescale;
+
+#[cfg(feature = "config-ext")]
+impl From for RateTimescale {
+fn from(value: LogRateLimitTimescale) -> Self {
+match value {
+LogRateLimitTimescale::Second => RateTimescale::Second,
+LogRateLimitTimescale::Minute => RateTimescale::Minute,
+LogRateLimitTimescale::Hour => RateTimescale::Hour,
+LogRateLimitTimescale::Day => RateTimescale::Day,
+}
+}
+}
+
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct TableName {
family: TableFamily,
@@ -586,6 +616,44 @@ impl SetName {
name: name.into(),
}
}
+
+pub fn name(&self) -> &str {
+self.name.as_ref()
+}
+
+#[cfg(feature = "config-ext")]
+pub fn ipset_name(
+family: Family,
+name: &IpsetName,
+vmid: Option,
+nomatch: bool,
+) -> String {
+use proxmox_ve_config::firewall::types::ipset::IpsetScope;
+
+let prefix = match family {
+Family::V4 => "v4",
+Family::V6 => "v6",
+};
+
+let name = match name.scope() {
+IpsetScope::Datacenter => name.to_string(),
+IpsetScope::Guest => {
+if let Some(vmid) = vmid {
+format!("guest-{vmid}/{}", name.name())
+} else {
+log::warn!("Creating IPSet for guest without vmid
parameter!");
+name.to_string()
+}
+}
+};
+
+let suffix = match nomatch {
+true => "-nomatch",
+false => "",
+};
+
+format!("{prefix}-{name}{suffix}")
+}
}
#[derive(Clone, Debug, Deserialize, Serialize)]
@@ -788,7 +856,17 @@ pub enum L3Protocol {
Ip6,
}
-#[derive(Clone, Debug, Deserialize, Serialize)]
+#[cfg(feature = "config-ext")]
+impl From for L3Protocol {
+fn from(value: Family) -> Self {
+match value {
+Family::V4 => L3Protocol::Ip,
+Family::V6 => L3Protocol::Ip6,
+}
+}
+}
+
+#[derive(Clone, Copy, Debug, Deserialize, Serialize)]
#[serde(rename_all = "lowercase")]
pub enum CtHelperProtocol {
TCP,
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-firewall v2 22/39] nftables: statement: add conversion traits for config types
Some types from the firewall configuration map directly onto nftables
statements. For those we implement conversion traits so we can
conveniently convert between the configuration types and the
respective nftables types.
As with the expressions, those are guarded behind a feature so the
nftables crate can be used standalone without having to pull in the
proxmox-ve-config crate.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-nftables/src/statement.rs | 71 ++-
1 file changed, 70 insertions(+), 1 deletion(-)
diff --git a/proxmox-nftables/src/statement.rs
b/proxmox-nftables/src/statement.rs
index e6371f6..e89f678 100644
--- a/proxmox-nftables/src/statement.rs
+++ b/proxmox-nftables/src/statement.rs
@@ -1,6 +1,15 @@
use anyhow::{bail, Error};
use serde::{Deserialize, Serialize};
+#[cfg(feature = "config-ext")]
+use proxmox_ve_config::firewall::types::log::LogLevel as ConfigLogLevel;
+#[cfg(feature = "config-ext")]
+use proxmox_ve_config::firewall::types::log::LogRateLimit;
+#[cfg(feature = "config-ext")]
+use proxmox_ve_config::firewall::types::rule::Verdict as ConfigVerdict;
+#[cfg(feature = "config-ext")]
+use proxmox_ve_config::guest::types::Vmid;
+
use crate::expression::Meta;
use crate::helper::{NfVec, Null};
use crate::types::{RateTimescale, RateUnit, Verdict};
@@ -104,7 +113,18 @@ impl> From for Statement {
}
}
-#[derive(Clone, Debug, Deserialize, Serialize)]
+#[cfg(feature = "config-ext")]
+impl From for Statement {
+fn from(value: ConfigVerdict) -> Self {
+match value {
+ConfigVerdict::Accept => Statement::make_accept(),
+ConfigVerdict::Reject => Statement::make_drop(),
+ConfigVerdict::Drop => Statement::make_drop(),
+}
+}
+}
+
+#[derive(Clone, Copy, Debug, Deserialize, Serialize)]
#[serde(rename_all = "lowercase")]
pub enum RejectType {
#[serde(rename = "tcp reset")]
@@ -145,6 +165,22 @@ pub struct Log {
}
impl Log {
+#[cfg(feature = "config-ext")]
+pub fn generate_prefix(
+vmid: impl Into>,
+log_level: LogLevel,
+chain_name: &str,
+verdict: ConfigVerdict,
+) -> String {
+format!(
+":{}:{}:{}: {}: ",
+vmid.into().unwrap_or(Vmid::new(0)),
+log_level.nflog_level(),
+chain_name,
+verdict,
+)
+}
+
pub fn new_nflog(prefix: String, group: i64) -> Self {
Self {
prefix: Some(prefix),
@@ -168,6 +204,25 @@ pub enum LogLevel {
Audit,
}
+#[cfg(feature = "config-ext")]
+impl TryFrom for LogLevel {
+type Error = Error;
+
+fn try_from(value: ConfigLogLevel) -> Result {
+match value {
+ConfigLogLevel::Emergency => Ok(LogLevel::Emerg),
+ConfigLogLevel::Alert => Ok(LogLevel::Alert),
+ConfigLogLevel::Critical => Ok(LogLevel::Crit),
+ConfigLogLevel::Error => Ok(LogLevel::Err),
+ConfigLogLevel::Warning => Ok(LogLevel::Warn),
+ConfigLogLevel::Notice => Ok(LogLevel::Notice),
+ConfigLogLevel::Info => Ok(LogLevel::Info),
+ConfigLogLevel::Debug => Ok(LogLevel::Debug),
+_ => bail!("cannot convert config log level to nftables"),
+}
+}
+}
+
impl LogLevel {
pub fn nflog_level(&self) -> u8 {
match self {
@@ -231,6 +286,20 @@ pub struct AnonymousLimit {
pub inv: Option,
}
+#[cfg(feature = "config-ext")]
+impl From for AnonymousLimit {
+fn from(config: LogRateLimit) -> Self {
+AnonymousLimit {
+rate: config.rate(),
+per: config.per().into(),
+rate_unit: None,
+burst: Some(config.burst()),
+burst_unit: None,
+inv: None,
+}
+}
+}
+
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct Vmap {
key: Expression,
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-firewall v2 34/39] firewall: add integration test
Signed-off-by: Stefan Hanreich
---
.gitignore|1 +
debian/control|1 +
proxmox-firewall/Cargo.toml |4 +
proxmox-firewall/src/lib.rs |4 +
proxmox-firewall/tests/input/100.conf | 10 +
proxmox-firewall/tests/input/100.fw | 22 +
proxmox-firewall/tests/input/101.conf | 11 +
proxmox-firewall/tests/input/101.fw | 19 +
proxmox-firewall/tests/input/chains.json |1 +
proxmox-firewall/tests/input/cluster.fw | 26 +
proxmox-firewall/tests/input/host.fw | 23 +
proxmox-firewall/tests/integration_tests.rs | 90 +
.../integration_tests__firewall.snap | 3530 +
13 files changed, 3742 insertions(+)
create mode 100644 proxmox-firewall/src/lib.rs
create mode 100644 proxmox-firewall/tests/input/100.conf
create mode 100644 proxmox-firewall/tests/input/100.fw
create mode 100644 proxmox-firewall/tests/input/101.conf
create mode 100644 proxmox-firewall/tests/input/101.fw
create mode 100644 proxmox-firewall/tests/input/chains.json
create mode 100644 proxmox-firewall/tests/input/cluster.fw
create mode 100644 proxmox-firewall/tests/input/host.fw
create mode 100644 proxmox-firewall/tests/integration_tests.rs
create mode 100644
proxmox-firewall/tests/snapshots/integration_tests__firewall.snap
diff --git a/.gitignore b/.gitignore
index 90749ee..c5474ef 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,3 +7,4 @@ proxmox-firewall-*/
*.build
*.buildinfo
*.changes
+*.snap.new
diff --git a/debian/control b/debian/control
index fe9467b..174375e 100644
--- a/debian/control
+++ b/debian/control
@@ -19,6 +19,7 @@ Build-Depends: cargo:native,
librust-serde-with+default-dev,
librust-libc-0.2+default-dev,
librust-proxmox-schema-3+default-dev,
+ librust-insta-dev,
libstd-rust-dev,
netbase,
python3,
diff --git a/proxmox-firewall/Cargo.toml b/proxmox-firewall/Cargo.toml
index 1e6a4b8..686aa16 100644
--- a/proxmox-firewall/Cargo.toml
+++ b/proxmox-firewall/Cargo.toml
@@ -20,3 +20,7 @@ serde_json = "1"
proxmox-nftables = { path = "../proxmox-nftables", features = ["config-ext"] }
proxmox-ve-config = { path = "../proxmox-ve-config" }
+
+[dev-dependencies]
+insta = { version = "1.21", features = ["json"] }
+proxmox-sys = "0.5.3"
diff --git a/proxmox-firewall/src/lib.rs b/proxmox-firewall/src/lib.rs
new file mode 100644
index 000..c4b037a
--- /dev/null
+++ b/proxmox-firewall/src/lib.rs
@@ -0,0 +1,4 @@
+pub mod config;
+pub mod firewall;
+pub mod object;
+pub mod rule;
diff --git a/proxmox-firewall/tests/input/100.conf
b/proxmox-firewall/tests/input/100.conf
new file mode 100644
index 000..495f899
--- /dev/null
+++ b/proxmox-firewall/tests/input/100.conf
@@ -0,0 +1,10 @@
+arch: amd64
+cores: 1
+features: nesting=1
+hostname: host1
+memory: 512
+net1:
name=eth0,bridge=simple1,firewall=1,hwaddr=BC:24:11:4D:B0:FF,ip=dhcp,ip6=fd80::1234/64,type=veth
+ostype: debian
+rootfs: local-lvm:vm-90001-disk-0,size=2G
+swap: 512
+unprivileged: 1
diff --git a/proxmox-firewall/tests/input/100.fw
b/proxmox-firewall/tests/input/100.fw
new file mode 100644
index 000..6cf9fff
--- /dev/null
+++ b/proxmox-firewall/tests/input/100.fw
@@ -0,0 +1,22 @@
+[OPTIONS]
+
+enable: 1
+ndp: 1
+ipfilter: 1
+dhcp: 1
+log_level_in: crit
+log_level_out: alert
+policy_in: DROP
+policy_out: REJECT
+macfilter: 0
+
+[IPSET ipfilter-net1]
+
+dc/network1
+
+[RULES]
+
+GROUP network1 -i net1
+IN ACCEPT -source 192.168.0.1/24,127.0.0.1-127.255.255.0,172.16.0.1 -dport
123,222:333 -sport http -p tcp
+IN DROP --icmp-type echo-request --proto icmp --log info
+
diff --git a/proxmox-firewall/tests/input/101.conf
b/proxmox-firewall/tests/input/101.conf
new file mode 100644
index 000..394e2e4
--- /dev/null
+++ b/proxmox-firewall/tests/input/101.conf
@@ -0,0 +1,11 @@
+boot: order=ide2
+cores: 2
+cpu: x86-64-v2-AES
+memory: 2048
+meta: creation-qemu=8.1.5,ctime=1712322773
+numa: 0
+ostype: l26
+scsihw: virtio-scsi-single
+smbios1: uuid=78ec7794-78f7-4c03-bf08-18b721a6
+sockets: 1
+vmgenid: ec7d4834-cd0a-4376-9c1d-af8a82da8d54
diff --git a/proxmox-firewall/tests/input/101.fw
b/proxmox-firewall/tests/input/101.fw
new file mode 100644
index 000..c77cb5a
--- /dev/null
+++ b/proxmox-firewall/tests/input/101.fw
@@ -0,0 +1,19 @@
+[OPTIONS]
+
+ndp: 0
+enable: 1
+dhcp: 1
+radv: 0
+policy_out: ACCEPT
+
+[ALIASES]
+
+analias 123.123.123.123
+
+[IPSET testing]
+
+
+[RULES]
+
+IN ACCEPT -source guest/analias -dest dc/network2 -log nolog
+
diff --git a/proxmox-firewall/tests/input/chains.json
b/proxmox-firewall/tests/input/chains.json
new file mode 100644
index 000..327c295
--- /dev/null
+++ b/proxmox-firewall/tests/input/chains.json
@@ -0,0 +1 @@
+{"nftables": [{"metainfo": {"version": "1.0.6", "release_name": "Lester Gooch
#5", "json_schema
[pve-devel] [PATCH proxmox-firewall v2 23/39] nftables: commands: add types
Add rust types for most of the nftables commands as defined by
libnftables-json [1].
Different commands require different keys to be set for the same type
of object. E.g. deleting an object usually only requires a name +
name of the container (table/chain/rule). Creating an object usually
requires a few more keys, depending on the type of object created.
In order to be able to model the different objects for the different
commands, I've created specific models for a command where necessary.
Parts that are common across multiple commands (e.g. names) have been
moved to their own structs, so they can be reused.
[1]
https://manpages.debian.org/bookworm/libnftables1/libnftables-json.5.en.html#COMMAND_OBJECTS
Signed-off-by: Stefan Hanreich
---
proxmox-nftables/src/command.rs | 233 ++
proxmox-nftables/src/lib.rs | 2 +
proxmox-nftables/src/types.rs | 770 +++-
3 files changed, 1004 insertions(+), 1 deletion(-)
create mode 100644 proxmox-nftables/src/command.rs
diff --git a/proxmox-nftables/src/command.rs b/proxmox-nftables/src/command.rs
new file mode 100644
index 000..193fe46
--- /dev/null
+++ b/proxmox-nftables/src/command.rs
@@ -0,0 +1,233 @@
+use std::ops::{Deref, DerefMut};
+
+use crate::helper::Null;
+use crate::types::*;
+use serde::{Deserialize, Serialize};
+
+#[derive(Clone, Debug, Default, Deserialize, Serialize)]
+pub struct Commands {
+nftables: Vec,
+}
+
+impl Commands {
+pub fn new(commands: Vec) -> Self {
+Self { nftables: commands }
+}
+}
+
+impl Deref for Commands {
+type Target = Vec;
+
+fn deref(&self) -> &Self::Target {
+&self.nftables
+}
+}
+
+impl DerefMut for Commands {
+fn deref_mut(&mut self) -> &mut Self::Target {
+&mut self.nftables
+}
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum Command {
+Add(Add),
+Create(Add),
+Delete(Delete),
+Flush(Flush),
+List(List),
+// Insert(super::Rule),
+// Rename(RenameChain),
+// Replace(super::Rule),
+}
+
+#[derive(Clone, Copy, Debug, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum List {
+Chains(Null),
+Sets(Null),
+}
+
+impl List {
+#[inline]
+pub fn chains() -> Command {
+Command::List(List::Chains(Null))
+}
+
+#[inline]
+pub fn sets() -> Command {
+Command::List(List::Sets(Null))
+}
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum Add {
+Table(AddTable),
+Chain(AddChain),
+Rule(AddRule),
+Set(AddSet),
+Map(AddMap),
+Limit(AddLimit),
+Element(AddElement),
+#[serde(rename = "ct helper")]
+CtHelper(AddCtHelper),
+}
+
+impl Add {
+#[inline]
+pub fn table(table: impl Into) -> Command {
+Command::Add(Add::Table(table.into()))
+}
+
+#[inline]
+pub fn chain(chain: impl Into) -> Command {
+Command::Add(Add::Chain(chain.into()))
+}
+
+#[inline]
+pub fn rule(rule: impl Into) -> Command {
+Command::Add(Add::Rule(rule.into()))
+}
+
+#[inline]
+pub fn set(set: impl Into) -> Command {
+Command::Add(Add::Set(set.into()))
+}
+
+#[inline]
+pub fn map(map: impl Into) -> Command {
+Command::Add(Add::Map(map.into()))
+}
+
+#[inline]
+pub fn limit(limit: impl Into) -> Command {
+Command::Add(Add::Limit(limit.into()))
+}
+
+#[inline]
+pub fn element(element: impl Into) -> Command {
+Command::Add(Add::Element(element.into()))
+}
+
+#[inline]
+pub fn ct_helper(ct_helper: impl Into) -> Command {
+Command::Add(Add::CtHelper(ct_helper.into()))
+}
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum Flush {
+Table(TableName),
+Chain(ChainName),
+Set(SetName),
+Map(SetName),
+Ruleset(Null),
+}
+
+impl Flush {
+#[inline]
+pub fn table(table: impl Into) -> Command {
+Command::Flush(Flush::Table(table.into()))
+}
+
+#[inline]
+pub fn chain(chain: impl Into) -> Command {
+Command::Flush(Flush::Chain(chain.into()))
+}
+
+#[inline]
+pub fn set(set: impl Into) -> Command {
+Command::Flush(Flush::Set(set.into()))
+}
+
+#[inline]
+pub fn map(map: impl Into) -> Command {
+Command::Flush(Flush::Map(map.into()))
+}
+
+#[inline]
+pub fn ruleset() -> Command {
+Command::Flush(Flush::Ruleset(Null))
+}
+}
+
+impl From for Flush {
+#[inline]
+fn from(value: TableName) -> Self {
+Flush::Table(value)
+}
+}
+
+impl From for Flush {
+#[inline]
+fn from(value: ChainName) -> Self {
+Flush::Chain(value)
+}
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum Delete {
+Table(TableName),
+Chain(ChainName),
+Set(SetName),
+}
+
+impl De
[pve-devel] [PATCH qemu-server v2 35/39] firewall: add handling for new nft firewall
When the nftables firewall is enabled, we do not need to create
firewall bridges.
Signed-off-by: Stefan Hanreich
---
vm-network-scripts/pve-bridge | 9 +++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/vm-network-scripts/pve-bridge b/vm-network-scripts/pve-bridge
index 85997a0..ac2eb3b 100755
--- a/vm-network-scripts/pve-bridge
+++ b/vm-network-scripts/pve-bridge
@@ -6,6 +6,7 @@ use warnings;
use PVE::QemuServer;
use PVE::Tools qw(run_command);
use PVE::Network;
+use PVE::Firewall;
my $have_sdn;
eval {
@@ -44,13 +45,17 @@ die "unable to get network config '$netid'\n"
my $net = PVE::QemuServer::parse_net($netconf);
die "unable to parse network config '$netid'\n" if !$net;
+my $cluster_fw_conf = PVE::Firewall::load_clusterfw_conf();
+my $host_fw_conf = PVE::Firewall::load_hostfw_conf($cluster_fw_conf);
+my $firewall = $net->{firewall} && !($host_fw_conf->{options}->{nftables} //
0);
+
if ($have_sdn) {
PVE::Network::SDN::Vnets::add_dhcp_mapping($net->{bridge},
$net->{macaddr}, $vmid, $conf->{name});
PVE::Network::SDN::Zones::tap_create($iface, $net->{bridge});
-PVE::Network::SDN::Zones::tap_plug($iface, $net->{bridge}, $net->{tag},
$net->{firewall}, $net->{trunks}, $net->{rate});
+PVE::Network::SDN::Zones::tap_plug($iface, $net->{bridge}, $net->{tag},
$firewall, $net->{trunks}, $net->{rate});
} else {
PVE::Network::tap_create($iface, $net->{bridge});
-PVE::Network::tap_plug($iface, $net->{bridge}, $net->{tag},
$net->{firewall}, $net->{trunks}, $net->{rate});
+PVE::Network::tap_plug($iface, $net->{bridge}, $net->{tag}, $firewall,
$net->{trunks}, $net->{rate});
}
exit 0;
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-firewall v2 09/39] config: firewall: add types for rules
Additionally we implement FromStr for all rule types and parts, which
can be used for parsing firewall config rules. Initial rule parsing
works by parsing the different options into a HashMap and only then
de-serializing a struct from the parsed options.
This intermediate step makes rule parsing a lot easier, since we can
reuse the deserialization logic from serde. Also, we can split the
parsing/deserialization logic from the validation logic.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/src/firewall/parse.rs | 185
proxmox-ve-config/src/firewall/types/mod.rs | 3 +
proxmox-ve-config/src/firewall/types/rule.rs | 412
.../src/firewall/types/rule_match.rs | 965 ++
4 files changed, 1565 insertions(+)
create mode 100644 proxmox-ve-config/src/firewall/types/rule.rs
create mode 100644 proxmox-ve-config/src/firewall/types/rule_match.rs
diff --git a/proxmox-ve-config/src/firewall/parse.rs
b/proxmox-ve-config/src/firewall/parse.rs
index b02f98d..e2ce463 100644
--- a/proxmox-ve-config/src/firewall/parse.rs
+++ b/proxmox-ve-config/src/firewall/parse.rs
@@ -1,3 +1,5 @@
+use std::fmt;
+
use anyhow::{bail, format_err, Error};
/// Parses out a "name" which can be alphanumeric and include dashes.
@@ -91,3 +93,186 @@ pub fn parse_bool(value: &str) -> Result {
},
)
}
+
+/// `&str` deserializer which also accepts an `Option`.
+///
+/// Serde's `StringDeserializer` does not.
+#[derive(Clone, Copy, Debug)]
+pub struct SomeStrDeserializer<'a, E>(serde::de::value::StrDeserializer<'a,
E>);
+
+impl<'de, 'a, E> serde::de::Deserializer<'de> for SomeStrDeserializer<'a, E>
+where
+E: serde::de::Error,
+{
+type Error = E;
+
+fn deserialize_any(self, visitor: V) -> Result
+where
+V: serde::de::Visitor<'de>,
+{
+self.0.deserialize_any(visitor)
+}
+
+fn deserialize_option(self, visitor: V) -> Result
+where
+V: serde::de::Visitor<'de>,
+{
+visitor.visit_some(self.0)
+}
+
+fn deserialize_str(self, visitor: V) -> Result
+where
+V: serde::de::Visitor<'de>,
+{
+self.0.deserialize_str(visitor)
+}
+
+fn deserialize_string(self, visitor: V) -> Result
+where
+V: serde::de::Visitor<'de>,
+{
+self.0.deserialize_string(visitor)
+}
+
+fn deserialize_enum(
+self,
+_name: &str,
+_variants: &'static [&'static str],
+visitor: V,
+) -> Result
+where
+V: serde::de::Visitor<'de>,
+{
+visitor.visit_enum(self.0)
+}
+
+serde::forward_to_deserialize_any! {
+bool i8 i16 i32 i64 i128 u8 u16 u32 u64 u128 f32 f64 char
+bytes byte_buf unit unit_struct newtype_struct seq tuple
+tuple_struct map struct identifier ignored_any
+}
+}
+
+/// `&str` wrapper which implements `IntoDeserializer` via
`SomeStrDeserializer`.
+#[derive(Clone, Debug)]
+pub struct SomeStr<'a>(pub &'a str);
+
+impl<'a> From<&'a str> for SomeStr<'a> {
+fn from(s: &'a str) -> Self {
+Self(s)
+}
+}
+
+impl<'de, 'a, E> serde::de::IntoDeserializer<'de, E> for SomeStr<'a>
+where
+E: serde::de::Error,
+{
+type Deserializer = SomeStrDeserializer<'a, E>;
+
+fn into_deserializer(self) -> Self::Deserializer {
+SomeStrDeserializer(self.0.into_deserializer())
+}
+}
+
+/// `String` deserializer which also accepts an `Option`.
+///
+/// Serde's `StringDeserializer` does not.
+#[derive(Clone, Debug)]
+pub struct SomeStringDeserializer(serde::de::value::StringDeserializer);
+
+impl<'de, E> serde::de::Deserializer<'de> for SomeStringDeserializer
+where
+E: serde::de::Error,
+{
+type Error = E;
+
+fn deserialize_any(self, visitor: V) -> Result
+where
+V: serde::de::Visitor<'de>,
+{
+self.0.deserialize_any(visitor)
+}
+
+fn deserialize_option(self, visitor: V) -> Result
+where
+V: serde::de::Visitor<'de>,
+{
+visitor.visit_some(self.0)
+}
+
+fn deserialize_str(self, visitor: V) -> Result
+where
+V: serde::de::Visitor<'de>,
+{
+self.0.deserialize_str(visitor)
+}
+
+fn deserialize_string(self, visitor: V) -> Result
+where
+V: serde::de::Visitor<'de>,
+{
+self.0.deserialize_string(visitor)
+}
+
+fn deserialize_enum(
+self,
+_name: &str,
+_variants: &'static [&'static str],
+visitor: V,
+) -> Result
+where
+V: serde::de::Visitor<'de>,
+{
+visitor.visit_enum(self.0)
+}
+
+serde::forward_to_deserialize_any! {
+bool i8 i16 i32 i64 i128 u8 u16 u32 u64 u128 f32 f64 char
+bytes byte_buf unit unit_struct newtype_struct seq tuple
+tuple_struct map struct identifier ignored_any
+}
+}
+
+/// `&str` wrapper which implements `IntoDeserializer` via
`SomeStringD
[pve-devel] [PATCH proxmox-firewall v2 07/39] config: guest: add helpers for parsing guest network config
Currently this is parsing the config files via the filesystem. In the
future we could also get this information from pmxcfs directly via
IPC which should be more performant, particularly for a large number
of VMs.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/src/firewall/parse.rs | 20 +
proxmox-ve-config/src/guest/mod.rs | 115 ++
proxmox-ve-config/src/guest/types.rs| 38 ++
proxmox-ve-config/src/guest/vm.rs | 510
proxmox-ve-config/src/lib.rs| 1 +
5 files changed, 684 insertions(+)
create mode 100644 proxmox-ve-config/src/guest/mod.rs
create mode 100644 proxmox-ve-config/src/guest/types.rs
create mode 100644 proxmox-ve-config/src/guest/vm.rs
diff --git a/proxmox-ve-config/src/firewall/parse.rs
b/proxmox-ve-config/src/firewall/parse.rs
index 772e081..b02f98d 100644
--- a/proxmox-ve-config/src/firewall/parse.rs
+++ b/proxmox-ve-config/src/firewall/parse.rs
@@ -52,6 +52,26 @@ pub fn match_non_whitespace(line: &str) -> Option<(&str,
&str)> {
Some((text, rest))
}
}
+
+/// parses out all digits and returns the remainder
+///
+/// returns [`None`] if the digit part would be empty
+///
+/// Returns a tuple with the digits and the remainder (not trimmed).
+pub fn match_digits(line: &str) -> Option<(&str, &str)> {
+let split_position = line.as_bytes().iter().position(|&b|
!b.is_ascii_digit());
+
+let (digits, rest) = match split_position {
+Some(pos) => line.split_at(pos),
+None => (line, ""),
+};
+
+if !digits.is_empty() {
+return Some((digits, rest));
+}
+
+None
+}
pub fn parse_bool(value: &str) -> Result {
Ok(
if value == "0"
diff --git a/proxmox-ve-config/src/guest/mod.rs
b/proxmox-ve-config/src/guest/mod.rs
new file mode 100644
index 000..74fd8ab
--- /dev/null
+++ b/proxmox-ve-config/src/guest/mod.rs
@@ -0,0 +1,115 @@
+use core::ops::Deref;
+use std::collections::HashMap;
+
+use anyhow::{Context, Error};
+use serde::Deserialize;
+
+use proxmox_sys::nodename;
+use types::Vmid;
+
+pub mod types;
+pub mod vm;
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq, Deserialize)]
+pub enum GuestType {
+#[serde(rename = "qemu")]
+Vm,
+#[serde(rename = "lxc")]
+Ct,
+}
+
+impl GuestType {
+pub fn iface_prefix(self) -> &'static str {
+match self {
+GuestType::Vm => "tap",
+GuestType::Ct => "veth",
+}
+}
+
+fn config_folder(&self) -> &'static str {
+match self {
+GuestType::Vm => "qemu-server",
+GuestType::Ct => "lxc",
+}
+}
+}
+
+#[derive(Deserialize)]
+pub struct GuestEntry {
+node: String,
+
+#[serde(rename = "type")]
+ty: GuestType,
+
+#[serde(rename = "version")]
+_version: usize,
+}
+
+impl GuestEntry {
+pub fn new(node: String, ty: GuestType) -> Self {
+Self {
+node,
+ty,
+_version: Default::default(),
+}
+}
+
+pub fn is_local(&self) -> bool {
+nodename() == self.node
+}
+
+pub fn ty(&self) -> &GuestType {
+&self.ty
+}
+}
+
+const VMLIST_CONFIG_PATH: &str = "/etc/pve/.vmlist";
+
+#[derive(Deserialize)]
+pub struct GuestMap {
+#[serde(rename = "version")]
+_version: usize,
+#[serde(rename = "ids", default)]
+guests: HashMap,
+}
+
+impl From> for GuestMap {
+fn from(guests: HashMap) -> Self {
+Self {
+guests,
+_version: Default::default(),
+}
+}
+}
+
+impl Deref for GuestMap {
+type Target = HashMap;
+
+fn deref(&self) -> &Self::Target {
+&self.guests
+}
+}
+
+impl GuestMap {
+pub fn new() -> Result {
+let data = std::fs::read(VMLIST_CONFIG_PATH)
+.with_context(|| format!("failed to read guest map from
{VMLIST_CONFIG_PATH}"))?;
+
+serde_json::from_slice(&data).with_context(|| "failed to parse guest
map".to_owned())
+}
+
+pub fn firewall_config_path(vmid: &Vmid) -> String {
+format!("/etc/pve/firewall/{}.fw", vmid)
+}
+
+/// returns the local configuration path for a given Vmid.
+///
+/// The caller must ensure that the given Vmid exists and is local to the
node
+pub fn config_path(vmid: &Vmid, entry: &GuestEntry) -> String {
+format!(
+"/etc/pve/local/{}/{}.conf",
+entry.ty().config_folder(),
+vmid
+)
+}
+}
diff --git a/proxmox-ve-config/src/guest/types.rs
b/proxmox-ve-config/src/guest/types.rs
new file mode 100644
index 000..217c537
--- /dev/null
+++ b/proxmox-ve-config/src/guest/types.rs
@@ -0,0 +1,38 @@
+use std::fmt;
+use std::str::FromStr;
+
+use anyhow::{format_err, Error};
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq, PartialOrd, Ord, Hash)]
+pub struct Vmid(u32);
+
+impl Vmid {
+pub fn new(id: u32) -> Self {
+Vmid(id)
+}
[pve-devel] [PATCH proxmox-firewall v2 15/39] config: firewall: add firewall macros
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/resources/macros.json | 914
proxmox-ve-config/src/firewall/fw_macros.rs | 69 ++
proxmox-ve-config/src/firewall/mod.rs | 1 +
3 files changed, 984 insertions(+)
create mode 100644 proxmox-ve-config/resources/macros.json
create mode 100644 proxmox-ve-config/src/firewall/fw_macros.rs
diff --git a/proxmox-ve-config/resources/macros.json
b/proxmox-ve-config/resources/macros.json
new file mode 100644
index 000..67e1d89
--- /dev/null
+++ b/proxmox-ve-config/resources/macros.json
@@ -0,0 +1,914 @@
+{
+ "Amanda": {
+"code": [
+ {
+"dport": "10080",
+"proto": "udp"
+ },
+ {
+"dport": "10080",
+"proto": "tcp"
+ }
+],
+"desc": "Amanda Backup"
+ },
+ "Auth": {
+"code": [
+ {
+"dport": "113",
+"proto": "tcp"
+ }
+],
+"desc": "Auth (identd) traffic"
+ },
+ "BGP": {
+"code": [
+ {
+"dport": "179",
+"proto": "tcp"
+ }
+],
+"desc": "Border Gateway Protocol traffic"
+ },
+ "BitTorrent": {
+"code": [
+ {
+"dport": "6881:6889",
+"proto": "tcp"
+ },
+ {
+"dport": "6881",
+"proto": "udp"
+ }
+],
+"desc": "BitTorrent traffic for BitTorrent 3.1 and earlier"
+ },
+ "BitTorrent32": {
+"code": [
+ {
+"dport": "6881:6999",
+"proto": "tcp"
+ },
+ {
+"dport": "6881",
+"proto": "udp"
+ }
+],
+"desc": "BitTorrent traffic for BitTorrent 3.2 and later"
+ },
+ "CVS": {
+"code": [
+ {
+"dport": "2401",
+"proto": "tcp"
+ }
+],
+"desc": "Concurrent Versions System pserver traffic"
+ },
+ "Ceph": {
+"code": [
+ {
+"dport": "6789",
+"proto": "tcp"
+ },
+ {
+"dport": "3300",
+"proto": "tcp"
+ },
+ {
+"dport": "6800:7300",
+"proto": "tcp"
+ }
+],
+"desc": "Ceph Storage Cluster traffic (Ceph Monitors, OSD & MDS Daemons)"
+ },
+ "Citrix": {
+"code": [
+ {
+"dport": "1494",
+"proto": "tcp"
+ },
+ {
+"dport": "1604",
+"proto": "udp"
+ },
+ {
+"dport": "2598",
+"proto": "tcp"
+ }
+],
+"desc": "Citrix/ICA traffic (ICA, ICA Browser, CGP)"
+ },
+ "DAAP": {
+"code": [
+ {
+"dport": "3689",
+"proto": "tcp"
+ },
+ {
+"dport": "3689",
+"proto": "udp"
+ }
+],
+"desc": "Digital Audio Access Protocol traffic (iTunes, Rythmbox daemons)"
+ },
+ "DCC": {
+"code": [
+ {
+"dport": "6277",
+"proto": "tcp"
+ }
+],
+"desc": "Distributed Checksum Clearinghouse spam filtering mechanism"
+ },
+ "DHCPfwd": {
+"code": [
+ {
+"dport": "67:68",
+"proto": "udp",
+"sport": "67:68"
+ }
+],
+"desc": "Forwarded DHCP traffic"
+ },
+ "DHCPv6": {
+"code": [
+ {
+"dport": "546:547",
+"proto": "udp",
+"sport": "546:547"
+ }
+],
+"desc": "DHCPv6 traffic"
+ },
+ "DNS": {
+"code": [
+ {
+"dport": "53",
+"proto": "udp"
+ },
+ {
+"dport": "53",
+"proto": "tcp"
+ }
+],
+"desc": "Domain Name System traffic (upd and tcp)"
+ },
+ "Distcc": {
+"code": [
+ {
+"dport": "3632",
+"proto": "tcp"
+ }
+],
+"desc": "Distributed Compiler service"
+ },
+ "FTP": {
+"code": [
+ {
+"dport": "21",
+"proto": "tcp"
+ }
+],
+"desc": "File Transfer Protocol"
+ },
+ "Finger": {
+"code": [
+ {
+"dport": "79",
+"proto": "tcp"
+ }
+],
+"desc": "Finger protocol (RFC 742)"
+ },
+ "GNUnet": {
+"code": [
+ {
+"dport": "2086",
+"proto": "tcp"
+ },
+ {
+"dport": "2086",
+"proto": "udp"
+ },
+ {
+"dport": "1080",
+"proto": "tcp"
+ },
+ {
+"dport": "1080",
+"proto": "udp"
+ }
+],
+"desc": "GNUnet secure peer-to-peer networking traffic"
+ },
+ "GRE": {
+"code": [
+ {
+"proto": "47"
+ }
+],
+"desc": "Generic Routing Encapsulation tunneling protocol"
+ },
+ "Git": {
+"code": [
+ {
+"dport": "9418",
+"proto": "tcp"
+ }
+],
+"desc": "Git distributed revision control traffic"
+ },
+ "HKP": {
+"code": [
+ {
+"dport": "11371",
+"proto": "tcp"
+ }
+],
+"desc": "OpenPGP HTTP key server protocol traffic"
+ },
+ "HTTP": {
+"code": [
+ {
+"dport": "80",
+"proto": "tcp"
+ }
+],
+"desc": "Hypertext Transfer Protocol (WWW)
[pve-devel] [PATCH proxmox-firewall v2 20/39] nftables: expression: implement conversion traits for firewall config
Some types from the firewall configuration map directly onto nftables
expressions. For those we implement conversion traits so we can
conveniently convert between the configuration types and the
respective nftables types.
Those are guarded behind a feature so the nftables crate can be used
standalone without having to pull in the proxmox-ve-config crate.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-nftables/Cargo.toml| 5 +-
proxmox-nftables/src/expression.rs | 124 +++--
2 files changed, 122 insertions(+), 7 deletions(-)
diff --git a/proxmox-nftables/Cargo.toml b/proxmox-nftables/Cargo.toml
index 909869b..7e607e8 100644
--- a/proxmox-nftables/Cargo.toml
+++ b/proxmox-nftables/Cargo.toml
@@ -10,6 +10,9 @@ authors = [
description = "Proxmox VE nftables"
license = "AGPL-3"
+[features]
+config-ext = ["dep:proxmox-ve-config"]
+
[dependencies]
log = "0.4"
@@ -17,4 +20,4 @@ serde = { version = "1", features = [ "derive" ] }
serde_json = "1"
serde_plain = "1"
-proxmox-ve-config = { path = "../proxmox-ve-config" }
+proxmox-ve-config = { path = "../proxmox-ve-config", optional = true }
diff --git a/proxmox-nftables/src/expression.rs
b/proxmox-nftables/src/expression.rs
index 5478291..3b8ade0 100644
--- a/proxmox-nftables/src/expression.rs
+++ b/proxmox-nftables/src/expression.rs
@@ -2,7 +2,14 @@ use crate::types::{ElemConfig, Verdict};
use serde::{Deserialize, Serialize};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
-use crate::helper::NfVec;
+#[cfg(feature = "config-ext")]
+use proxmox_ve_config::firewall::types::address::{Family, IpEntry, IpList};
+#[cfg(feature = "config-ext")]
+use proxmox_ve_config::firewall::types::port::{PortEntry, PortList};
+#[cfg(feature = "config-ext")]
+use proxmox_ve_config::firewall::types::rule_match::{IcmpCode, IcmpType,
Icmpv6Code, Icmpv6Type};
+#[cfg(feature = "config-ext")]
+use proxmox_ve_config::firewall::types::Cidr;
#[derive(Clone, Debug, Deserialize, Serialize)]
#[serde(rename_all = "lowercase")]
@@ -147,11 +154,88 @@ impl From<&Ipv4Addr> for Expression {
}
}
-#[derive(Clone, Copy, Debug, Eq, PartialEq, Deserialize, Serialize)]
-#[serde(rename_all = "lowercase")]
-pub enum IpFamily {
-Ip,
-Ip6,
+#[cfg(feature = "config-ext")]
+impl From<&IpList> for Expression {
+fn from(value: &IpList) -> Self {
+if value.len() == 1 {
+return Expression::from(value.first().unwrap());
+}
+
+Expression::set(value.iter().map(Expression::from))
+}
+}
+
+#[cfg(feature = "config-ext")]
+impl From<&IpEntry> for Expression {
+fn from(value: &IpEntry) -> Self {
+match value {
+IpEntry::Cidr(cidr) => Expression::from(Prefix::from(cidr)),
+IpEntry::Range(beg, end) =>
Expression::Range(Box::new((beg.into(), end.into(,
+}
+}
+}
+
+#[cfg(feature = "config-ext")]
+impl From<&IcmpType> for Expression {
+fn from(value: &IcmpType) -> Self {
+match value {
+IcmpType::Numeric(id) => Expression::from(*id),
+IcmpType::Named(name) => Expression::from(*name),
+}
+}
+}
+
+#[cfg(feature = "config-ext")]
+impl From<&IcmpCode> for Expression {
+fn from(value: &IcmpCode) -> Self {
+match value {
+IcmpCode::Numeric(id) => Expression::from(*id),
+IcmpCode::Named(name) => Expression::from(*name),
+}
+}
+}
+
+#[cfg(feature = "config-ext")]
+impl From<&Icmpv6Type> for Expression {
+fn from(value: &Icmpv6Type) -> Self {
+match value {
+Icmpv6Type::Numeric(id) => Expression::from(*id),
+Icmpv6Type::Named(name) => Expression::from(*name),
+}
+}
+}
+
+#[cfg(feature = "config-ext")]
+impl From<&Icmpv6Code> for Expression {
+fn from(value: &Icmpv6Code) -> Self {
+match value {
+Icmpv6Code::Numeric(id) => Expression::from(*id),
+Icmpv6Code::Named(name) => Expression::from(*name),
+}
+}
+}
+
+#[cfg(feature = "config-ext")]
+impl From<&PortEntry> for Expression {
+fn from(value: &PortEntry) -> Self {
+match value {
+PortEntry::Port(port) => Expression::from(*port),
+PortEntry::Range(beg, end) => {
+Expression::Range(Box::new(((*beg).into(), (*end).into(
+}
+}
+}
+}
+
+#[cfg(feature = "config-ext")]
+impl From<&PortList> for Expression {
+fn from(value: &PortList) -> Self {
+if value.len() == 1 {
+return Expression::from(value.first().unwrap());
+}
+
+Expression::set(value.iter().map(Expression::from))
+}
}
#[derive(Clone, Debug, Deserialize, Serialize)]
@@ -197,6 +281,24 @@ pub enum CtDirection {
Reply,
}
serde_plain::derive_display_from_serialize!(CtDirection);
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq, Deserialize, Serialize)]
+#[serde(rename_all = "lo
[pve-devel] [PATCH proxmox-firewall v2 26/39] firewall: add firewall crate
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
Cargo.toml | 1 +
proxmox-firewall/Cargo.toml | 17 +
proxmox-firewall/src/main.rs | 5 +
3 files changed, 23 insertions(+)
create mode 100644 proxmox-firewall/Cargo.toml
create mode 100644 proxmox-firewall/src/main.rs
diff --git a/Cargo.toml b/Cargo.toml
index 877f103..f353fbf 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -2,4 +2,5 @@
members = [
"proxmox-ve-config",
"proxmox-nftables",
+"proxmox-firewall",
]
diff --git a/proxmox-firewall/Cargo.toml b/proxmox-firewall/Cargo.toml
new file mode 100644
index 000..b59d973
--- /dev/null
+++ b/proxmox-firewall/Cargo.toml
@@ -0,0 +1,17 @@
+[package]
+name = "proxmox-firewall"
+version = "0.1.0"
+edition = "2021"
+authors = [
+"Wolfgang Bumiller ",
+"Stefan Hanreich ",
+"Proxmox Support Team ",
+]
+description = "Proxmox VE nftables firewall implementation"
+license = "AGPL-3"
+
+[dependencies]
+anyhow = "1"
+
+proxmox-nftables = { path = "../proxmox-nftables", features = ["config-ext"] }
+proxmox-ve-config = { path = "../proxmox-ve-config" }
diff --git a/proxmox-firewall/src/main.rs b/proxmox-firewall/src/main.rs
new file mode 100644
index 000..248ac39
--- /dev/null
+++ b/proxmox-firewall/src/main.rs
@@ -0,0 +1,5 @@
+use anyhow::Error;
+
+fn main() -> Result<(), Error> {
+Ok(())
+}
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-firewall v2 11/39] config: firewall: add generic parser for firewall configs
Since the basic format of cluster, host and guest firewall
configurations is the same, we create a generic parser that can handle
the common config format. The main difference is in the available
options, which can be passed via a generic parameter.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/src/firewall/common.rs | 184
proxmox-ve-config/src/firewall/mod.rs| 1 +
proxmox-ve-config/src/firewall/parse.rs | 210 +++
3 files changed, 395 insertions(+)
create mode 100644 proxmox-ve-config/src/firewall/common.rs
diff --git a/proxmox-ve-config/src/firewall/common.rs
b/proxmox-ve-config/src/firewall/common.rs
new file mode 100644
index 000..a08f19c
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/common.rs
@@ -0,0 +1,184 @@
+use std::collections::{BTreeMap, HashMap};
+use std::io;
+
+use anyhow::{bail, format_err, Error};
+use serde::de::IntoDeserializer;
+
+use crate::firewall::parse::{parse_named_section_tail, split_key_value,
SomeString};
+use crate::firewall::types::ipset::{IpsetName, IpsetScope};
+use crate::firewall::types::{Alias, Group, Ipset, Rule};
+
+#[derive(Debug, Default)]
+pub struct Config
+where
+O: Default + std::fmt::Debug + serde::de::DeserializeOwned,
+{
+pub(crate) options: O,
+pub(crate) rules: Vec,
+pub(crate) aliases: BTreeMap,
+pub(crate) ipsets: BTreeMap,
+pub(crate) groups: BTreeMap,
+}
+
+enum Sec {
+None,
+Options,
+Aliases,
+Rules,
+Ipset(String, Ipset),
+Group(String, Group),
+}
+
+#[derive(Default)]
+pub struct ParserConfig {
+/// Network interfaces must be of the form `netX`.
+pub guest_iface_names: bool,
+pub ipset_scope: Option,
+}
+
+impl Config
+where
+O: Default + std::fmt::Debug + serde::de::DeserializeOwned,
+{
+pub fn new() -> Self {
+Self::default()
+}
+
+pub fn parse(input: R, parser_cfg: &ParserConfig) ->
Result {
+let mut section = Sec::None;
+
+let mut this = Self::new();
+let mut options = HashMap::new();
+
+for line in input.lines() {
+let line = line?;
+let line = line.trim();
+
+if line.is_empty() || line.starts_with('#') {
+continue;
+}
+
+log::trace!("parsing config line {line}");
+
+if line.eq_ignore_ascii_case("[OPTIONS]") {
+this.set_section(&mut section, Sec::Options)?;
+} else if line.eq_ignore_ascii_case("[ALIASES]") {
+this.set_section(&mut section, Sec::Aliases)?;
+} else if line.eq_ignore_ascii_case("[RULES]") {
+this.set_section(&mut section, Sec::Rules)?;
+} else if let Some(line) = line.strip_prefix("[IPSET") {
+let (name, comment) = parse_named_section_tail("ipset", line)?;
+
+let scope = parser_cfg.ipset_scope.ok_or_else(|| {
+format_err!("IPSET in config, but no scope set in parser
config")
+})?;
+
+let ipset_name = IpsetName::new(scope, name.to_string());
+let mut ipset = Ipset::new(ipset_name);
+ipset.comment = comment.map(str::to_owned);
+
+this.set_section(&mut section, Sec::Ipset(name.to_string(),
ipset))?;
+} else if let Some(line) = line.strip_prefix("[group") {
+let (name, comment) = parse_named_section_tail("group", line)?;
+let mut group = Group::new();
+
+group.set_comment(comment.map(str::to_owned));
+
+this.set_section(&mut section, Sec::Group(name.to_owned(),
group))?;
+} else if line.starts_with('[') {
+bail!("invalid section {line:?}");
+} else {
+match &mut section {
+Sec::None => bail!("config line with no section:
{line:?}"),
+Sec::Options => Self::parse_option(line, &mut options)?,
+Sec::Aliases => this.parse_alias(line)?,
+Sec::Rules => this.parse_rule(line, parser_cfg)?,
+Sec::Ipset(_name, ipset) => ipset.parse_entry(line)?,
+Sec::Group(_name, group) => group.parse_entry(line)?,
+}
+}
+}
+this.set_section(&mut section, Sec::None)?;
+
+this.options = O::deserialize(IntoDeserializer::<
+'_,
+crate::firewall::parse::SerdeStringError,
+>::into_deserializer(options))?;
+
+Ok(this)
+}
+
+fn parse_option(line: &str, options: &mut HashMap) ->
Result<(), Error> {
+let (key, value) = split_key_value(line)
+.ok_or_else(|| format_err!("expected colon separated key and
value, found {line:?}"))?;
+
+if options.insert(key.to_string(), value.into()).is_some() {
+bail!("duplicate op
[pve-devel] [PATCH proxmox-firewall v2 13/39] config: firewall: add host specific config + option types
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/src/firewall/host.rs | 372 +
proxmox-ve-config/src/firewall/mod.rs | 1 +
2 files changed, 373 insertions(+)
create mode 100644 proxmox-ve-config/src/firewall/host.rs
diff --git a/proxmox-ve-config/src/firewall/host.rs
b/proxmox-ve-config/src/firewall/host.rs
new file mode 100644
index 000..2fd1f36
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/host.rs
@@ -0,0 +1,372 @@
+use std::io;
+use std::net::IpAddr;
+
+use anyhow::{bail, Error};
+use serde::Deserialize;
+
+use crate::host::utils::{host_ips, network_interface_cidrs};
+use proxmox_sys::nodename;
+
+use crate::firewall::parse;
+use crate::firewall::types::log::LogLevel;
+use crate::firewall::types::rule::Direction;
+use crate::firewall::types::{Alias, Cidr, Rule};
+
+/// default setting for the enabled key
+pub const HOST_ENABLED_DEFAULT: bool = true;
+/// default setting for the nftables key
+pub const HOST_NFTABLES_DEFAULT: bool = false;
+/// default return value for [`Config::allow_ndp()`]
+pub const HOST_ALLOW_NDP_DEFAULT: bool = true;
+/// default return value for [`Config::block_smurfs()`]
+pub const HOST_BLOCK_SMURFS_DEFAULT: bool = true;
+/// default return value for [`Config::block_synflood()`]
+pub const HOST_BLOCK_SYNFLOOD_DEFAULT: bool = false;
+/// default rate limit for synflood rule (packets / second)
+pub const HOST_BLOCK_SYNFLOOD_RATE_DEFAULT: i64 = 200;
+/// default rate limit for synflood rule (packets / second)
+pub const HOST_BLOCK_SYNFLOOD_BURST_DEFAULT: i64 = 1000;
+/// default return value for [`Config::block_invalid_tcp()`]
+pub const HOST_BLOCK_INVALID_TCP_DEFAULT: bool = false;
+/// default return value for [`Config::block_invalid_conntrack()`]
+pub const HOST_BLOCK_INVALID_CONNTRACK: bool = false;
+/// default setting for logging of invalid conntrack entries
+pub const HOST_LOG_INVALID_CONNTRACK: bool = false;
+
+#[derive(Debug, Default, Deserialize)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub struct Options {
+#[serde(default, with = "parse::serde_option_bool")]
+enable: Option,
+
+#[serde(default, with = "parse::serde_option_bool")]
+nftables: Option,
+
+log_level_in: Option,
+log_level_out: Option,
+
+#[serde(default, with = "parse::serde_option_bool")]
+log_nf_conntrack: Option,
+#[serde(default, with = "parse::serde_option_bool")]
+ndp: Option,
+
+#[serde(default, with = "parse::serde_option_bool")]
+nf_conntrack_allow_invalid: Option,
+
+// is Option> for easier deserialization
+#[serde(default, with = "parse::serde_option_conntrack_helpers")]
+nf_conntrack_helpers: Option>,
+
+#[serde(default, with = "parse::serde_option_number")]
+nf_conntrack_max: Option,
+#[serde(default, with = "parse::serde_option_number")]
+nf_conntrack_tcp_timeout_established: Option,
+#[serde(default, with = "parse::serde_option_number")]
+nf_conntrack_tcp_timeout_syn_recv: Option,
+
+#[serde(default, with = "parse::serde_option_bool")]
+nosmurfs: Option,
+
+#[serde(default, with = "parse::serde_option_bool")]
+protection_synflood: Option,
+#[serde(default, with = "parse::serde_option_number")]
+protection_synflood_burst: Option,
+#[serde(default, with = "parse::serde_option_number")]
+protection_synflood_rate: Option,
+
+smurf_log_level: Option,
+tcp_flags_log_level: Option,
+
+#[serde(default, with = "parse::serde_option_bool")]
+tcpflags: Option,
+}
+
+#[derive(Debug, Default)]
+pub struct Config {
+pub(crate) config: super::common::Config,
+}
+
+impl Config {
+pub fn new() -> Self {
+Self {
+config: Default::default(),
+}
+}
+
+pub fn parse(input: R) -> Result {
+let config = super::common::Config::parse(input, &Default::default())?;
+
+if !config.groups.is_empty() {
+bail!("host firewall config cannot declare groups");
+}
+
+if !config.aliases.is_empty() {
+bail!("host firewall config cannot declare aliases");
+}
+
+if !config.ipsets.is_empty() {
+bail!("host firewall config cannot declare ipsets");
+}
+
+Ok(Self { config })
+}
+
+pub fn rules(&self) -> &[Rule] {
+&self.config.rules
+}
+
+pub fn management_ips() -> Result, Error> {
+let mut management_cidrs = Vec::new();
+
+for host_ip in host_ips() {
+for network_interface_cidr in network_interface_cidrs() {
+match (host_ip, network_interface_cidr) {
+(IpAddr::V4(ip), Cidr::Ipv4(cidr)) => {
+if cidr.contains_address(ip) {
+management_cidrs.push(*network_interface_cidr);
+}
+}
+(IpAddr::V6(ip), Cidr::Ipv6(cidr)) => {
+
[pve-devel] [PATCH proxmox-firewall v2 14/39] config: firewall: add guest-specific config + option types
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/src/firewall/guest.rs | 237
proxmox-ve-config/src/firewall/mod.rs | 1 +
2 files changed, 238 insertions(+)
create mode 100644 proxmox-ve-config/src/firewall/guest.rs
diff --git a/proxmox-ve-config/src/firewall/guest.rs
b/proxmox-ve-config/src/firewall/guest.rs
new file mode 100644
index 000..c7e282f
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/guest.rs
@@ -0,0 +1,237 @@
+use std::collections::BTreeMap;
+use std::io;
+
+use crate::guest::types::Vmid;
+use crate::guest::vm::NetworkConfig;
+
+use crate::firewall::types::alias::{Alias, AliasName};
+use crate::firewall::types::ipset::IpsetScope;
+use crate::firewall::types::log::LogLevel;
+use crate::firewall::types::rule::{Direction, Rule, Verdict};
+use crate::firewall::types::Ipset;
+
+use anyhow::{bail, Error};
+use serde::Deserialize;
+
+use crate::firewall::parse::serde_option_bool;
+
+/// default return value for [`Config::is_enabled()`]
+pub const GUEST_ENABLED_DEFAULT: bool = false;
+/// default return value for [`Config::allow_ndp()`]
+pub const GUEST_ALLOW_NDP_DEFAULT: bool = true;
+/// default return value for [`Config::allow_dhcp()`]
+pub const GUEST_ALLOW_DHCP_DEFAULT: bool = true;
+/// default return value for [`Config::allow_ra()`]
+pub const GUEST_ALLOW_RA_DEFAULT: bool = false;
+/// default return value for [`Config::macfilter()`]
+pub const GUEST_MACFILTER_DEFAULT: bool = true;
+/// default return value for [`Config::ipfilter()`]
+pub const GUEST_IPFILTER_DEFAULT: bool = false;
+/// default return value for [`Config::default_policy()`]
+pub const GUEST_POLICY_IN_DEFAULT: Verdict = Verdict::Drop;
+/// default return value for [`Config::default_policy()`]
+pub const GUEST_POLICY_OUT_DEFAULT: Verdict = Verdict::Accept;
+
+#[derive(Debug, Default, Deserialize)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub struct Options {
+#[serde(default, with = "serde_option_bool")]
+dhcp: Option,
+
+#[serde(default, with = "serde_option_bool")]
+enable: Option,
+
+#[serde(default, with = "serde_option_bool")]
+ipfilter: Option,
+
+#[serde(default, with = "serde_option_bool")]
+ndp: Option,
+
+#[serde(default, with = "serde_option_bool")]
+radv: Option,
+
+log_level_in: Option,
+log_level_out: Option,
+
+#[serde(default, with = "serde_option_bool")]
+macfilter: Option,
+
+#[serde(rename = "policy_in")]
+policy_in: Option,
+
+#[serde(rename = "policy_out")]
+policy_out: Option,
+}
+
+#[derive(Debug)]
+pub struct Config {
+vmid: Vmid,
+
+/// The interface prefix: "veth" for containers, "tap" for VMs.
+iface_prefix: &'static str,
+
+network_config: NetworkConfig,
+config: super::common::Config,
+}
+
+impl Config {
+pub fn parse(
+vmid: &Vmid,
+iface_prefix: &'static str,
+firewall_input: T,
+network_input: U,
+) -> Result {
+let parser_cfg = super::common::ParserConfig {
+guest_iface_names: true,
+ipset_scope: Some(IpsetScope::Guest),
+};
+
+let config = super::common::Config::parse(firewall_input,
&parser_cfg)?;
+if !config.groups.is_empty() {
+bail!("guest firewall config cannot declare groups");
+}
+
+let network_config = NetworkConfig::parse(network_input)?;
+
+Ok(Self {
+vmid: *vmid,
+iface_prefix,
+config,
+network_config,
+})
+}
+
+pub fn vmid(&self) -> Vmid {
+self.vmid
+}
+
+pub fn alias(&self, name: &AliasName) -> Option<&Alias> {
+self.config.alias(name.name())
+}
+
+pub fn iface_name_by_key(&self, key: &str) -> Result {
+let index = NetworkConfig::index_from_net_key(key)?;
+Ok(format!("{}{}i{index}", self.iface_prefix, self.vmid))
+}
+
+pub fn iface_name_by_index(&self, index: i64) -> String {
+format!("{}{}i{index}", self.iface_prefix, self.vmid)
+}
+
+/// returns the value of the enabled config key or
[`GUEST_ENABLED_DEFAULT`] if unset
+pub fn is_enabled(&self) -> bool {
+self.config.options.enable.unwrap_or(GUEST_ENABLED_DEFAULT)
+}
+
+pub fn rules(&self) -> &[Rule] {
+&self.config.rules
+}
+
+pub fn log_level(&self, dir: Direction) -> LogLevel {
+match dir {
+Direction::In =>
self.config.options.log_level_in.unwrap_or_default(),
+Direction::Out =>
self.config.options.log_level_out.unwrap_or_default(),
+}
+}
+
+/// returns the value of the ndp config key or [`GUEST_ALLOW_NDP_DEFAULT`]
if unset
+pub fn allow_ndp(&self) -> bool {
+self.config.options.ndp.unwrap_or(GUEST_ALLOW_NDP_DEFAULT)
+}
+
+/// returns the value of the dhcp config key or
[`GUEST_ALLOW_DHCP_DEFAULT`] if unset
+pub fn allow_dhcp(&se
[pve-devel] [PATCH proxmox-firewall v2 08/39] config: firewall: add types for ipsets
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/src/firewall/types/ipset.rs | 349 ++
proxmox-ve-config/src/firewall/types/mod.rs | 2 +
2 files changed, 351 insertions(+)
create mode 100644 proxmox-ve-config/src/firewall/types/ipset.rs
diff --git a/proxmox-ve-config/src/firewall/types/ipset.rs
b/proxmox-ve-config/src/firewall/types/ipset.rs
new file mode 100644
index 000..c1af642
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/types/ipset.rs
@@ -0,0 +1,349 @@
+use core::fmt::Display;
+use std::ops::{Deref, DerefMut};
+use std::str::FromStr;
+
+use anyhow::{bail, format_err, Error};
+use serde_with::DeserializeFromStr;
+
+use crate::firewall::parse::match_non_whitespace;
+use crate::firewall::types::address::Cidr;
+use crate::firewall::types::alias::AliasName;
+use crate::guest::vm::NetworkConfig;
+
+#[derive(Debug, Clone, Copy, Eq, PartialEq)]
+pub enum IpsetScope {
+Datacenter,
+Guest,
+}
+
+impl FromStr for IpsetScope {
+type Err = Error;
+
+fn from_str(s: &str) -> Result {
+Ok(match s {
+"+dc" => IpsetScope::Datacenter,
+"+guest" => IpsetScope::Guest,
+_ => bail!("invalid scope for ipset: {s}"),
+})
+}
+}
+
+impl Display for IpsetScope {
+fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+let prefix = match self {
+Self::Datacenter => "dc",
+Self::Guest => "guest",
+};
+
+f.write_str(prefix)
+}
+}
+
+#[derive(Debug, Clone, DeserializeFromStr)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub struct IpsetName {
+pub scope: IpsetScope,
+pub name: String,
+}
+
+impl IpsetName {
+pub fn new(scope: IpsetScope, name: impl Into) -> Self {
+Self {
+scope,
+name: name.into(),
+}
+}
+
+pub fn name(&self) -> &str {
+&self.name
+}
+
+pub fn scope(&self) -> IpsetScope {
+self.scope
+}
+}
+
+impl FromStr for IpsetName {
+type Err = Error;
+
+fn from_str(s: &str) -> Result {
+match s.split_once('/') {
+Some((prefix, name)) if !name.is_empty() => Ok(Self {
+scope: prefix.parse()?,
+name: name.to_string(),
+}),
+_ => {
+bail!("Invalid IPSet name: {s}")
+}
+}
+}
+}
+
+impl Display for IpsetName {
+fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+write!(f, "{}/{}", self.scope, self.name)
+}
+}
+
+#[derive(Debug)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub enum IpsetAddress {
+Alias(AliasName),
+Cidr(Cidr),
+}
+
+impl FromStr for IpsetAddress {
+type Err = Error;
+
+fn from_str(s: &str) -> Result {
+if let Ok(cidr) = s.parse() {
+return Ok(IpsetAddress::Cidr(cidr));
+}
+
+if let Ok(name) = s.parse() {
+return Ok(IpsetAddress::Alias(name));
+}
+
+bail!("Invalid address in IPSet: {s}")
+}
+}
+
+impl> From for IpsetAddress {
+fn from(cidr: T) -> Self {
+IpsetAddress::Cidr(cidr.into())
+}
+}
+
+#[derive(Debug)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub struct IpsetEntry {
+pub nomatch: bool,
+pub address: IpsetAddress,
+pub comment: Option,
+}
+
+impl> From for IpsetEntry {
+fn from(value: T) -> Self {
+Self {
+nomatch: false,
+address: value.into(),
+comment: None,
+}
+}
+}
+
+impl FromStr for IpsetEntry {
+type Err = Error;
+
+fn from_str(line: &str) -> Result {
+let line = line.trim_start();
+
+let (nomatch, line) = match line.strip_prefix('!') {
+Some(line) => (true, line),
+None => (false, line),
+};
+
+let (address, line) =
+match_non_whitespace(line.trim_start()).ok_or_else(||
format_err!("missing value"))?;
+
+let address: IpsetAddress = address.parse()?;
+let line = line.trim_start();
+
+let comment = match line.strip_prefix('#') {
+Some(comment) => Some(comment.trim().to_string()),
+None if !line.is_empty() => bail!("trailing characters in ipset
entry: {line:?}"),
+None => None,
+};
+
+Ok(Self {
+nomatch,
+address,
+comment,
+})
+}
+}
+
+#[derive(Debug)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub struct Ipfilter<'a> {
+index: i64,
+ipset: &'a Ipset,
+}
+
+impl Ipfilter<'_> {
+pub fn index(&self) -> i64 {
+self.index
+}
+
+pub fn ipset(&self) -> &Ipset {
+self.ipset
+}
+
+pub fn name_for_index(index: i64) -> String {
+format!("ipfilter-net{index}")
+}
+}
+
+#[derive(Debug)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub struct Ipset {
+pub name: IpsetName,
+set: Vec
[pve-devel] [PATCH proxmox-firewall v2 12/39] config: firewall: add cluster-specific config + option types
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/src/firewall/cluster.rs | 374 ++
proxmox-ve-config/src/firewall/mod.rs | 1 +
2 files changed, 375 insertions(+)
create mode 100644 proxmox-ve-config/src/firewall/cluster.rs
diff --git a/proxmox-ve-config/src/firewall/cluster.rs
b/proxmox-ve-config/src/firewall/cluster.rs
new file mode 100644
index 000..223124b
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/cluster.rs
@@ -0,0 +1,374 @@
+use std::collections::BTreeMap;
+use std::io;
+
+use anyhow::Error;
+use serde::Deserialize;
+
+use crate::firewall::common::ParserConfig;
+use crate::firewall::types::ipset::{Ipset, IpsetScope};
+use crate::firewall::types::log::LogRateLimit;
+use crate::firewall::types::rule::{Direction, Verdict};
+use crate::firewall::types::{Alias, Group, Rule};
+
+use crate::firewall::parse::{serde_option_bool, serde_option_log_ratelimit};
+
+#[derive(Debug, Default)]
+pub struct Config {
+pub(crate) config: super::common::Config,
+}
+
+/// default setting for [`Config::is_enabled()`]
+pub const CLUSTER_ENABLED_DEFAULT: bool = false;
+/// default setting for [`Config::ebtables()`]
+pub const CLUSTER_EBTABLES_DEFAULT: bool = false;
+/// default setting for [`Config::default_policy()`]
+pub const CLUSTER_POLICY_IN_DEFAULT: Verdict = Verdict::Drop;
+/// default setting for [`Config::default_policy()`]
+pub const CLUSTER_POLICY_OUT_DEFAULT: Verdict = Verdict::Accept;
+
+impl Config {
+pub fn parse(input: R) -> Result {
+let parser_config = ParserConfig {
+guest_iface_names: false,
+ipset_scope: Some(IpsetScope::Datacenter),
+};
+
+Ok(Self {
+config: super::common::Config::parse(input, &parser_config)?,
+})
+}
+
+pub fn rules(&self) -> &Vec {
+&self.config.rules
+}
+
+pub fn groups(&self) -> &BTreeMap {
+&self.config.groups
+}
+
+pub fn ipsets(&self) -> &BTreeMap {
+&self.config.ipsets
+}
+
+pub fn alias(&self, name: &str) -> Option<&Alias> {
+self.config.alias(name)
+}
+
+pub fn is_enabled(&self) -> bool {
+self.config
+.options
+.enable
+.unwrap_or(CLUSTER_ENABLED_DEFAULT)
+}
+
+/// returns the ebtables option from the cluster config or
[`CLUSTER_EBTABLES_DEFAULT`] if
+/// unset
+///
+/// this setting is leftover from the old firewall, but has no effect on
the nftables firewall
+pub fn ebtables(&self) -> bool {
+self.config
+.options
+.ebtables
+.unwrap_or(CLUSTER_EBTABLES_DEFAULT)
+}
+
+/// returns policy_in / out or [`CLUSTER_POLICY_IN_DEFAULT`] /
[`CLUSTER_POLICY_OUT_DEFAULT`] if
+/// unset
+pub fn default_policy(&self, dir: Direction) -> Verdict {
+match dir {
+Direction::In => self
+.config
+.options
+.policy_in
+.unwrap_or(CLUSTER_POLICY_IN_DEFAULT),
+Direction::Out => self
+.config
+.options
+.policy_out
+.unwrap_or(CLUSTER_POLICY_OUT_DEFAULT),
+}
+}
+
+/// returns the rate_limit for logs or [`None`] if rate limiting is
disabled
+///
+/// If there is no rate limit set, then [`LogRateLimit::default`] is used
+pub fn log_ratelimit(&self) -> Option {
+let rate_limit = self
+.config
+.options
+.log_ratelimit
+.clone()
+.unwrap_or_default();
+
+match rate_limit.enabled() {
+true => Some(rate_limit),
+false => None,
+}
+}
+}
+
+#[derive(Debug, Default, Deserialize)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub struct Options {
+#[serde(default, with = "serde_option_bool")]
+enable: Option,
+
+#[serde(default, with = "serde_option_bool")]
+ebtables: Option,
+
+#[serde(default, with = "serde_option_log_ratelimit")]
+log_ratelimit: Option,
+
+policy_in: Option,
+policy_out: Option,
+}
+
+#[cfg(test)]
+mod tests {
+use crate::firewall::types::{
+address::IpList,
+alias::{AliasName, AliasScope},
+ipset::{IpsetAddress, IpsetEntry},
+log::{LogLevel, LogRateLimitTimescale},
+rule::{Kind, RuleGroup},
+rule_match::{
+Icmpv6, Icmpv6Code, IpAddrMatch, IpMatch, Ports, Protocol,
RuleMatch, Tcp, Udp,
+},
+Cidr,
+};
+
+use super::*;
+
+#[test]
+fn test_parse_config() {
+const CONFIG: &str = r#"
+[OPTIONS]
+enable: 1
+log_ratelimit: 1,rate=10/second,burst=20
+ebtables: 0
+policy_in: REJECT
+policy_out: REJECT
+
+[ALIASES]
+
+another 8.8.8.18
+analias 7.7.0.0/16 # much
+wide ::/64
+
+[IPSET a-set]
+
+!5.5.5.5
+1.2.3.4/30
+dc/analias # a comment
+dc/wide
+::/96
+
+[
[pve-devel] [PATCH proxmox-firewall v2 17/39] nftables: add crate for libnftables bindings
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
Cargo.toml | 1 +
proxmox-nftables/Cargo.toml | 16
proxmox-nftables/src/lib.rs | 0
3 files changed, 17 insertions(+)
create mode 100644 proxmox-nftables/Cargo.toml
create mode 100644 proxmox-nftables/src/lib.rs
diff --git a/Cargo.toml b/Cargo.toml
index a8d33ab..877f103 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,4 +1,5 @@
[workspace]
members = [
"proxmox-ve-config",
+"proxmox-nftables",
]
diff --git a/proxmox-nftables/Cargo.toml b/proxmox-nftables/Cargo.toml
new file mode 100644
index 000..764e231
--- /dev/null
+++ b/proxmox-nftables/Cargo.toml
@@ -0,0 +1,16 @@
+[package]
+name = "proxmox-nftables"
+version = "0.1.0"
+edition = "2021"
+authors = [
+"Wolfgang Bumiller ",
+"Stefan Hanreich ",
+"Proxmox Support Team ",
+]
+description = "Proxmox VE nftables"
+license = "AGPL-3"
+
+[dependencies]
+log = "0.4"
+
+proxmox-ve-config = { path = "../proxmox-ve-config", optional = true }
diff --git a/proxmox-nftables/src/lib.rs b/proxmox-nftables/src/lib.rs
new file mode 100644
index 000..e69de29
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-firewall v2 27/39] firewall: add base ruleset
This is the skeleton for the firewall that contains all the base
chains required for the firewall.
The file applies atomically, which means that it flushes all objects
and recreates them - except for the cluster/host/guest chain. This
means that it can be run at any point in time, since it only updates
the chains that are not managed by the firewall itself.
This also means that when we change the rules in the chains (e.g.
during an update) we can always just re-run the nft-file and the
firewall should use the new chains while still retaining the
configuration generated by the firewall daemon.
This also means that when re-creating the firewall rules, the
cluster/host/guest chains need to be flushed manually before creating
new rules.
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
.../resources/proxmox-firewall.nft| 305 ++
1 file changed, 305 insertions(+)
create mode 100644 proxmox-firewall/resources/proxmox-firewall.nft
diff --git a/proxmox-firewall/resources/proxmox-firewall.nft
b/proxmox-firewall/resources/proxmox-firewall.nft
new file mode 100644
index 000..67dd8c8
--- /dev/null
+++ b/proxmox-firewall/resources/proxmox-firewall.nft
@@ -0,0 +1,305 @@
+#!/usr/sbin/nft -f
+
+define ipv6_mask = :::::
+
+add table inet proxmox-firewall
+add table bridge proxmox-firewall-guests
+
+add chain inet proxmox-firewall do-reject
+add chain inet proxmox-firewall accept-management
+add chain inet proxmox-firewall block-synflood
+add chain inet proxmox-firewall log-drop-invalid-tcp
+add chain inet proxmox-firewall block-invalid-tcp
+add chain inet proxmox-firewall allow-ndp-in
+add chain inet proxmox-firewall block-ndp-in
+add chain inet proxmox-firewall allow-ndp-out
+add chain inet proxmox-firewall block-ndp-out
+add chain inet proxmox-firewall block-conntrack-invalid
+add chain inet proxmox-firewall block-smurfs
+add chain inet proxmox-firewall log-drop-smurfs
+add chain inet proxmox-firewall default-in
+add chain inet proxmox-firewall default-out
+add chain inet proxmox-firewall input {type filter hook input priority filter;
policy drop;}
+add chain inet proxmox-firewall output {type filter hook output priority
filter; policy accept;}
+
+add chain bridge proxmox-firewall-guests allow-dhcp-in
+add chain bridge proxmox-firewall-guests allow-dhcp-out
+add chain bridge proxmox-firewall-guests block-dhcp-in
+add chain bridge proxmox-firewall-guests block-dhcp-out
+add chain bridge proxmox-firewall-guests allow-ndp-in
+add chain bridge proxmox-firewall-guests block-ndp-in
+add chain bridge proxmox-firewall-guests allow-ndp-out
+add chain bridge proxmox-firewall-guests block-ndp-out
+add chain bridge proxmox-firewall-guests allow-ra-out
+add chain bridge proxmox-firewall-guests block-ra-out
+add chain bridge proxmox-firewall-guests after-vm-in
+add chain bridge proxmox-firewall-guests do-reject
+add chain bridge proxmox-firewall-guests vm-out {type filter hook prerouting
priority 0; policy accept;}
+add chain bridge proxmox-firewall-guests vm-in {type filter hook postrouting
priority 0; policy accept;}
+
+flush chain inet proxmox-firewall do-reject
+flush chain inet proxmox-firewall accept-management
+flush chain inet proxmox-firewall block-synflood
+flush chain inet proxmox-firewall log-drop-invalid-tcp
+flush chain inet proxmox-firewall block-invalid-tcp
+flush chain inet proxmox-firewall allow-ndp-in
+flush chain inet proxmox-firewall block-ndp-in
+flush chain inet proxmox-firewall allow-ndp-out
+flush chain inet proxmox-firewall block-ndp-out
+flush chain inet proxmox-firewall block-conntrack-invalid
+flush chain inet proxmox-firewall block-smurfs
+flush chain inet proxmox-firewall log-drop-smurfs
+flush chain inet proxmox-firewall default-in
+flush chain inet proxmox-firewall default-out
+flush chain inet proxmox-firewall input
+flush chain inet proxmox-firewall output
+
+flush chain bridge proxmox-firewall-guests allow-dhcp-in
+flush chain bridge proxmox-firewall-guests allow-dhcp-out
+flush chain bridge proxmox-firewall-guests block-dhcp-in
+flush chain bridge proxmox-firewall-guests block-dhcp-out
+flush chain bridge proxmox-firewall-guests allow-ndp-in
+flush chain bridge proxmox-firewall-guests block-ndp-in
+flush chain bridge proxmox-firewall-guests allow-ndp-out
+flush chain bridge proxmox-firewall-guests block-ndp-out
+flush chain bridge proxmox-firewall-guests allow-ra-out
+flush chain bridge proxmox-firewall-guests block-ra-out
+flush chain bridge proxmox-firewall-guests after-vm-in
+flush chain bridge proxmox-firewall-guests do-reject
+flush chain bridge proxmox-firewall-guests vm-out
+flush chain bridge proxmox-firewall-guests vm-in
+
+table inet proxmox-firewall {
+chain do-reject {
+meta pkttype broadcast drop
+ip saddr 224.0.0.0/4 drop
+
+meta l4proto tcp reject with tcp reset
+meta l4proto icmp reject with icmp type port-unreachable
+reject with icmp type host-prohibited
+
[pve-devel] [PATCH proxmox-firewall v2 31/39] firewall: add ruleset generation logic
We create the rules from the firewall config by utilizing the
ToNftRules and ToNftObjects traits to convert the firewall config
structs to nftables objects/chains/rules.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-firewall/Cargo.toml | 3 +
proxmox-firewall/src/firewall.rs | 899 +++
proxmox-firewall/src/main.rs | 1 +
3 files changed, 903 insertions(+)
create mode 100644 proxmox-firewall/src/firewall.rs
diff --git a/proxmox-firewall/Cargo.toml b/proxmox-firewall/Cargo.toml
index 431e71a..1e6a4b8 100644
--- a/proxmox-firewall/Cargo.toml
+++ b/proxmox-firewall/Cargo.toml
@@ -15,5 +15,8 @@ log = "0.4"
env_logger = "0.10"
anyhow = "1"
+serde = { version = "1", features = [ "derive" ] }
+serde_json = "1"
+
proxmox-nftables = { path = "../proxmox-nftables", features = ["config-ext"] }
proxmox-ve-config = { path = "../proxmox-ve-config" }
diff --git a/proxmox-firewall/src/firewall.rs b/proxmox-firewall/src/firewall.rs
new file mode 100644
index 000..1279a81
--- /dev/null
+++ b/proxmox-firewall/src/firewall.rs
@@ -0,0 +1,899 @@
+use std::collections::BTreeMap;
+use std::fs;
+
+use anyhow::Error;
+
+use proxmox_nftables::command::{Add, Commands, Delete, Flush};
+use proxmox_nftables::expression::{Meta, Payload};
+use proxmox_nftables::helper::NfVec;
+use proxmox_nftables::statement::{AnonymousLimit, Log, LogLevel, Match, Set,
SetOperation};
+use proxmox_nftables::types::{
+AddElement, AddRule, ChainPart, MapValue, RateTimescale, SetName,
TableFamily, TableName,
+TablePart, Verdict,
+};
+use proxmox_nftables::{Expression, Statement};
+
+use proxmox_ve_config::firewall::ct_helper::get_cthelper;
+use proxmox_ve_config::firewall::guest::Config as GuestConfig;
+use proxmox_ve_config::firewall::host::Config as HostConfig;
+
+use proxmox_ve_config::firewall::types::address::Ipv6Cidr;
+use proxmox_ve_config::firewall::types::ipset::{
+Ipfilter, Ipset, IpsetEntry, IpsetName, IpsetScope,
+};
+use proxmox_ve_config::firewall::types::log::{LogLevel as ConfigLogLevel,
LogRateLimit};
+use proxmox_ve_config::firewall::types::rule::{Direction, Verdict as
ConfigVerdict};
+use proxmox_ve_config::firewall::types::Group;
+use proxmox_ve_config::guest::types::Vmid;
+
+use crate::config::FirewallConfig;
+use crate::object::{NftObjectEnv, ToNftObjects};
+use crate::rule::{NftRule, NftRuleEnv};
+
+static CLUSTER_TABLE_NAME: &str = "proxmox-firewall";
+static HOST_TABLE_NAME: &str = "proxmox-firewall";
+static GUEST_TABLE_NAME: &str = "proxmox-firewall-guests";
+
+static NF_CONNTRACK_MAX_FILE: &str =
"/proc/sys/net/netfilter/nf_conntrack_max";
+static NF_CONNTRACK_TCP_TIMEOUT_ESTABLISHED: &str =
+"/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established";
+static NF_CONNTRACK_TCP_TIMEOUT_SYN_RECV: &str =
+"/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_syn_recv";
+static LOG_CONNTRACK_FILE: &str = "/var/lib/pve-firewall/log_nf_conntrack";
+
+#[derive(Default)]
+pub struct Firewall {
+config: FirewallConfig,
+}
+
+impl From for Firewall {
+fn from(config: FirewallConfig) -> Self {
+Self { config }
+}
+}
+
+impl Firewall {
+pub fn new() -> Self {
+Self {
+..Default::default()
+}
+}
+
+pub fn is_enabled(&self) -> bool {
+self.config.is_enabled()
+}
+
+fn cluster_table(&self) -> TablePart {
+TablePart::new(TableFamily::Inet, CLUSTER_TABLE_NAME)
+}
+
+fn host_table(&self) -> TablePart {
+TablePart::new(TableFamily::Inet, HOST_TABLE_NAME)
+}
+
+fn guest_table(&self) -> TablePart {
+TablePart::new(TableFamily::Bridge, GUEST_TABLE_NAME)
+}
+
+fn guest_vmap(&self, dir: Direction) -> SetName {
+SetName::new(self.guest_table(), format!("vm-map-{dir}"))
+}
+
+fn cluster_chain(&self, dir: Direction) -> ChainPart {
+ChainPart::new(self.cluster_table(), format!("cluster-{dir}"))
+}
+
+fn host_chain(&self, dir: Direction) -> ChainPart {
+ChainPart::new(self.host_table(), format!("host-{dir}"))
+}
+
+fn guest_chain(&self, dir: Direction, vmid: Vmid) -> ChainPart {
+ChainPart::new(self.guest_table(), format!("guest-{vmid}-{dir}"))
+}
+
+fn group_chain(&self, table: TablePart, name: &str, dir: Direction) ->
ChainPart {
+ChainPart::new(table, format!("group-{name}-{dir}"))
+}
+
+fn host_conntrack_chain(&self) -> ChainPart {
+ChainPart::new(self.host_table(), "ct-in".to_string())
+}
+
+fn host_option_chain(&self, dir: Direction) -> ChainPart {
+ChainPart::new(self.host_table(), format!("option-{dir}"))
+}
+
+fn synflood_limit_chain(&self) -> ChainPart {
+ChainPart::new(self.host_table(), "ratelimit-synflood")
+}
+
+fn log_invalid_tcp_chain(&self) -> ChainPart {
+ChainPart::new(self.host_table(), "log-invalid-tcp")
+}
+
+fn l
[pve-devel] [PATCH pve-firewall v2 37/39] add configuration option for new nftables firewall
Introduces new nftables configuration option that en/disables the new
nftables firewall.
pve-firewall reads this option and only generates iptables rules when
nftables is set to `0`. Conversely proxmox-firewall only generates
nftables rules when the option is set to `1`.
Signed-off-by: Stefan Hanreich
---
src/PVE/Firewall.pm | 20
1 file changed, 16 insertions(+), 4 deletions(-)
diff --git a/src/PVE/Firewall.pm b/src/PVE/Firewall.pm
index 81a8798..b39843d 100644
--- a/src/PVE/Firewall.pm
+++ b/src/PVE/Firewall.pm
@@ -1408,6 +1408,12 @@ our $host_option_properties = {
default => 0,
optional => 1
},
+nftables => {
+ description => "Enable nftables based firewall",
+ type => 'boolean',
+ default => 0,
+ optional => 1,
+},
};
our $vm_option_properties = {
@@ -2929,7 +2935,7 @@ sub parse_hostfw_option {
my $loglevels = "emerg|alert|crit|err|warning|notice|info|debug|nolog";
-if ($line =~
m/^(enable|nosmurfs|tcpflags|ndp|log_nf_conntrack|nf_conntrack_allow_invalid|protection_synflood):\s*(0|1)\s*$/i)
{
+if ($line =~
m/^(enable|nosmurfs|tcpflags|ndp|log_nf_conntrack|nf_conntrack_allow_invalid|protection_synflood|nftables):\s*(0|1)\s*$/i)
{
$opt = lc($1);
$value = int($2);
} elsif ($line =~
m/^(log_level_in|log_level_out|tcp_flags_log_level|smurf_log_level):\s*(($loglevels)\s*)?$/i)
{
@@ -4676,7 +4682,11 @@ sub remove_pvefw_chains_ebtables {
sub init {
my $cluster_conf = load_clusterfw_conf();
my $cluster_options = $cluster_conf->{options};
-my $enable = $cluster_options->{enable};
+
+my $host_conf = load_hostfw_conf($cluster_conf);
+my $host_options = $host_conf->{options};
+
+my $enable = $cluster_options->{enable} && !$host_options->{nftables};
return if !$enable;
@@ -4689,12 +4699,14 @@ sub update {
my $cluster_conf = load_clusterfw_conf();
my $cluster_options = $cluster_conf->{options};
- if (!$cluster_options->{enable}) {
+ my $hostfw_conf = load_hostfw_conf($cluster_conf);
+ my $host_options = $hostfw_conf->{options};
+
+ if (!$cluster_options->{enable} || $host_options->{nftables}) {
PVE::Firewall::remove_pvefw_chains();
return;
}
- my $hostfw_conf = load_hostfw_conf($cluster_conf);
my ($ruleset, $ipset_ruleset, $rulesetv6, $ebtables_ruleset) =
compile($cluster_conf, $hostfw_conf);
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH pve-manager v2 38/39] firewall: expose configuration option for new nftables firewall
Signed-off-by: Stefan Hanreich
---
www/manager6/grid/FirewallOptions.js | 1 +
1 file changed, 1 insertion(+)
diff --git a/www/manager6/grid/FirewallOptions.js
b/www/manager6/grid/FirewallOptions.js
index 0ac9979c4..6aacb47be 100644
--- a/www/manager6/grid/FirewallOptions.js
+++ b/www/manager6/grid/FirewallOptions.js
@@ -83,6 +83,7 @@ Ext.define('PVE.FirewallOptions', {
add_log_row('log_level_out');
add_log_row('tcp_flags_log_level', 120);
add_log_row('smurf_log_level');
+ add_boolean_row('nftables', gettext('nftables (tech preview)'), 0);
} else if (me.fwtype === 'vm') {
me.rows.enable = {
required: true,
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-firewall v2 25/39] nftables: add libnftables bindings
Add a thin wrapper around libnftables, which can be used to run
commands defined by the rust types.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-nftables/src/context.rs | 243
proxmox-nftables/src/error.rs | 43 ++
proxmox-nftables/src/lib.rs | 3 +
3 files changed, 289 insertions(+)
create mode 100644 proxmox-nftables/src/context.rs
create mode 100644 proxmox-nftables/src/error.rs
diff --git a/proxmox-nftables/src/context.rs b/proxmox-nftables/src/context.rs
new file mode 100644
index 000..9ab51fb
--- /dev/null
+++ b/proxmox-nftables/src/context.rs
@@ -0,0 +1,243 @@
+use std::ffi::CString;
+use std::os::raw::{c_int, c_uint};
+use std::path::Path;
+
+use crate::command::{CommandOutput, Commands};
+use crate::error::NftError;
+
+#[rustfmt::skip]
+pub mod debug {
+use super::c_uint;
+
+pub const SCANNER: c_uint = 0x1;
+pub const PARSER : c_uint = 0x2;
+pub const EVALUATION : c_uint = 0x4;
+pub const NETLINK: c_uint = 0x8;
+pub const MNL: c_uint = 0x10;
+pub const PROTO_CTX : c_uint = 0x20;
+pub const SEGTREE: c_uint = 0x40;
+}
+
+#[rustfmt::skip]
+pub mod output {
+use super::c_uint;
+
+pub const REVERSEDNS : c_uint = 1;
+pub const SERVICE: c_uint = 1 << 1;
+pub const STATELESS : c_uint = 1 << 2;
+pub const HANDLE : c_uint = 1 << 3;
+pub const JSON : c_uint = 1 << 4;
+pub const ECHO : c_uint = 1 << 5;
+pub const GUID : c_uint = 1 << 6;
+pub const NUMERIC_PROTO : c_uint = 1 << 7;
+pub const NUMERIC_PRIO : c_uint = 1 << 8;
+pub const NUMERIC_SYMBOL : c_uint = 1 << 9;
+pub const NUMERIC_TIME : c_uint = 1 << 10;
+pub const TERSE : c_uint = 1 << 11;
+
+pub const NUMERIC_ALL: c_uint = NUMERIC_PROTO | NUMERIC_PRIO |
NUMERIC_SYMBOL;
+}
+
+#[link(name = "nftables")]
+extern "C" {
+fn nft_ctx_new(flags: u32) -> RawNftCtx;
+fn nft_ctx_free(ctx: RawNftCtx);
+
+//fn nft_ctx_get_dry_run(ctx: RawNftCtx) -> bool;
+fn nft_ctx_set_dry_run(ctx: RawNftCtx, dry: bool);
+
+fn nft_ctx_output_get_flags(ctx: RawNftCtx) -> c_uint;
+fn nft_ctx_output_set_flags(ctx: RawNftCtx, flags: c_uint);
+
+// fn nft_ctx_output_get_debug(ctx: RawNftCtx) -> c_uint;
+fn nft_ctx_output_set_debug(ctx: RawNftCtx, mask: c_uint);
+
+//fn nft_ctx_set_output(ctx: RawNftCtx, file: RawCFile) -> RawCFile;
+fn nft_ctx_buffer_output(ctx: RawNftCtx) -> c_int;
+fn nft_ctx_unbuffer_output(ctx: RawNftCtx) -> c_int;
+fn nft_ctx_get_output_buffer(ctx: RawNftCtx) -> *const i8;
+
+fn nft_ctx_buffer_error(ctx: RawNftCtx) -> c_int;
+fn nft_ctx_unbuffer_error(ctx: RawNftCtx) -> c_int;
+fn nft_ctx_get_error_buffer(ctx: RawNftCtx) -> *const i8;
+
+fn nft_run_cmd_from_buffer(ctx: RawNftCtx, buf: *const i8) -> c_int;
+fn nft_run_cmd_from_filename(ctx: RawNftCtx, filename: *const i8) -> c_int;
+}
+
+#[derive(Clone, Copy)]
+#[repr(transparent)]
+struct RawNftCtx(*mut u8);
+
+pub struct NftCtx(RawNftCtx);
+
+impl Drop for NftCtx {
+fn drop(&mut self) {
+if !self.0 .0.is_null() {
+unsafe {
+nft_ctx_free(self.0);
+}
+}
+}
+}
+
+impl NftCtx {
+pub fn new() -> Result {
+let mut this = Self(unsafe { nft_ctx_new(0) });
+
+if this.0 .0.is_null() {
+return Err(NftError::msg("failed to instantiate nft context"));
+}
+
+this.enable_json();
+
+Ok(this)
+}
+
+fn modify_flags(&mut self, func: impl FnOnce(c_uint) -> c_uint) {
+unsafe { nft_ctx_output_set_flags(self.0,
func(nft_ctx_output_get_flags(self.0))) }
+}
+
+pub fn enable_debug(&mut self) {
+unsafe { nft_ctx_output_set_debug(self.0, debug::PARSER |
debug::SCANNER) }
+}
+
+pub fn disable_debug(&mut self) {
+unsafe { nft_ctx_output_set_debug(self.0, 0) }
+}
+
+fn enable_json(&mut self) {
+self.modify_flags(|flags| flags | output::JSON);
+}
+
+pub fn set_dry_run(&mut self, on: bool) {
+unsafe { nft_ctx_set_dry_run(self.0, on) }
+}
+
+fn start_output_buffering(&mut self) -> Result<(), NftError> {
+let rc = unsafe { nft_ctx_buffer_output(self.0) };
+NftError::expect_zero(rc, || "failed to start output buffering")
+}
+
+fn stop_output_buffering(&mut self) {
+let _ = unsafe { nft_ctx_unbuffer_output(self.0) };
+// ignore errors
+}
+
+fn get_output_buffer(&mut self) -> Result {
+let buf = unsafe { nft_ctx_get_output_buffer(self.0) };
+
+if buf.is_null() {
+return Err(NftError::msg("failed to get output buffer"));
+}
+
+unsafe { std::ffi::CStr::from_ptr(buf) }
+.to_str()
+.map_err(NftError::msg)
+.map(str::to_string)
+}
[pve-devel] [PATCH proxmox-firewall v2 30/39] firewall: add object generation logic
ToNftObjects is basically a conversion trait that converts firewall
config structs into nftables objects. It returns a list of commands
that create the respective nftables objects.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-firewall/src/main.rs | 1 +
proxmox-firewall/src/object.rs | 140 +
2 files changed, 141 insertions(+)
create mode 100644 proxmox-firewall/src/object.rs
diff --git a/proxmox-firewall/src/main.rs b/proxmox-firewall/src/main.rs
index ae832e3..a4979a7 100644
--- a/proxmox-firewall/src/main.rs
+++ b/proxmox-firewall/src/main.rs
@@ -1,6 +1,7 @@
use anyhow::Error;
mod config;
+mod object;
mod rule;
fn main() -> Result<(), Error> {
diff --git a/proxmox-firewall/src/object.rs b/proxmox-firewall/src/object.rs
new file mode 100644
index 000..32c4ddb
--- /dev/null
+++ b/proxmox-firewall/src/object.rs
@@ -0,0 +1,140 @@
+use anyhow::{format_err, Error};
+use proxmox_nftables::{
+command::{Add, Flush},
+expression::Prefix,
+types::{
+AddCtHelper, AddElement, CtHelperProtocol, ElementType, L3Protocol,
SetConfig, SetFlag,
+SetName, TablePart,
+},
+Command, Expression,
+};
+use proxmox_ve_config::{
+firewall::{
+ct_helper::CtHelperMacro,
+types::{address::Family, alias::AliasName, ipset::IpsetAddress, Alias,
Ipset},
+},
+guest::types::Vmid,
+};
+
+use crate::config::FirewallConfig;
+
+pub(crate) struct NftObjectEnv<'a, 'b> {
+pub(crate) table: &'a TablePart,
+pub(crate) firewall_config: &'b FirewallConfig,
+pub(crate) vmid: Option,
+}
+
+impl NftObjectEnv<'_, '_> {
+pub(crate) fn alias(&self, name: &AliasName) -> Option<&Alias> {
+self.firewall_config.alias(name, self.vmid)
+}
+}
+
+pub(crate) trait ToNftObjects {
+fn to_nft_objects(&self, env: &NftObjectEnv) -> Result,
Error>;
+}
+
+impl ToNftObjects for CtHelperMacro {
+fn to_nft_objects(&self, env: &NftObjectEnv) -> Result,
Error> {
+let mut commands = Vec::new();
+
+if let Some(_protocol) = self.tcp() {
+commands.push(Add::ct_helper(AddCtHelper {
+table: env.table.clone(),
+name: self.tcp_helper_name(),
+ty: self.name().to_string(),
+protocol: CtHelperProtocol::TCP,
+l3proto: self.family().map(L3Protocol::from),
+}));
+}
+
+if let Some(_protocol) = self.udp() {
+commands.push(Add::ct_helper(AddCtHelper {
+table: env.table.clone(),
+name: self.udp_helper_name(),
+ty: self.name().to_string(),
+protocol: CtHelperProtocol::UDP,
+l3proto: self.family().map(L3Protocol::from),
+}));
+}
+
+Ok(commands)
+}
+}
+
+impl ToNftObjects for Ipset {
+fn to_nft_objects(&self, env: &NftObjectEnv) -> Result,
Error> {
+let mut commands = Vec::new();
+log::trace!("generating objects for ipset: {self:?}");
+
+for family in env.table.family().families() {
+let mut elements = Vec::new();
+let mut nomatch_elements = Vec::new();
+
+for element in self.iter() {
+let cidr = match &element.address {
+IpsetAddress::Cidr(cidr) => cidr,
+IpsetAddress::Alias(alias) => env
+.alias(alias)
+.ok_or(format_err!("could not find alias {alias} in
environment"))?
+.address(),
+};
+
+if family != cidr.family() {
+continue;
+}
+
+let expression = Expression::from(Prefix::from(cidr));
+
+if element.nomatch {
+nomatch_elements.push(expression);
+} else {
+elements.push(expression);
+}
+}
+
+let element_type = match family {
+Family::V4 => ElementType::Ipv4Addr,
+Family::V6 => ElementType::Ipv6Addr,
+};
+
+let set_name = SetName::new(
+env.table.clone(),
+SetName::ipset_name(family, self.name(), env.vmid, false),
+);
+
+let set_config =
+SetConfig::new(set_name.clone(),
vec![element_type]).with_flag(SetFlag::Interval);
+
+let nomatch_name = SetName::new(
+env.table.clone(),
+SetName::ipset_name(family, self.name(), env.vmid, true),
+);
+
+let nomatch_config = SetConfig::new(nomatch_name.clone(),
vec![element_type])
+.with_flag(SetFlag::Interval);
+
+commands.append(&mut vec![
+Add::set(set_config),
+Flush::set(set_name.clone()),
+Add::set(nomatch_con
[pve-devel] [PATCH proxmox-firewall v2 21/39] nftables: statement: add types
Adds an enum containing most of the statements defined in the
nftables-json schema [1].
[1]
https://manpages.debian.org/bookworm/libnftables1/libnftables-json.5.en.html#STATEMENTS
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-nftables/Cargo.toml | 1 +
proxmox-nftables/src/lib.rs | 2 +
proxmox-nftables/src/statement.rs | 321 ++
proxmox-nftables/src/types.rs | 18 +-
4 files changed, 341 insertions(+), 1 deletion(-)
create mode 100644 proxmox-nftables/src/statement.rs
diff --git a/proxmox-nftables/Cargo.toml b/proxmox-nftables/Cargo.toml
index 7e607e8..153716d 100644
--- a/proxmox-nftables/Cargo.toml
+++ b/proxmox-nftables/Cargo.toml
@@ -15,6 +15,7 @@ config-ext = ["dep:proxmox-ve-config"]
[dependencies]
log = "0.4"
+anyhow = "1"
serde = { version = "1", features = [ "derive" ] }
serde_json = "1"
diff --git a/proxmox-nftables/src/lib.rs b/proxmox-nftables/src/lib.rs
index 712858b..40f6bab 100644
--- a/proxmox-nftables/src/lib.rs
+++ b/proxmox-nftables/src/lib.rs
@@ -1,5 +1,7 @@
pub mod expression;
pub mod helper;
+pub mod statement;
pub mod types;
pub use expression::Expression;
+pub use statement::Statement;
diff --git a/proxmox-nftables/src/statement.rs
b/proxmox-nftables/src/statement.rs
new file mode 100644
index 000..e6371f6
--- /dev/null
+++ b/proxmox-nftables/src/statement.rs
@@ -0,0 +1,321 @@
+use anyhow::{bail, Error};
+use serde::{Deserialize, Serialize};
+
+use crate::expression::Meta;
+use crate::helper::{NfVec, Null};
+use crate::types::{RateTimescale, RateUnit, Verdict};
+use crate::Expression;
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum Statement {
+Match(Match),
+Mangle(Mangle),
+Limit(Limit),
+Notrack(Null),
+Reject(Reject),
+Set(Set),
+Log(Log),
+#[serde(rename = "ct helper")]
+CtHelper(String),
+Vmap(Vmap),
+Comment(String),
+
+#[serde(untagged)]
+Verdict(Verdict),
+}
+
+impl Statement {
+pub const fn make_accept() -> Self {
+Statement::Verdict(Verdict::Accept(Null))
+}
+
+pub const fn make_drop() -> Self {
+Statement::Verdict(Verdict::Drop(Null))
+}
+
+pub const fn make_return() -> Self {
+Statement::Verdict(Verdict::Return(Null))
+}
+
+pub const fn make_continue() -> Self {
+Statement::Verdict(Verdict::Continue(Null))
+}
+
+pub fn jump(target: impl Into) -> Self {
+Statement::Verdict(Verdict::Jump {
+target: target.into(),
+})
+}
+
+pub fn goto(target: impl Into) -> Self {
+Statement::Verdict(Verdict::Goto {
+target: target.into(),
+})
+}
+}
+
+impl From for Statement {
+#[inline]
+fn from(m: Match) -> Statement {
+Statement::Match(m)
+}
+}
+
+impl From for Statement {
+#[inline]
+fn from(m: Mangle) -> Statement {
+Statement::Mangle(m)
+}
+}
+
+impl From for Statement {
+#[inline]
+fn from(m: Reject) -> Statement {
+Statement::Reject(m)
+}
+}
+
+impl From for Statement {
+#[inline]
+fn from(m: Set) -> Statement {
+Statement::Set(m)
+}
+}
+
+impl From for Statement {
+#[inline]
+fn from(m: Vmap) -> Statement {
+Statement::Vmap(m)
+}
+}
+
+impl From for Statement {
+#[inline]
+fn from(log: Log) -> Statement {
+Statement::Log(log)
+}
+}
+
+impl> From for Statement {
+#[inline]
+fn from(limit: T) -> Statement {
+Statement::Limit(limit.into())
+}
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+#[serde(rename_all = "lowercase")]
+pub enum RejectType {
+#[serde(rename = "tcp reset")]
+TcpRst,
+IcmpX,
+Icmp,
+IcmpV6,
+}
+
+#[derive(Clone, Debug, Default, Deserialize, Serialize)]
+pub struct Reject {
+#[serde(rename = "type", skip_serializing_if = "Option::is_none")]
+ty: Option,
+#[serde(skip_serializing_if = "Option::is_none")]
+expr: Option,
+}
+
+#[derive(Clone, Debug, Default, Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct Log {
+#[serde(skip_serializing_if = "Option::is_none")]
+prefix: Option,
+
+#[serde(skip_serializing_if = "Option::is_none")]
+group: Option,
+
+#[serde(skip_serializing_if = "Option::is_none")]
+snaplen: Option,
+
+#[serde(skip_serializing_if = "Option::is_none")]
+queue_threshold: Option,
+
+#[serde(skip_serializing_if = "Option::is_none")]
+level: Option,
+
+#[serde(default, skip_serializing_if = "Vec::is_empty")]
+flags: NfVec,
+}
+
+impl Log {
+pub fn new_nflog(prefix: String, group: i64) -> Self {
+Self {
+prefix: Some(prefix),
+group: Some(group),
+..Default::default()
+}
+}
+}
+
+#[derive(Clone, Copy, Debug, Deserialize, Serialize)]
+#[serde(rename_all = "
[pve-devel] [PATCH proxmox-firewall v2 32/39] firewall: add proxmox-firewall binary
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-firewall/src/main.rs | 34 ++
1 file changed, 34 insertions(+)
diff --git a/proxmox-firewall/src/main.rs b/proxmox-firewall/src/main.rs
index 53c1289..bff71b9 100644
--- a/proxmox-firewall/src/main.rs
+++ b/proxmox-firewall/src/main.rs
@@ -5,7 +5,41 @@ mod firewall;
mod object;
mod rule;
+use firewall::Firewall;
+use proxmox_nftables::NftCtx;
+
+const RULE_BASE: &str = include_str!("../resources/proxmox-firewall.nft");
+
fn main() -> Result<(), Error> {
env_logger::init();
+
+let mut nft = NftCtx::new()?;
+let firewall = Firewall::new();
+
+if !firewall.is_enabled() {
+log::info!("Removing existing firewall rules");
+let commands = firewall.remove_firewall();
+
+// can ignore failures, since it fails when table does not exist
+let _ = nft.run_commands(&commands);
+
+return Ok(());
+}
+
+let commands = firewall.full_host_fw()?;
+
+log::info!("Running proxmox-firewall.nft");
+let got = nft.run_nft_commands(RULE_BASE)?;
+log::info!("got response from nftables: {got:?}");
+
+log::info!("Running proxmox-firewall commands");
+
+for (idx, c) in commands.iter().enumerate() {
+log::debug!("cmd #{idx} {}", serde_json::to_string(&c)?);
+}
+
+let got = nft.run_commands(&commands)?;
+log::info!("got response from nftables: {got:?}");
+
Ok(())
}
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-firewall v2 02/39] config: firewall: add types for ip addresses
Includes types for all kinds of IP values that can occur in the
firewall config. Additionally, FromStr implementations are available
for parsing from the config files.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/src/firewall/mod.rs | 1 +
.../src/firewall/types/address.rs | 615 ++
proxmox-ve-config/src/firewall/types/mod.rs | 3 +
proxmox-ve-config/src/lib.rs | 1 +
4 files changed, 620 insertions(+)
create mode 100644 proxmox-ve-config/src/firewall/mod.rs
create mode 100644 proxmox-ve-config/src/firewall/types/address.rs
create mode 100644 proxmox-ve-config/src/firewall/types/mod.rs
diff --git a/proxmox-ve-config/src/firewall/mod.rs
b/proxmox-ve-config/src/firewall/mod.rs
new file mode 100644
index 000..cd40856
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/mod.rs
@@ -0,0 +1 @@
+pub mod types;
diff --git a/proxmox-ve-config/src/firewall/types/address.rs
b/proxmox-ve-config/src/firewall/types/address.rs
new file mode 100644
index 000..e48ac1b
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/types/address.rs
@@ -0,0 +1,615 @@
+use std::fmt;
+use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
+use std::ops::Deref;
+
+use anyhow::{bail, format_err, Error};
+use serde_with::DeserializeFromStr;
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub enum Family {
+V4,
+V6,
+}
+
+impl fmt::Display for Family {
+fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+match self {
+Family::V4 => f.write_str("Ipv4"),
+Family::V6 => f.write_str("Ipv6"),
+}
+}
+}
+
+#[derive(Clone, Copy, Debug)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub enum Cidr {
+Ipv4(Ipv4Cidr),
+Ipv6(Ipv6Cidr),
+}
+
+impl Cidr {
+pub fn new_v4(addr: impl Into, mask: u8) -> Result {
+Ok(Cidr::Ipv4(Ipv4Cidr::new(addr, mask)?))
+}
+
+pub fn new_v6(addr: impl Into, mask: u8) -> Result {
+Ok(Cidr::Ipv6(Ipv6Cidr::new(addr, mask)?))
+}
+
+pub const fn family(&self) -> Family {
+match self {
+Cidr::Ipv4(_) => Family::V4,
+Cidr::Ipv6(_) => Family::V6,
+}
+}
+
+pub fn is_ipv4(&self) -> bool {
+matches!(self, Cidr::Ipv4(_))
+}
+
+pub fn is_ipv6(&self) -> bool {
+matches!(self, Cidr::Ipv6(_))
+}
+}
+
+impl fmt::Display for Cidr {
+fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+match self {
+Self::Ipv4(ip) => f.write_str(ip.to_string().as_str()),
+Self::Ipv6(ip) => f.write_str(ip.to_string().as_str()),
+}
+}
+}
+
+impl std::str::FromStr for Cidr {
+type Err = Error;
+
+fn from_str(s: &str) -> Result {
+if let Ok(ip) = s.parse::() {
+return Ok(Cidr::Ipv4(ip));
+}
+
+if let Ok(ip) = s.parse::() {
+return Ok(Cidr::Ipv6(ip));
+}
+
+bail!("invalid ip address or CIDR: {s:?}");
+}
+}
+
+impl From for Cidr {
+fn from(cidr: Ipv4Cidr) -> Self {
+Cidr::Ipv4(cidr)
+}
+}
+
+impl From for Cidr {
+fn from(cidr: Ipv6Cidr) -> Self {
+Cidr::Ipv6(cidr)
+}
+}
+
+const IPV4_LENGTH: u8 = 32;
+
+#[derive(Clone, Copy, Debug)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub struct Ipv4Cidr {
+addr: Ipv4Addr,
+mask: u8,
+}
+
+impl Ipv4Cidr {
+pub fn new(addr: impl Into, mask: u8) -> Result {
+if mask > 32 {
+bail!("mask out of range for ipv4 cidr ({mask})");
+}
+
+Ok(Self {
+addr: addr.into(),
+mask,
+})
+}
+
+pub fn contains_address(&self, other: &Ipv4Addr) -> bool {
+let bits = u32::from_be_bytes(self.addr.octets());
+let other_bits = u32::from_be_bytes(other.octets());
+
+let shift_amount: u32 = IPV4_LENGTH.saturating_sub(self.mask).into();
+
+bits.checked_shr(shift_amount).unwrap_or(0)
+== other_bits.checked_shr(shift_amount).unwrap_or(0)
+}
+
+pub fn address(&self) -> &Ipv4Addr {
+&self.addr
+}
+
+pub fn mask(&self) -> u8 {
+self.mask
+}
+}
+
+impl> From for Ipv4Cidr {
+fn from(value: T) -> Self {
+Self {
+addr: value.into(),
+mask: 32,
+}
+}
+}
+
+impl std::str::FromStr for Ipv4Cidr {
+type Err = Error;
+
+fn from_str(s: &str) -> Result {
+Ok(match s.find('/') {
+None => Self {
+addr: s.parse()?,
+mask: 32,
+},
+Some(pos) => {
+let mask: u8 = s[(pos + 1)..]
+.parse()
+.map_err(|_| format_err!("invalid mask in ipv4 cidr:
{s:?}"))?;
+
+Self::new(s[..pos].parse::()?, mask)?
+}
+})
+}
+}
+
+impl fmt::Display for Ipv4Cidr {
+fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+
[pve-devel] [PATCH proxmox-firewall v2 05/39] config: firewall: add types for aliases
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/src/firewall/parse.rs | 52 ++
proxmox-ve-config/src/firewall/types/alias.rs | 160 ++
proxmox-ve-config/src/firewall/types/mod.rs | 2 +
3 files changed, 214 insertions(+)
create mode 100644 proxmox-ve-config/src/firewall/types/alias.rs
diff --git a/proxmox-ve-config/src/firewall/parse.rs
b/proxmox-ve-config/src/firewall/parse.rs
index a75daee..772e081 100644
--- a/proxmox-ve-config/src/firewall/parse.rs
+++ b/proxmox-ve-config/src/firewall/parse.rs
@@ -1,5 +1,57 @@
use anyhow::{bail, format_err, Error};
+/// Parses out a "name" which can be alphanumeric and include dashes.
+///
+/// Returns `None` if the name part would be empty.
+///
+/// Returns a tuple with the name and the remainder (not trimmed).
+///
+/// # Examples
+/// ```ignore
+/// assert_eq!(match_name("some-name someremainder"), Some(("some-name", "
someremainder")));
+/// assert_eq!(match_name("some-name@someremainder"), Some(("some-name",
"@someremainder")));
+/// assert_eq!(match_name(""), None);
+/// assert_eq!(match_name(" someremainder"), None);
+/// ```
+pub fn match_name(line: &str) -> Option<(&str, &str)> {
+let end = line
+.as_bytes()
+.iter()
+.position(|&b| !(b.is_ascii_alphanumeric() || b == b'-'));
+
+let (name, rest) = match end {
+Some(end) => line.split_at(end),
+None => (line, ""),
+};
+
+if name.is_empty() {
+None
+} else {
+Some((name, rest))
+}
+}
+
+/// Parses up to the next whitespace character or end of the string.
+///
+/// Returns `None` if the non-whitespace part would be empty.
+///
+/// Returns a tuple containing the parsed section and the *trimmed* remainder.
+pub fn match_non_whitespace(line: &str) -> Option<(&str, &str)> {
+let (text, rest) = line
+.as_bytes()
+.iter()
+.position(|&b| b.is_ascii_whitespace())
+.map(|pos| {
+let (a, b) = line.split_at(pos);
+(a, b.trim_start())
+})
+.unwrap_or((line, ""));
+if text.is_empty() {
+None
+} else {
+Some((text, rest))
+}
+}
pub fn parse_bool(value: &str) -> Result {
Ok(
if value == "0"
diff --git a/proxmox-ve-config/src/firewall/types/alias.rs
b/proxmox-ve-config/src/firewall/types/alias.rs
new file mode 100644
index 000..43c6486
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/types/alias.rs
@@ -0,0 +1,160 @@
+use std::fmt::Display;
+use std::str::FromStr;
+
+use anyhow::{bail, format_err, Error};
+use serde_with::DeserializeFromStr;
+
+use crate::firewall::parse::{match_name, match_non_whitespace};
+use crate::firewall::types::address::Cidr;
+
+#[derive(Debug, Clone)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub enum AliasScope {
+Datacenter,
+Guest,
+}
+
+impl FromStr for AliasScope {
+type Err = Error;
+
+fn from_str(s: &str) -> Result {
+Ok(match s {
+"dc" => AliasScope::Datacenter,
+"guest" => AliasScope::Guest,
+_ => bail!("invalid scope for alias: {s}"),
+})
+}
+}
+
+impl Display for AliasScope {
+fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+f.write_str(match self {
+AliasScope::Datacenter => "dc",
+AliasScope::Guest => "guest",
+})
+}
+}
+
+#[derive(Debug, Clone, DeserializeFromStr)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub struct AliasName {
+scope: AliasScope,
+name: String,
+}
+
+impl Display for AliasName {
+fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+f.write_fmt(format_args!("{}/{}", self.scope, self.name))
+}
+}
+
+impl FromStr for AliasName {
+type Err = Error;
+
+fn from_str(s: &str) -> Result {
+match s.split_once('/') {
+Some((prefix, name)) if !name.is_empty() => Ok(Self {
+scope: prefix.parse()?,
+name: name.to_string(),
+}),
+_ => {
+bail!("Invalid Alias name!")
+}
+}
+}
+}
+
+impl AliasName {
+pub fn new(scope: AliasScope, name: impl Into) -> Self {
+Self {
+scope,
+name: name.into(),
+}
+}
+
+pub fn name(&self) -> &str {
+&self.name
+}
+
+pub fn scope(&self) -> &AliasScope {
+&self.scope
+}
+}
+
+#[derive(Debug)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub struct Alias {
+name: String,
+address: Cidr,
+comment: Option,
+}
+
+impl Alias {
+pub fn new(
+name: impl Into,
+address: impl Into,
+comment: impl Into>,
+) -> Self {
+Self {
+name: name.into(),
+address: address.into(),
+comment: comment.into(),
+}
+}
+
+pub fn name(&self) -> &str {
+&self.name
[pve-devel] [PATCH proxmox-firewall v2 06/39] config: host: add helpers for host network configuration
Currently the helpers for obtaining the host network configuration
panic on error, which could be avoided by the use of
OnceLock::get_or_init, but this method is currently only available in
nightly versions.
Generally, if there is a problem with obtaining the network config for
the node I would deem it acceptable for now, since that would usually
mean something is amiss with the network configuration and a firewall
won't really do anything then anyway.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/Cargo.toml| 5 ++
proxmox-ve-config/src/host/mod.rs | 1 +
proxmox-ve-config/src/host/utils.rs | 77 +
proxmox-ve-config/src/lib.rs| 1 +
4 files changed, 84 insertions(+)
create mode 100644 proxmox-ve-config/src/host/mod.rs
create mode 100644 proxmox-ve-config/src/host/utils.rs
diff --git a/proxmox-ve-config/Cargo.toml b/proxmox-ve-config/Cargo.toml
index 7bb391e..cc689c8 100644
--- a/proxmox-ve-config/Cargo.toml
+++ b/proxmox-ve-config/Cargo.toml
@@ -13,8 +13,13 @@ license = "AGPL-3"
[dependencies]
log = "0.4"
anyhow = "1"
+nix = "0.26"
serde = { version = "1", features = [ "derive" ] }
serde_json = "1"
serde_plain = "1"
serde_with = "2.3.3"
+
+proxmox-schema = "3.1.0"
+proxmox-sys = "0.5.3"
+proxmox-sortable-macro = "0.1.3"
diff --git a/proxmox-ve-config/src/host/mod.rs
b/proxmox-ve-config/src/host/mod.rs
new file mode 100644
index 000..b5614dd
--- /dev/null
+++ b/proxmox-ve-config/src/host/mod.rs
@@ -0,0 +1 @@
+pub mod utils;
diff --git a/proxmox-ve-config/src/host/utils.rs
b/proxmox-ve-config/src/host/utils.rs
new file mode 100644
index 000..08c40c9
--- /dev/null
+++ b/proxmox-ve-config/src/host/utils.rs
@@ -0,0 +1,77 @@
+use std::net::{IpAddr, ToSocketAddrs};
+use std::sync::OnceLock;
+
+use crate::firewall::types::Cidr;
+
+use nix::sys::socket::{AddressFamily, SockaddrLike};
+use proxmox_sys::nodename;
+
+pub fn host_ips() -> &'static [IpAddr] {
+static IP_ADDRESSES: OnceLock> = OnceLock::new();
+
+// We should rather use get_or_try_init to avoid needing to panic
+// but it is currently experimental
+IP_ADDRESSES.get_or_init(|| {
+let hostname = nodename();
+
+log::trace!("resolving hostname");
+
+format!("{hostname}:0")
+.to_socket_addrs()
+.expect("local hostname is resolvable")
+.map(|addr| addr.ip())
+.collect()
+})
+}
+
+pub fn network_interface_cidrs() -> &'static [Cidr] {
+static INTERFACES: OnceLock> = OnceLock::new();
+
+// We should rather use get_or_try_init to avoid needing to panic
+// but it is currently experimental
+INTERFACES.get_or_init(|| {
+use nix::ifaddrs::getifaddrs;
+
+log::trace!("reading networking interface list");
+
+let mut cidrs = Vec::new();
+
+let interfaces = getifaddrs().expect("should be able to query network
interfaces");
+
+for interface in interfaces {
+if let (Some(address), Some(netmask)) = (interface.address,
interface.netmask) {
+match (address.family(), netmask.family()) {
+(Some(AddressFamily::Inet), Some(AddressFamily::Inet)) => {
+let address = address.as_sockaddr_in().expect("is an
IPv4 address").ip();
+
+let netmask = netmask
+.as_sockaddr_in()
+.expect("is an IPv4 address")
+.ip()
+.count_ones()
+.try_into()
+.expect("count_ones of u32 is < u8_max");
+
+cidrs.push(Cidr::new_v4(address,
netmask).expect("netmask is valid"));
+}
+(Some(AddressFamily::Inet6), Some(AddressFamily::Inet6))
=> {
+let address = address.as_sockaddr_in6().expect("is an
IPv6 address").ip();
+
+let netmask_address =
+netmask.as_sockaddr_in6().expect("is an IPv6
address").ip();
+
+let netmask =
u128::from_be_bytes(netmask_address.octets())
+.count_ones()
+.try_into()
+.expect("count_ones of u128 is < u8_max");
+
+cidrs.push(Cidr::new_v6(address,
netmask).expect("netmask is valid"));
+}
+_ => continue,
+}
+}
+}
+
+cidrs
+})
+}
diff --git a/proxmox-ve-config/src/lib.rs b/proxmox-ve-config/src/lib.rs
index a0734b8..2bf9352 100644
--- a/proxmox-ve-config/src/lib.rs
+++ b/proxmox-ve-config/src/lib.rs
@@ -1 +1,2 @@
pub mod firewall;
+pub mod host;
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
htt
[pve-devel] [PATCH proxmox-firewall v2 18/39] nftables: add helpers
Several objects, statements and expressions in nftables-json require
null values, for instance:
{ "flush": { "ruleset": null }}
For this purpose we define our own Null type, which we can then easily
use for defining types that accept Null as value.
Several keys accept as value either a singular element (string or
object) if there is only one object, but an array if there are
multiple objects. For instance when adding a single element to a set:
{ "element": {
...
"elem": "element1"
}}
but when adding multiple elements:
{ "element": {
...
"elem": ["element1", "element2"]
}}
NfVec is a wrapper for Vec that serializes into T iff Vec
contains one element, otherwise it serializes like a Vec would
normally do.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-nftables/Cargo.toml| 4 +
proxmox-nftables/src/helper.rs | 190 +
proxmox-nftables/src/lib.rs| 1 +
3 files changed, 195 insertions(+)
create mode 100644 proxmox-nftables/src/helper.rs
diff --git a/proxmox-nftables/Cargo.toml b/proxmox-nftables/Cargo.toml
index 764e231..ebece9d 100644
--- a/proxmox-nftables/Cargo.toml
+++ b/proxmox-nftables/Cargo.toml
@@ -13,4 +13,8 @@ license = "AGPL-3"
[dependencies]
log = "0.4"
+serde = { version = "1", features = [ "derive" ] }
+serde_json = "1"
+serde_plain = "1"
+
proxmox-ve-config = { path = "../proxmox-ve-config", optional = true }
diff --git a/proxmox-nftables/src/helper.rs b/proxmox-nftables/src/helper.rs
new file mode 100644
index 000..77ce347
--- /dev/null
+++ b/proxmox-nftables/src/helper.rs
@@ -0,0 +1,190 @@
+use std::fmt;
+use std::marker::PhantomData;
+
+use serde::{Deserialize, Serialize};
+
+#[derive(Clone, Copy, Debug)]
+pub struct Null;
+
+impl<'de> Deserialize<'de> for Null {
+fn deserialize(deserializer: D) -> Result
+where
+D: serde::Deserializer<'de>,
+{
+use serde::de::Error;
+
+match Option::<()>::deserialize(deserializer)? {
+None => Ok(Self),
+Some(_) => Err(D::Error::custom("expected null")),
+}
+}
+}
+
+impl Serialize for Null {
+fn serialize(&self, serializer: S) -> Result
+where
+S: serde::Serializer,
+{
+serializer.serialize_none()
+}
+}
+
+impl fmt::Display for Null {
+fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+f.write_str("null")
+}
+}
+
+#[derive(Clone, Debug)]
+pub struct NfVec(pub(crate) Vec);
+
+impl Default for NfVec {
+fn default() -> Self {
+Self::new()
+}
+}
+
+impl NfVec {
+pub const fn new() -> Self {
+Self(Vec::new())
+}
+
+pub fn one(value: T) -> Self {
+Self(vec![value])
+}
+}
+
+impl From> for NfVec {
+fn from(v: Vec) -> Self {
+Self(v)
+}
+}
+
+impl From> for Vec {
+fn from(v: NfVec) -> Self {
+v.0
+}
+}
+
+impl FromIterator for NfVec {
+fn from_iter>(iter: I) -> Self {
+Self(iter.into_iter().collect())
+}
+}
+
+impl std::ops::Deref for NfVec {
+type Target = Vec;
+
+fn deref(&self) -> &Self::Target {
+&self.0
+}
+}
+
+impl std::ops::DerefMut for NfVec {
+fn deref_mut(&mut self) -> &mut Self::Target {
+&mut self.0
+}
+}
+
+impl Serialize for NfVec {
+fn serialize(&self, serializer: S) -> Result
+where
+S: serde::Serializer,
+{
+if self.len() == 1 {
+self[0].serialize(serializer)
+} else {
+self.0.serialize(serializer)
+}
+}
+}
+
+macro_rules! visit_value {
+($( ($visit:ident, $($ty:tt)+), )+) => {
+$(
+fn $visit(self, value: $($ty)+) -> Result
+where
+E: Error,
+{
+T::deserialize(value.into_deserializer()).map(NfVec::one)
+}
+)+
+};
+}
+
+impl<'de, T: Deserialize<'de>> Deserialize<'de> for NfVec {
+fn deserialize(deserializer: D) -> Result
+where
+D: serde::Deserializer<'de>,
+{
+use serde::de::{Error, IntoDeserializer};
+
+struct V(PhantomData);
+
+impl<'de, T: Deserialize<'de>> serde::de::Visitor<'de> for V {
+type Value = NfVec;
+
+fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
+f.write_str("an array or single element")
+}
+
+fn visit_seq(self, seq: A) -> Result
+where
+A: serde::de::SeqAccess<'de>,
+{
+
Vecdeserialize(serde::de::value::SeqAccessDeserializer::new(seq)).map(NfVec)
+}
+
+fn visit_map(self, map: A) -> Result
+where
+A: serde::de::MapAccess<'de>,
+{
+
T::deserialize(serde::de::value::MapAccessDeserializer::new(map)).map(N
[pve-devel] [PATCH container/docs/firewall/manager/proxmox-firewall/qemu-server v2 00/39] proxmox firewall nftables implementation
## Introduction
This RFC provides a drop-in replacement for the current pve-firewall package
that is based on Rust and nftables.
It consists of three crates:
* proxmox-ve-config
for parsing firewall and guest configuration files, as well as some helpers
to access host configuration (particularly networking)
* proxmox-nftables
contains bindings for libnftables as well as types that implement the JSON
schema defined by libnftables-json
* proxmox-firewall
uses the other two crates to read the firewall configuration and create the
respective nftables configuration
## Installation
* Build & install all deb packages on your PVE instance
* Enable the nftables firewall by going to
Web UI > > Firewall > Options > nftables
* Enable the firewall datacenter-wide if you haven't already
* Restarting running VMs/CTs is required so the changes to the fwbr creation
go into effect
For your convenience I have provided pre-built packages on our share under
`shanreich-proxmox-firewall`.
The source code is also available on my staff repo as `proxmox-firewall`.
## Configuration
The firewall should work as a drop-in replacement for the pve-firewall, so you
should be able to configure the firewall as usual via the Web UI or
configuration files.
## Known Issues
There is currently one major issue that we still need to solve: REJECTing
packets from the guest firewalls is currently not possible for incoming traffic
(it will instead be dropped).
This is due to the fact that we are using the postrouting hook of nftables in a
table with type bridge for incoming traffic. In the bridge table in the
postrouting hook we cannot tell whether the packet has also been sent to other
ports in the bridge (e.g. when a MAC has not yet been learned and the packet
then gets flooded to all bridge ports). If we would then REJECT a packet in the
postrouting hook this can lead to a bug where the firewall rules for one guest
REJECT a packet and send a response (RST for TCP, ICMP port/host-unreachable
otherwise).
This has also been explained in the respective commit introducing the
restriction [1].
We were able to circumvent this restriction in the old firewall due to using
firewall bridges and rejecting in the firewall bridge itself. Doing this leads
to the behavior described above, which has tripped up some of our users before
[2] [3] and which is, frankly, wrong.
I currently see two possible solutions for this, both of which carry downsides.
Your input on this matter would be much appreciated, particularly if you can
think of another solution which I cannot currently see:
1. Only REJECT packets in the prerouting chain of the firewall bridge with the
destination MAC address set to the MAC address of the network device, otherwise
DROP
The downside of this is that we, once again, will have to resort to using
firewall bridges, which we wanted to eliminate. This would also be the sole
reason for still having to resort to using firewall bridges.
2. Only allow DROP in the guest firewall for incoming traffic
This would be quite awkward since, well, rejecting traffic would be quite nice
for a firewall I'd say ;)
I'm happy for all input regarding this matter.
## Useful Commands
You can check if firewall rules got created by running
```
nft list ruleset
```
You can also check that `iptables` rules are not created via
```
iptables-save
```
Further info about the services:
```
systemctl status proxmox-firewall.{service,timer}
```
You can grab the debug output from the new firewall like so:
```
RUST_LOG=trace proxmox-firewall
```
## Upcoming
There are many ideas for further features, here is a (non-exhaustive) list:
* SNAT/DNAT handling
* SDN integration + bridge/vnet-level firewalling
* counters
* flow offloading
* synproxy support
* rate-limiting for rules
* connlimit support
* brouting support
The first thing I'll be working on though is proper rustdocs for both libraries
as well as the firewall.
Changes from v1 -> v2:
* now builds cleanly in sbuild (thanks @Fabian)
* made base ruleset more efficient
* fixed issues with some rules in the base ruleset
* refactored config loading in order to make it mockable (thanks @Lukas)
* added an integration test that spans the whole pipeline from config ->
nftables rules
* changed many maps from HashMap to BTreeMap to make the generated nftables
output stable (particularly important for the integration test)
* improved logging output
* implemented ARP spoofing prevention
* failing to change sysctl settings now only emits a warning instead of aborting
the rule generation process
* fixed some bugs wrt rule generation
* added a few macros (HTTP/3, PBS)
* added documentation to pve-docs
* EUI64 link-local addresses are now added to the automatically generated ip
filters
* incorporated suggestions from @Max and @Lukas (tyvm!)
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/net/bridge/netfilter/nft_reject_bridge.c?h=v6.8.2&id=127917c29a432c3b798e014a
[pve-devel] [PATCH proxmox-firewall v2 03/39] config: firewall: add types for ports
Adds types for all kinds of port-related values in the firewall config
as well as FromStr implementations for parsing them from the config.
Also adds a helper for parsing the named ports from `/etc/services`.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/src/firewall/mod.rs| 1 +
proxmox-ve-config/src/firewall/ports.rs | 80
proxmox-ve-config/src/firewall/types/mod.rs | 1 +
proxmox-ve-config/src/firewall/types/port.rs | 181 +++
4 files changed, 263 insertions(+)
create mode 100644 proxmox-ve-config/src/firewall/ports.rs
create mode 100644 proxmox-ve-config/src/firewall/types/port.rs
diff --git a/proxmox-ve-config/src/firewall/mod.rs
b/proxmox-ve-config/src/firewall/mod.rs
index cd40856..a9f65bf 100644
--- a/proxmox-ve-config/src/firewall/mod.rs
+++ b/proxmox-ve-config/src/firewall/mod.rs
@@ -1 +1,2 @@
+pub mod ports;
pub mod types;
diff --git a/proxmox-ve-config/src/firewall/ports.rs
b/proxmox-ve-config/src/firewall/ports.rs
new file mode 100644
index 000..9d5d1be
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/ports.rs
@@ -0,0 +1,80 @@
+use anyhow::{format_err, Error};
+use std::sync::OnceLock;
+
+#[derive(Default)]
+struct NamedPorts {
+ports: std::collections::HashMap,
+}
+
+impl NamedPorts {
+fn new() -> Self {
+use std::io::BufRead;
+
+log::trace!("loading /etc/services");
+
+let mut this = Self::default();
+
+let file = match std::fs::File::open("/etc/services") {
+Ok(file) => file,
+Err(_) => return this,
+};
+
+for line in std::io::BufReader::new(file).lines() {
+let line = match line {
+Ok(line) => line,
+Err(_) => break,
+};
+
+let line = line.trim_start();
+
+if line.is_empty() || line.starts_with('#') {
+continue;
+}
+
+let mut parts = line.split_ascii_whitespace();
+
+let name = match parts.next() {
+None => continue,
+Some(name) => name.to_string(),
+};
+
+let proto: u16 = match parts.next() {
+None => continue,
+Some(proto) => match proto.split('/').next() {
+None => continue,
+Some(num) => match num.parse() {
+Ok(num) => num,
+Err(_) => continue,
+},
+},
+};
+
+this.ports.insert(name, proto);
+for alias in parts {
+if alias.starts_with('#') {
+break;
+}
+this.ports.insert(alias.to_string(), proto);
+}
+}
+
+this
+}
+
+fn find(&self, name: &str) -> Option {
+self.ports.get(name).copied()
+}
+}
+
+fn named_ports() -> &'static NamedPorts {
+static NAMED_PORTS: OnceLock = OnceLock::new();
+
+NAMED_PORTS.get_or_init(NamedPorts::new)
+}
+
+/// Parse a named port with the help of `/etc/services`.
+pub fn parse_named_port(name: &str) -> Result {
+named_ports()
+.find(name)
+.ok_or_else(|| format_err!("unknown port name {name:?}"))
+}
diff --git a/proxmox-ve-config/src/firewall/types/mod.rs
b/proxmox-ve-config/src/firewall/types/mod.rs
index de534b4..b740e5d 100644
--- a/proxmox-ve-config/src/firewall/types/mod.rs
+++ b/proxmox-ve-config/src/firewall/types/mod.rs
@@ -1,3 +1,4 @@
pub mod address;
+pub mod port;
pub use address::Cidr;
diff --git a/proxmox-ve-config/src/firewall/types/port.rs
b/proxmox-ve-config/src/firewall/types/port.rs
new file mode 100644
index 000..c1252d9
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/types/port.rs
@@ -0,0 +1,181 @@
+use std::fmt;
+use std::ops::Deref;
+
+use anyhow::{bail, Error};
+use serde_with::DeserializeFromStr;
+
+use crate::firewall::ports::parse_named_port;
+
+#[derive(Clone, Debug)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub enum PortEntry {
+Port(u16),
+Range(u16, u16),
+}
+
+impl fmt::Display for PortEntry {
+fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+match self {
+Self::Port(p) => write!(f, "{p}"),
+Self::Range(beg, end) => write!(f, "{beg}-{end}"),
+}
+}
+}
+
+fn parse_port(port: &str) -> Result {
+if let Ok(port) = port.parse::() {
+return Ok(port);
+}
+
+if let Ok(port) = parse_named_port(port) {
+return Ok(port);
+}
+
+bail!("invalid port specification: {port}")
+}
+
+impl std::str::FromStr for PortEntry {
+type Err = Error;
+
+fn from_str(s: &str) -> Result {
+Ok(match s.trim().split_once(':') {
+None => PortEntry::from(parse_port(s)?),
+Some((first, second)) => {
+PortEntry::try_from((parse_port(first)?, parse_port(second
[pve-devel] [PATCH proxmox-firewall v2 10/39] config: firewall: add types for security groups
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/src/firewall/types/group.rs | 36 +++
proxmox-ve-config/src/firewall/types/mod.rs | 2 ++
2 files changed, 38 insertions(+)
create mode 100644 proxmox-ve-config/src/firewall/types/group.rs
diff --git a/proxmox-ve-config/src/firewall/types/group.rs
b/proxmox-ve-config/src/firewall/types/group.rs
new file mode 100644
index 000..7455268
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/types/group.rs
@@ -0,0 +1,36 @@
+use anyhow::Error;
+
+use crate::firewall::types::Rule;
+
+#[derive(Debug)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub struct Group {
+rules: Vec,
+comment: Option,
+}
+
+impl Group {
+pub const fn new() -> Self {
+Self {
+rules: Vec::new(),
+comment: None,
+}
+}
+
+pub fn rules(&self) -> &Vec {
+&self.rules
+}
+
+pub fn comment(&self) -> Option<&str> {
+self.comment.as_deref()
+}
+
+pub fn set_comment(&mut self, comment: Option) {
+self.comment = comment;
+}
+
+pub(crate) fn parse_entry(&mut self, line: &str) -> Result<(), Error> {
+self.rules.push(line.parse()?);
+Ok(())
+}
+}
diff --git a/proxmox-ve-config/src/firewall/types/mod.rs
b/proxmox-ve-config/src/firewall/types/mod.rs
index b4a6b12..8fd551e 100644
--- a/proxmox-ve-config/src/firewall/types/mod.rs
+++ b/proxmox-ve-config/src/firewall/types/mod.rs
@@ -1,5 +1,6 @@
pub mod address;
pub mod alias;
+pub mod group;
pub mod ipset;
pub mod log;
pub mod port;
@@ -8,5 +9,6 @@ pub mod rule_match;
pub use address::Cidr;
pub use alias::Alias;
+pub use group::Group;
pub use ipset::Ipset;
pub use rule::Rule;
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH proxmox-firewall v2 04/39] config: firewall: add types for log level and rate limit
Adds types for log and (log-)rate-limiting firewall config options as
well as FromStr implementations for parsing them from the config.
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
proxmox-ve-config/Cargo.toml| 1 +
proxmox-ve-config/src/firewall/mod.rs | 2 +
proxmox-ve-config/src/firewall/parse.rs | 21 ++
proxmox-ve-config/src/firewall/types/log.rs | 222
proxmox-ve-config/src/firewall/types/mod.rs | 1 +
5 files changed, 247 insertions(+)
create mode 100644 proxmox-ve-config/src/firewall/parse.rs
create mode 100644 proxmox-ve-config/src/firewall/types/log.rs
diff --git a/proxmox-ve-config/Cargo.toml b/proxmox-ve-config/Cargo.toml
index 80b336a..7bb391e 100644
--- a/proxmox-ve-config/Cargo.toml
+++ b/proxmox-ve-config/Cargo.toml
@@ -16,4 +16,5 @@ anyhow = "1"
serde = { version = "1", features = [ "derive" ] }
serde_json = "1"
+serde_plain = "1"
serde_with = "2.3.3"
diff --git a/proxmox-ve-config/src/firewall/mod.rs
b/proxmox-ve-config/src/firewall/mod.rs
index a9f65bf..2e0f31e 100644
--- a/proxmox-ve-config/src/firewall/mod.rs
+++ b/proxmox-ve-config/src/firewall/mod.rs
@@ -1,2 +1,4 @@
pub mod ports;
pub mod types;
+
+pub(crate) mod parse;
diff --git a/proxmox-ve-config/src/firewall/parse.rs
b/proxmox-ve-config/src/firewall/parse.rs
new file mode 100644
index 000..a75daee
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/parse.rs
@@ -0,0 +1,21 @@
+use anyhow::{bail, format_err, Error};
+
+pub fn parse_bool(value: &str) -> Result {
+Ok(
+if value == "0"
+|| value.eq_ignore_ascii_case("false")
+|| value.eq_ignore_ascii_case("off")
+|| value.eq_ignore_ascii_case("no")
+{
+false
+} else if value == "1"
+|| value.eq_ignore_ascii_case("true")
+|| value.eq_ignore_ascii_case("on")
+|| value.eq_ignore_ascii_case("yes")
+{
+true
+} else {
+bail!("not a boolean: {value:?}");
+},
+)
+}
diff --git a/proxmox-ve-config/src/firewall/types/log.rs
b/proxmox-ve-config/src/firewall/types/log.rs
new file mode 100644
index 000..72344e4
--- /dev/null
+++ b/proxmox-ve-config/src/firewall/types/log.rs
@@ -0,0 +1,222 @@
+use std::fmt;
+use std::str::FromStr;
+
+use crate::firewall::parse::parse_bool;
+use anyhow::{bail, Error};
+use serde::{Deserialize, Serialize};
+
+#[derive(Copy, Clone, Debug, Deserialize, Serialize, Default)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+#[serde(rename_all = "lowercase")]
+pub enum LogRateLimitTimescale {
+#[default]
+Second,
+Minute,
+Hour,
+Day,
+}
+
+impl FromStr for LogRateLimitTimescale {
+type Err = Error;
+
+fn from_str(str: &str) -> Result {
+match str {
+"second" => Ok(LogRateLimitTimescale::Second),
+"minute" => Ok(LogRateLimitTimescale::Minute),
+"hour" => Ok(LogRateLimitTimescale::Hour),
+"day" => Ok(LogRateLimitTimescale::Day),
+_ => bail!("Invalid time scale provided"),
+}
+}
+}
+
+#[derive(Debug, Deserialize, Clone)]
+#[cfg_attr(test, derive(Eq, PartialEq))]
+pub struct LogRateLimit {
+enabled: bool,
+rate: i64, // in packets
+per: LogRateLimitTimescale,
+burst: i64, // in packets
+}
+
+impl LogRateLimit {
+pub fn new(enabled: bool, rate: i64, per: LogRateLimitTimescale, burst:
i64) -> Self {
+Self {
+enabled,
+rate,
+per,
+burst,
+}
+}
+
+pub fn enabled(&self) -> bool {
+self.enabled
+}
+
+pub fn rate(&self) -> i64 {
+self.rate
+}
+
+pub fn burst(&self) -> i64 {
+self.burst
+}
+
+pub fn per(&self) -> LogRateLimitTimescale {
+self.per
+}
+}
+
+impl Default for LogRateLimit {
+fn default() -> Self {
+Self {
+enabled: true,
+rate: 1,
+burst: 5,
+per: LogRateLimitTimescale::Second,
+}
+}
+}
+
+impl FromStr for LogRateLimit {
+type Err = Error;
+
+fn from_str(str: &str) -> Result {
+let mut limit = Self::default();
+
+for element in str.split(',') {
+match element.split_once('=') {
+None => {
+limit.enabled = parse_bool(element)?;
+}
+Some((key, value)) if !key.is_empty() && !value.is_empty() =>
match key {
+"enable" => limit.enabled = parse_bool(value)?,
+"burst" => limit.burst = i64::from_str(value)?,
+"rate" => match value.split_once('/') {
+None => {
+limit.rate = i64::from_str(value)?;
+}
+Some((rate, unit)) => {
+if unit.is_empty() {
+
[pve-devel] [PATCH proxmox-firewall v2 01/39] config: add proxmox-ve-config crate
Reviewed-by: Lukas Wagner
Reviewed-by: Max Carrara
Co-authored-by: Wolfgang Bumiller
Signed-off-by: Stefan Hanreich
---
.cargo/config| 5 +
.gitignore | 6 ++
Cargo.toml | 4
proxmox-ve-config/Cargo.toml | 19 +++
proxmox-ve-config/src/lib.rs | 0
5 files changed, 34 insertions(+)
create mode 100644 .cargo/config
create mode 100644 .gitignore
create mode 100644 Cargo.toml
create mode 100644 proxmox-ve-config/Cargo.toml
create mode 100644 proxmox-ve-config/src/lib.rs
diff --git a/.cargo/config b/.cargo/config
new file mode 100644
index 000..3b5b6e4
--- /dev/null
+++ b/.cargo/config
@@ -0,0 +1,5 @@
+[source]
+[source.debian-packages]
+directory = "/usr/share/cargo/registry"
+[source.crates-io]
+replace-with = "debian-packages"
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..3cb8114
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,6 @@
+/target
+/Cargo.lock
+proxmox-firewall-*/
+*.deb
+*.buildinfo
+*.changes
diff --git a/Cargo.toml b/Cargo.toml
new file mode 100644
index 000..a8d33ab
--- /dev/null
+++ b/Cargo.toml
@@ -0,0 +1,4 @@
+[workspace]
+members = [
+"proxmox-ve-config",
+]
diff --git a/proxmox-ve-config/Cargo.toml b/proxmox-ve-config/Cargo.toml
new file mode 100644
index 000..80b336a
--- /dev/null
+++ b/proxmox-ve-config/Cargo.toml
@@ -0,0 +1,19 @@
+[package]
+name = "proxmox-ve-config"
+version = "0.1.0"
+edition = "2021"
+authors = [
+"Wolfgang Bumiller ",
+"Stefan Hanreich ",
+"Proxmox Support Team ",
+]
+description = "Proxmox VE config parsing"
+license = "AGPL-3"
+
+[dependencies]
+log = "0.4"
+anyhow = "1"
+
+serde = { version = "1", features = [ "derive" ] }
+serde_json = "1"
+serde_with = "2.3.3"
diff --git a/proxmox-ve-config/src/lib.rs b/proxmox-ve-config/src/lib.rs
new file mode 100644
index 000..e69de29
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH storage 3/9] plugin: dir: handle ova files for import
On April 17, 2024 3:10 pm, Dominik Csapak wrote:
> On 4/17/24 14:45, Fabian Grünbichler wrote:
>> On April 16, 2024 3:18 pm, Dominik Csapak wrote:
>>> +sub cleanup_extracted_image {
>>
>> same for this?
>>
>>> +my ($source) = @_;
>>> +
>>> +if ($source =~ m|^(/.+/\.tmp_[0-9]+_[0-9]+)/[^/]+$|) {
>>> + my $tmpdir = $1;
>>> +
>>> + unlink $source or $! == ENOENT or die "removing image $source failed:
>>> $!\n";
>>> + rmdir $tmpdir or $! == ENOENT or die "removing tmpdir $tmpdir failed:
>>> $!\n";
>>> +} else {
>>> + die "invalid extraced image path '$source'\n";
>>
>> nit: typo
>>
>> these are also not discoverable if the error handling in qemu-server
>> failed for some reason.. might be a source of unwanted space
>> consumption..
>
> any suggestions for better handling that cleanup?
> we could put it at the beginning of each cleanup step, that should
> at least make sure we cleaned up the temporary images
we could extract them into images/XXX/vm-XXX-disk-.. directly (or
rename/move them there after extraction), that way at least they could
be cleaned up via the storage API or rescan + delete (and via a regular
vdisk_free in qemu-server, instead of requiring a special helper).
other than that, I don't think we have an easy way of
- exposing them in list & free_image
- while ensuring nobody deletes them while the import is still going on
(the target VM ownership checks ensure that at least via the UI if we
make it an owned volume)
it would also allow skipping the conversion if the storage+format
already match the target spec as well..
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH pve-flutter-frontend] node overview: power menu: reorder/reword confirm buttons
Am 17/04/2024 um 10:53 schrieb Dominik Csapak: > move the confirm action to the right as mentioned in the material spec[0] > also rewords the buttons to 'cancel' and 'shutdown/reboot' > for that to work properly slightly rename the confirm message > > 0: > https://m3.material.io/components/dialogs/guidelines#befd7f4d-1029-4957-b1b5-da13fc0bbf3c > > Signed-off-by: Dominik Csapak > --- > replaces v3 of my previous patch since v2 was applied: > https://lists.proxmox.com/pipermail/pve-devel/2024-April/063106.html > > lib/widgets/pve_node_power_settings_widget.dart | 14 +++--- > 1 file changed, 7 insertions(+), 7 deletions(-) > > applied, with rewording the commit message a bit to make it easier to read (flow wise) and adding a Suggested-by trailer to give Folke credit for his review/suggestion – thanks! ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH storage 3/9] plugin: dir: handle ova files for import
On April 17, 2024 3:07 pm, Dominik Csapak wrote:
> On 4/17/24 12:52, Fiona Ebner wrote:
>> Am 16.04.24 um 15:18 schrieb Dominik Csapak:
>>> since we want to handle ova files (which are only ovf+vmdks bundled in a
>>> tar file) for import, add code that handles that.
>>>
>>> we introduce a valid volname for files contained in ovas like this:
>>>
>>> storage:import/archive.ova/disk-1.vmdk
>>>
>>> by basically treating the last part of the path as the name for the
>>> contained disk we want.
>>>
>>> we then provide 3 functions to use for that:
>>>
>>> * copy_needs_extraction: determines from the given volid (like above) if
>>>that needs extraction to copy it, currently only 'import' vtype +
>>>defined format returns true here (if we have more options in the
>>>future, we can of course easily extend that)
>>>
>>> * extract_disk_from_import_file: this actually extracts the file from
>>>the archive. Currently only ova is supported, so the extraction with
>>>'tar' is hardcoded, but again we can easily extend/modify that should
>>>we need to.
>>>
>>>we currently extract into the import storage in a directory named:
>>>`.tmp__` which should not clash with concurrent
>>>operations (though we do extract it multiple times then)
>>>
>>
>> Could we do "extract upon upload", "tar upon download" instead? Sure
>> some people surely want to drop the ova manually, but we could tell them
>> they need to extract it first too. Depending on the amount of headache
>> this would save us, it might be worth it.
>
> we could, but this opens a whole other can of worms, namely
> what to do with conflicting filenames for different ovas?
>
> we'd then either have to magically match the paths from the ovfs
> to some subdir that don't overlap
we could just use the ova name as dir name, and never store the ova
under that name but use some tmp placeholder for that ;)
>
> or we'd have to abort everytime we encounter identical disk names
>
> IMHO this would be less practical than just extract on demand...
>
>>
>>>alternatively we could implement either a 'tmpstorage' parameter,
>>>or use e.g. '/var/tmp/' or similar, but re-using the current storage
>>>seemed ok.
>>>
>>> * cleanup_extracted_image: intended to cleanup the extracted images from
>>>above, including the surrounding temporary directory
>>>
>>> we have to modify the `parse_ovf` a bit to handle the missing disk
>>> images, and we parse the size out of the ovf part (since this is
>>> informal only, it should be no problem if we cannot parse it sometimes)
>>>
>>> Signed-off-by: Dominik Csapak
>>> ---
>>> src/PVE/API2/Storage/Status.pm | 1 +
>>> src/PVE/Storage.pm | 59 ++
>>> src/PVE/Storage/DirPlugin.pm | 13 +++-
>>> src/PVE/Storage/OVF.pm | 53 ++
>>> src/PVE/Storage/Plugin.pm | 5 +++
>>> 5 files changed, 123 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
>>> index f7e324f..77ed57c 100644
>>> --- a/src/PVE/API2/Storage/Status.pm
>>> +++ b/src/PVE/API2/Storage/Status.pm
>>> @@ -749,6 +749,7 @@ __PACKAGE__->register_method({
>>> 'efi-state-lost',
>>> 'guest-is-running',
>>> 'nvme-unsupported',
>>> + 'ova-needs-extracting',
>>> 'ovmf-with-lsi-unsupported',
>>> 'serial-port-socket-only',
>>> ],
>>> diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
>>> index f8ea93d..bc073ef 100755
>>> --- a/src/PVE/Storage.pm
>>> +++ b/src/PVE/Storage.pm
>>> @@ -2189,4 +2189,63 @@ sub get_import_metadata {
>>> return $plugin->get_import_metadata($scfg, $volname, $storeid);
>>> }
>>>
>>
>> Shouldn't the following three functions call into plugin methods
>> instead? That'd seem much more future-proof to me.
>
> could be, i just did not want to extend the plugin api for that
> but as fabian wrote, maybe we should put them in qemu-server
> altogether for now?
>
> (after thinking about it a bit, i'd be in favor of putting it in
> qemu-server, because mainly i don't want to add to the plugin api further)
>
> what do you think @fiona @fabian?
another alternative would be to put them into the non-storage-plugin OVF
helper module?
>>> +sub copy_needs_extraction {
>>> +my ($volid) = @_;
>>> +my ($storeid, $volname) = parse_volume_id($volid);
>>> +my $cfg = config();
>>> +my $scfg = storage_config($cfg, $storeid);
>>> +my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
>>> +
>>> +my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $file_format)
>>> =
>>> + $plugin->parse_volname($volname);
>>> +
>>> +return $vtype eq 'import' && defined($file_format);
>>
>> E.g this seems rather hacky, and puts a weird coupling on a future
>> import plu
Re: [pve-devel] [PATCH storage/qemu-server/pve-manager] implement ova/ovf import for directory type storages
On 4/17/24 15:11, Fabian Grünbichler wrote: On April 16, 2024 3:18 pm, Dominik Csapak wrote: This series enables importing ova/ovf from directory based storages, inclusive upload/download via the webui (ova only). It also improves the ovf importer by parsing the ostype, nics, bootorder (and firmware from vmware exported files). I currently opted to move the OVF.pm to pve-storage, since there is no real other place where we could put it. Building a seperate package from qemu-servers git repo would also not be ideal, since we still have a cyclic dev dependency then (If someone has a better idea how to handle that, please do tell, and i can do that in a v2) There are surely some wrinkles left i did not think of, but all in all, it should be pretty usable. E.g. i downloaded some ovas, uploaded them on my cephfs in my virtual cluster, and successfully imported that with live-import. The biggest caveat when importing from ovas is that we have to temporarily extract the disk images. I opted for doing that into the import storage, but if we have a better idea where to put that, i can implement it in a v2 (or as a follow up). For example, we could add a new 'tmpdir' parameter to the create call and use that for extractig. something is wrong with the permissions, since the import images are not added to check_volume_access, I can now upload an OVA, but not see it afterwards ;) I guess if a user has upload rights for improt images (Datastore.AllocateTemplate), they should also be able to see and use (and remove) import images? ah yes, i forgot to add it there. but FWICS isos can have the same problem? upload only requires 'Datastore.AllocateTemplate' but seeing them requires 'Datastore.AllocateSpace' or 'Datastore.Audit' is that a mistake? ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH storage 7/9] ovf: implement parsing nics
On 4/17/24 14:09, Fiona Ebner wrote:
Am 16.04.24 um 15:19 schrieb Dominik Csapak:
by iterating over the relevant parts and trying to parse out the
'ResourceSubType'. The content of that is not standardized, but I only
ever found examples that are compatible with vmware, meaning it's
either 'e1000', 'e1000e' or 'vmxnet3' (in various capitalizations; thus
the `lc()`)
As a fallback i used vmxnet3, since i guess most OVAs are tuned for
vmware.
I'm not familiar enough with the OVA/OVF ecosystem, but is this really
the best default. I'd kinda expect e1000(e) to cause less issues in case
we were not able to get the type from the OVA/OVF. And people coming
from VMWare are likely going to use the dedicated importer.
i did choose that, since from what i saw looking for ovas, they are mostly
tailored for vmware consumption, so i thought it'd make sense to use
that as default.
not opposed to use e1000 though. i think in practice it won't make much
difference
Signed-off-by: Dominik Csapak
---
src/PVE/Storage/DirPlugin.pm | 2 +-
src/PVE/Storage/OVF.pm | 20 +++-
src/test/run_ovf_tests.pl| 5 +
3 files changed, 25 insertions(+), 2 deletions(-)
diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm
index 8a248c7..21c8350 100644
--- a/src/PVE/Storage/DirPlugin.pm
+++ b/src/PVE/Storage/DirPlugin.pm
@@ -294,7 +294,7 @@ sub get_import_metadata {
'create-args' => $res->{qm},
'disks' => $disks,
warnings => $warnings,
- net => [],
+ net => $res->{net},
};
}
diff --git a/src/PVE/Storage/OVF.pm b/src/PVE/Storage/OVF.pm
index f438de2..c3e7ed9 100644
--- a/src/PVE/Storage/OVF.pm
+++ b/src/PVE/Storage/OVF.pm
@@ -120,6 +120,12 @@ sub get_ostype {
return $ostype_ids->{$id} // 'other';
}
+my $allowed_nic_models = [
+'e1000',
+'e1000e',
+'vmxnet3',
+];
+
sub find_by {
my ($key, $param) = @_;
foreach my $resource (@resources) {
@@ -355,9 +361,21 @@ ovf:Item[rasd:InstanceID='%s']/rasd:ResourceType",
$controller_id);
$qm->{boot} = "order=" . join(';', @$boot);
+my $nic_id = dtmf_name_to_id('Ethernet Adapter');
+my $xpath_find_nics =
"/ovf:Envelope/ovf:VirtualSystem/ovf:VirtualHardwareSection/ovf:Item[rasd:ResourceType=${nic_id}]";
+my @nic_items = $xpc->findnodes($xpath_find_nics);
+
+my $net = {};
+
+my $net_count = 0;
+foreach my $item_node (@nic_items) {
Style nit: please use for instead of foreach
+ my $model = $xpc->findvalue('rasd:ResourceSubType', $item_node);
+ $model = lc($model);
+ $model = 'vmxnet3' if ! grep $model, @$allowed_nic_models;
+ $net->{"net${net_count}"} = { model => $model };
}
$net_count is never increased.
-return {qm => $qm, disks => \@disks};
+return {qm => $qm, disks => \@disks, net => $net};
}
1;
diff --git a/src/test/run_ovf_tests.pl b/src/test/run_ovf_tests.pl
index 8cf5662..d9a7b4b 100755
--- a/src/test/run_ovf_tests.pl
+++ b/src/test/run_ovf_tests.pl
@@ -54,6 +54,11 @@ is($win10noNs->{disks}->[0]->{disk_address}, 'scsi0',
'single disk vm (no defaul
is($win10noNs->{disks}->[0]->{backing_file},
"$test_manifests/Win10-Liz-disk1.vmdk", 'single disk vm (no default rasd NS) has the
correct disk backing device');
is($win10noNs->{disks}->[0]->{virtual_size}, 2048, 'single disk vm (no
default rasd NS) has the correct size');
+print "testing nics\n";
+is($win2008->{net}->{net0}->{model}, 'e1000', 'win2008 has correct nic model');
+is($win10->{net}->{net0}->{model}, 'e1000e', 'win10 has correct nic model');
+is($win10noNs->{net}->{net0}->{model}, 'e1000e', 'win10 (no default rasd NS)
has correct nic model');
+
print "\ntesting vm.conf extraction\n";
is($win2008->{qm}->{boot}, 'order=scsi0;scsi1', 'win2008 VM boot is correct');
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH storage 6/9] ovf: implement rudimentary boot order
On 4/17/24 13:54, Fiona Ebner wrote:
Am 16.04.24 um 15:18 schrieb Dominik Csapak:
simply add all parsed disks to the boot order in the order we encounter
them (similar to the esxi plugin).
Signed-off-by: Dominik Csapak
---
src/PVE/Storage/OVF.pm| 6 ++
src/test/run_ovf_tests.pl | 3 +++
2 files changed, 9 insertions(+)
diff --git a/src/PVE/Storage/OVF.pm b/src/PVE/Storage/OVF.pm
index f56c34d..f438de2 100644
--- a/src/PVE/Storage/OVF.pm
+++ b/src/PVE/Storage/OVF.pm
@@ -245,6 +245,8 @@ sub parse_ovf {
# when all the nodes has been found out, we copy the relevant information
to
# a $pve_disk hash ref, which we push to @disks;
+my $boot = [];
Nit: might be better to name it more verbosely since it's a long
function, e.g. boot_order, boot_disk_keys, or similar
+
foreach my $item_node (@disk_items) {
my $disk_node;
@@ -348,6 +350,10 @@ ovf:Item[rasd:InstanceID='%s']/rasd:ResourceType",
$controller_id);
};
$pve_disk->{virtual_size} = $virtual_size if defined($virtual_size);
push @disks, $pve_disk;
+ push @$boot, $pve_disk_address;
+}
This bracket should not be here and the next line below the next bracket
(fixed by the next patch).
+
+$qm->{boot} = "order=" . join(';', @$boot);
Won't this fail later if there are no disks?
yes, oops, will check if boot(_order) is empty
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH storage 4/9] ovf: implement parsing the ostype
On 4/17/24 13:32, Fiona Ebner wrote:
Am 16.04.24 um 15:18 schrieb Dominik Csapak:
use the standards info about the ostypes to map to our own
(see comment for link to the relevant part of the dmtf schema)
every type that is not listed we map to 'other', so no need to have it
in a list.
Signed-off-by: Dominik Csapak
Reviewed-by: Fiona Ebner
diff --git a/src/test/run_ovf_tests.pl b/src/test/run_ovf_tests.pl
index 1ef78cc..e949c15 100755
--- a/src/test/run_ovf_tests.pl
+++ b/src/test/run_ovf_tests.pl
@@ -59,13 +59,16 @@ print "\ntesting vm.conf extraction\n";
is($win2008->{qm}->{name}, 'Win2008-R2x64', 'win2008 VM name is correct');
is($win2008->{qm}->{memory}, '2048', 'win2008 VM memory is correct');
is($win2008->{qm}->{cores}, '1', 'win2008 VM cores are correct');
+is($win2008->{qm}->{ostype}, 'win7', 'win2008 VM ostype is correcty');
is($win10->{qm}->{name}, 'Win10-Liz', 'win10 VM name is correct');
is($win10->{qm}->{memory}, '6144', 'win10 VM memory is correct');
is($win10->{qm}->{cores}, '4', 'win10 VM cores are correct');
+is($win10->{qm}->{ostype}, 'other', 'win10 VM ostype is correct');
Yes, 'other', because the ovf config has id=1, but is there a special
reason why? Maybe worth a comment here and below to avoid potential
confusion.
my guess is that the ovf spec did not include windows 10 yet (or something
similar like the esxi exporter not knowing the newest spec)
and i did not want to change the testcase just for this
is($win10noNs->{qm}->{name}, 'Win10-Liz', 'win10 VM (no default rasd NS) name is correct');
is($win10noNs->{qm}->{memory}, '6144', 'win10 VM (no default rasd NS) memory
is correct');
is($win10noNs->{qm}->{cores}, '4', 'win10 VM (no default rasd NS) cores are
correct');
+is($win10noNs->{qm}->{ostype}, 'other', 'win10 VM (no default rasd NS) ostype
is correct');
done_testing();
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH storage 2/9] plugin: dir: implement import content type
On 4/17/24 12:07, Fiona Ebner wrote:
Am 16.04.24 um 15:18 schrieb Dominik Csapak:
in DirPlugin and not Plugin (because of cyclic dependency of
Plugin -> OVF -> Storage -> Plugin otherwise)
only ovf is currently supported (though ova will be shown in import
listing), expects the files to not be in a subdir, and adjacent to the
ovf file.
Signed-off-by: Dominik Csapak
---
src/PVE/Storage.pm | 8 ++-
src/PVE/Storage/DirPlugin.pm | 37 +-
src/PVE/Storage/OVF.pm | 2 ++
src/PVE/Storage/Plugin.pm | 18 ++-
src/test/parse_volname_test.pm | 13 +++
src/test/path_to_volume_id_test.pm | 16 +
6 files changed, 91 insertions(+), 3 deletions(-)
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 40314a8..f8ea93d 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -114,6 +114,8 @@ our $VZTMPL_EXT_RE_1 = qr/\.tar\.(gz|xz|zst)/i;
our $BACKUP_EXT_RE_2 = qr/\.(tgz|(?:tar|vma)(?:\.(${\PVE::Storage::Plugin::COMPRESSOR_RE}))?)/;
+our $IMPORT_EXT_RE_1 = qr/\.(ov[af])/;
+
# FIXME remove with PVE 8.0, add versioned breaks for pve-manager
our $vztmpl_extension_re = $VZTMPL_EXT_RE_1;
@@ -612,6 +614,7 @@ sub path_to_volume_id {
my $backupdir = $plugin->get_subdir($scfg, 'backup');
my $privatedir = $plugin->get_subdir($scfg, 'rootdir');
my $snippetsdir = $plugin->get_subdir($scfg, 'snippets');
+ my $importdir = $plugin->get_subdir($scfg, 'import');
if ($path =~ m!^$imagedir/(\d+)/([^/\s]+)$!) {
my $vmid = $1;
@@ -640,6 +643,9 @@ sub path_to_volume_id {
} elsif ($path =~ m!^$snippetsdir/([^/]+)$!) {
my $name = $1;
return ('snippets', "$sid:snippets/$name");
+ } elsif ($path =~ m!^$importdir/([^/]+${IMPORT_EXT_RE_1})$!) {
+ my $name = $1;
+ return ('import', "$sid:import/$name");
}
}
@@ -2170,7 +2176,7 @@ sub normalize_content_filename {
# If a storage provides an 'import' content type, it should be able to provide
# an object implementing the import information interface.
sub get_import_metadata {
-my ($cfg, $volid) = @_;
+my ($cfg, $volid, $target) = @_;
$target is added here but not passed along when calling the plugin's
function
leftover from a previous iteration of the patches
my ($storeid, $volname) = parse_volume_id($volid);
Pre-existing and not directly related, but in the ESXi plugin the
prototype seems wrong too:
sub get_import_metadata : prototype($) {
my ($class, $scfg, $volname, $storeid) = @_;
same here
diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm
index 2efa8d5..4dc7708 100644
--- a/src/PVE/Storage/DirPlugin.pm
+++ b/src/PVE/Storage/DirPlugin.pm
@@ -10,6 +10,7 @@ use IO::File;
use POSIX;
use PVE::Storage::Plugin;
+use PVE::Storage::OVF;
use PVE::JSONSchema qw(get_standard_option);
use base qw(PVE::Storage::Plugin);
@@ -22,7 +23,7 @@ sub type {
sub plugindata {
return {
- content => [ { images => 1, rootdir => 1, vztmpl => 1, iso => 1, backup => 1,
snippets => 1, none => 1 },
+ content => [ { images => 1, rootdir => 1, vztmpl => 1, iso => 1, backup => 1,
snippets => 1, none => 1, import => 1 },
{ images => 1, rootdir => 1 }],
format => [ { raw => 1, qcow2 => 1, vmdk => 1, subvol => 1 } , 'raw' ],
};
@@ -247,4 +248,38 @@ sub check_config {
return $opts;
}
+sub get_import_metadata {
+my ($class, $scfg, $volname, $storeid, $target) = @_;
+
+if ($volname !~ m!^([^/]+)/.*${PVE::Storage::IMPORT_EXT_RE_1}$!) {
+ die "volume '$volname' does not look like an importable vm config\n";
+}
+
+my $path = $class->path($scfg, $volname, $storeid, undef);
+
+# NOTE: all types must be added to the return schema of the
import-metadata API endpoint
To be extra clear (was confused for a moment): "all types of warnings"
+my $warnings = [];
+
+my $res = PVE::Storage::OVF::parse_ovf($path, $isOva);
$isOva does not exist yet (only added by a later patch).
+my $disks = {};
+for my $disk ($res->{disks}->@*) {
+ my $id = $disk->{disk_address};
+ my $size = $disk->{virtual_size};
+ my $path = $disk->{relative_path};
+ $disks->{$id} = {
+ volid => "$storeid:import/$path",
+ defined($size) ? (size => $size) : (),
+ };
+}
+
+return {
+ type => 'vm',
+ source => $volname,
+ 'create-args' => $res->{qm},
+ 'disks' => $disks,
+ warnings => $warnings,
+ net => [],
+};
+}
+
1;
diff --git a/src/PVE/Storage/OVF.pm b/src/PVE/Storage/OVF.pm
index 90ca453..4a322b9 100644
--- a/src/PVE/Storage/OVF.pm
+++ b/src/PVE/Storage/OVF.pm
@@ -222,6 +222,7 @@ ovf:Item[rasd:InstanceID='%s']/rasd:ResourceType",
$controller_id);
}
($backing_file_path) = $backing_file_path
Re: [pve-devel] [PATCH storage/qemu-server/pve-manager] implement ova/ovf import for directory type storages
On April 16, 2024 3:18 pm, Dominik Csapak wrote: > This series enables importing ova/ovf from directory based storages, > inclusive upload/download via the webui (ova only). > > It also improves the ovf importer by parsing the ostype, nics, bootorder > (and firmware from vmware exported files). > > I currently opted to move the OVF.pm to pve-storage, since there is no > real other place where we could put it. Building a seperate package > from qemu-servers git repo would also not be ideal, since we still > have a cyclic dev dependency then > (If someone has a better idea how to handle that, please do tell, and > i can do that in a v2) > > There are surely some wrinkles left i did not think of, but all in all, > it should be pretty usable. E.g. i downloaded some ovas, uploaded them > on my cephfs in my virtual cluster, and successfully imported that with > live-import. > > The biggest caveat when importing from ovas is that we have to > temporarily extract the disk images. I opted for doing that into the > import storage, but if we have a better idea where to put that, i can > implement it in a v2 (or as a follow up). For example, we could add a > new 'tmpdir' parameter to the create call and use that for extractig. something is wrong with the permissions, since the import images are not added to check_volume_access, I can now upload an OVA, but not see it afterwards ;) I guess if a user has upload rights for improt images (Datastore.AllocateTemplate), they should also be able to see and use (and remove) import images? ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH storage 3/9] plugin: dir: handle ova files for import
On 4/17/24 14:45, Fabian Grünbichler wrote:
On April 16, 2024 3:18 pm, Dominik Csapak wrote:
since we want to handle ova files (which are only ovf+vmdks bundled in a
tar file) for import, add code that handles that.
we introduce a valid volname for files contained in ovas like this:
storage:import/archive.ova/disk-1.vmdk
by basically treating the last part of the path as the name for the
contained disk we want.
we then provide 3 functions to use for that:
* copy_needs_extraction: determines from the given volid (like above) if
that needs extraction to copy it, currently only 'import' vtype +
defined format returns true here (if we have more options in the
future, we can of course easily extend that)
* extract_disk_from_import_file: this actually extracts the file from
the archive. Currently only ova is supported, so the extraction with
'tar' is hardcoded, but again we can easily extend/modify that should
we need to.
we currently extract into the import storage in a directory named:
`.tmp__` which should not clash with concurrent
operations (though we do extract it multiple times then)
alternatively we could implement either a 'tmpstorage' parameter,
or use e.g. '/var/tmp/' or similar, but re-using the current storage
seemed ok.
* cleanup_extracted_image: intended to cleanup the extracted images from
above, including the surrounding temporary directory
the helpers could also all live in qemu-server for now, which would also
make extending it to use a different storage, or direct importing via a
pipe easier? see below ;)
we have to modify the `parse_ovf` a bit to handle the missing disk
images, and we parse the size out of the ovf part (since this is
informal only, it should be no problem if we cannot parse it sometimes)
Signed-off-by: Dominik Csapak
---
src/PVE/API2/Storage/Status.pm | 1 +
src/PVE/Storage.pm | 59 ++
src/PVE/Storage/DirPlugin.pm | 13 +++-
src/PVE/Storage/OVF.pm | 53 ++
src/PVE/Storage/Plugin.pm | 5 +++
5 files changed, 123 insertions(+), 8 deletions(-)
diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index f7e324f..77ed57c 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -749,6 +749,7 @@ __PACKAGE__->register_method({
'efi-state-lost',
'guest-is-running',
'nvme-unsupported',
+ 'ova-needs-extracting',
'ovmf-with-lsi-unsupported',
'serial-port-socket-only',
],
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index f8ea93d..bc073ef 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -2189,4 +2189,63 @@ sub get_import_metadata {
return $plugin->get_import_metadata($scfg, $volname, $storeid);
}
+sub copy_needs_extraction {
+my ($volid) = @_;
+my ($storeid, $volname) = parse_volume_id($volid);
+my $cfg = config();
+my $scfg = storage_config($cfg, $storeid);
+my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+
+my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $file_format) =
+ $plugin->parse_volname($volname);
+
+return $vtype eq 'import' && defined($file_format);
+}
not sure this one is needed? it could also just be a call to
PVE::Storage::parse_volname in qemu-server?
+
+sub extract_disk_from_import_file {
similarly, this is basically PVE::Storage::get_import_dir + the
run_command call, and could live in qemu-server?
+my ($volid, $vmid) = @_;
+
+my ($storeid, $volname) = parse_volume_id($volid);
+my $cfg = config();
+my $scfg = storage_config($cfg, $storeid);
+my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+
+my ($vtype, $name, undef, undef, undef, undef, $file_format) =
+ $plugin->parse_volname($volname);
+
+die "only files with content type 'import' can be extracted\n"
+ if $vtype ne 'import' || !defined($file_format);
+
+# extract the inner file from the name
+if ($volid =~ m!${name}/([^/]+)$!) {
+ $name = $1;
we should probably be very conservative here and only allow [-_a-z0-9]
as a start - or something similar rather restrictive..
+}
+
+my ($source_file) = $plugin->path($scfg, $volname, $storeid);
+
+my $destdir = $plugin->get_subdir($scfg, 'import');
+my $pid = $$;
+$destdir .= "/.tmp_${pid}_${vmid}";
+mkdir $destdir;
+
+($source_file) = $source_file =~ m|^(/.*)|; # untaint
again a rather interesting untaint ;)
+
+run_command(['tar', '-x', '-C', $destdir, '-f', $source_file, $name]);
if $name was a symlink in the archive, you've now created a symlink
pointing wherever..
+
+return "$destdir/$name";
and this returns an absolute path to it, and now we are in tr
Re: [pve-devel] [PATCH storage 3/9] plugin: dir: handle ova files for import
On 4/17/24 12:52, Fiona Ebner wrote:
Am 16.04.24 um 15:18 schrieb Dominik Csapak:
since we want to handle ova files (which are only ovf+vmdks bundled in a
tar file) for import, add code that handles that.
we introduce a valid volname for files contained in ovas like this:
storage:import/archive.ova/disk-1.vmdk
by basically treating the last part of the path as the name for the
contained disk we want.
we then provide 3 functions to use for that:
* copy_needs_extraction: determines from the given volid (like above) if
that needs extraction to copy it, currently only 'import' vtype +
defined format returns true here (if we have more options in the
future, we can of course easily extend that)
* extract_disk_from_import_file: this actually extracts the file from
the archive. Currently only ova is supported, so the extraction with
'tar' is hardcoded, but again we can easily extend/modify that should
we need to.
we currently extract into the import storage in a directory named:
`.tmp__` which should not clash with concurrent
operations (though we do extract it multiple times then)
Could we do "extract upon upload", "tar upon download" instead? Sure
some people surely want to drop the ova manually, but we could tell them
they need to extract it first too. Depending on the amount of headache
this would save us, it might be worth it.
we could, but this opens a whole other can of worms, namely
what to do with conflicting filenames for different ovas?
we'd then either have to magically match the paths from the ovfs
to some subdir that don't overlap
or we'd have to abort everytime we encounter identical disk names
IMHO this would be less practical than just extract on demand...
alternatively we could implement either a 'tmpstorage' parameter,
or use e.g. '/var/tmp/' or similar, but re-using the current storage
seemed ok.
* cleanup_extracted_image: intended to cleanup the extracted images from
above, including the surrounding temporary directory
we have to modify the `parse_ovf` a bit to handle the missing disk
images, and we parse the size out of the ovf part (since this is
informal only, it should be no problem if we cannot parse it sometimes)
Signed-off-by: Dominik Csapak
---
src/PVE/API2/Storage/Status.pm | 1 +
src/PVE/Storage.pm | 59 ++
src/PVE/Storage/DirPlugin.pm | 13 +++-
src/PVE/Storage/OVF.pm | 53 ++
src/PVE/Storage/Plugin.pm | 5 +++
5 files changed, 123 insertions(+), 8 deletions(-)
diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
index f7e324f..77ed57c 100644
--- a/src/PVE/API2/Storage/Status.pm
+++ b/src/PVE/API2/Storage/Status.pm
@@ -749,6 +749,7 @@ __PACKAGE__->register_method({
'efi-state-lost',
'guest-is-running',
'nvme-unsupported',
+ 'ova-needs-extracting',
'ovmf-with-lsi-unsupported',
'serial-port-socket-only',
],
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index f8ea93d..bc073ef 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -2189,4 +2189,63 @@ sub get_import_metadata {
return $plugin->get_import_metadata($scfg, $volname, $storeid);
}
Shouldn't the following three functions call into plugin methods
instead? That'd seem much more future-proof to me.
could be, i just did not want to extend the plugin api for that
but as fabian wrote, maybe we should put them in qemu-server
altogether for now?
(after thinking about it a bit, i'd be in favor of putting it in
qemu-server, because mainly i don't want to add to the plugin api further)
what do you think @fiona @fabian?
+sub copy_needs_extraction {
+my ($volid) = @_;
+my ($storeid, $volname) = parse_volume_id($volid);
+my $cfg = config();
+my $scfg = storage_config($cfg, $storeid);
+my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+
+my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $file_format) =
+ $plugin->parse_volname($volname);
+
+return $vtype eq 'import' && defined($file_format);
E.g this seems rather hacky, and puts a weird coupling on a future
import plugin's parse_volname() function (presence of $file_format).
would it be better to check the volid again for '.ova/something$' ?
or do you have a better idea?
(especially if we want to have this maybe in qemu-server)
+}
+
+sub extract_disk_from_import_file {
+my ($volid, $vmid) = @_;
+
+my ($storeid, $volname) = parse_volume_id($volid);
+my $cfg = config();
+my $scfg = storage_config($cfg, $storeid);
+my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+
+my ($vtype, $name, undef, undef, undef, undef, $file_format) =
+ $plugin->parse_volname($volname);
+
+
Re: [pve-devel] [PATCH storage 3/9] plugin: dir: handle ova files for import
On April 16, 2024 3:18 pm, Dominik Csapak wrote:
> since we want to handle ova files (which are only ovf+vmdks bundled in a
> tar file) for import, add code that handles that.
>
> we introduce a valid volname for files contained in ovas like this:
>
> storage:import/archive.ova/disk-1.vmdk
>
> by basically treating the last part of the path as the name for the
> contained disk we want.
>
> we then provide 3 functions to use for that:
>
> * copy_needs_extraction: determines from the given volid (like above) if
> that needs extraction to copy it, currently only 'import' vtype +
> defined format returns true here (if we have more options in the
> future, we can of course easily extend that)
>
> * extract_disk_from_import_file: this actually extracts the file from
> the archive. Currently only ova is supported, so the extraction with
> 'tar' is hardcoded, but again we can easily extend/modify that should
> we need to.
>
> we currently extract into the import storage in a directory named:
> `.tmp__` which should not clash with concurrent
> operations (though we do extract it multiple times then)
>
> alternatively we could implement either a 'tmpstorage' parameter,
> or use e.g. '/var/tmp/' or similar, but re-using the current storage
> seemed ok.
>
> * cleanup_extracted_image: intended to cleanup the extracted images from
> above, including the surrounding temporary directory
the helpers could also all live in qemu-server for now, which would also
make extending it to use a different storage, or direct importing via a
pipe easier? see below ;)
>
> we have to modify the `parse_ovf` a bit to handle the missing disk
> images, and we parse the size out of the ovf part (since this is
> informal only, it should be no problem if we cannot parse it sometimes)
>
> Signed-off-by: Dominik Csapak
> ---
> src/PVE/API2/Storage/Status.pm | 1 +
> src/PVE/Storage.pm | 59 ++
> src/PVE/Storage/DirPlugin.pm | 13 +++-
> src/PVE/Storage/OVF.pm | 53 ++
> src/PVE/Storage/Plugin.pm | 5 +++
> 5 files changed, 123 insertions(+), 8 deletions(-)
>
> diff --git a/src/PVE/API2/Storage/Status.pm b/src/PVE/API2/Storage/Status.pm
> index f7e324f..77ed57c 100644
> --- a/src/PVE/API2/Storage/Status.pm
> +++ b/src/PVE/API2/Storage/Status.pm
> @@ -749,6 +749,7 @@ __PACKAGE__->register_method({
> 'efi-state-lost',
> 'guest-is-running',
> 'nvme-unsupported',
> + 'ova-needs-extracting',
> 'ovmf-with-lsi-unsupported',
> 'serial-port-socket-only',
> ],
> diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
> index f8ea93d..bc073ef 100755
> --- a/src/PVE/Storage.pm
> +++ b/src/PVE/Storage.pm
> @@ -2189,4 +2189,63 @@ sub get_import_metadata {
> return $plugin->get_import_metadata($scfg, $volname, $storeid);
> }
>
> +sub copy_needs_extraction {
> +my ($volid) = @_;
> +my ($storeid, $volname) = parse_volume_id($volid);
> +my $cfg = config();
> +my $scfg = storage_config($cfg, $storeid);
> +my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
> +
> +my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $file_format) =
> + $plugin->parse_volname($volname);
> +
> +return $vtype eq 'import' && defined($file_format);
> +}
not sure this one is needed? it could also just be a call to
PVE::Storage::parse_volname in qemu-server?
> +
> +sub extract_disk_from_import_file {
similarly, this is basically PVE::Storage::get_import_dir + the
run_command call, and could live in qemu-server?
> +my ($volid, $vmid) = @_;
> +
> +my ($storeid, $volname) = parse_volume_id($volid);
> +my $cfg = config();
> +my $scfg = storage_config($cfg, $storeid);
> +my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
> +
> +my ($vtype, $name, undef, undef, undef, undef, $file_format) =
> + $plugin->parse_volname($volname);
> +
> +die "only files with content type 'import' can be extracted\n"
> + if $vtype ne 'import' || !defined($file_format);
> +
> +# extract the inner file from the name
> +if ($volid =~ m!${name}/([^/]+)$!) {
> + $name = $1;
we should probably be very conservative here and only allow [-_a-z0-9]
as a start - or something similar rather restrictive..
> +}
> +
> +my ($source_file) = $plugin->path($scfg, $volname, $storeid);
> +
> +my $destdir = $plugin->get_subdir($scfg, 'import');
> +my $pid = $$;
> +$destdir .= "/.tmp_${pid}_${vmid}";
> +mkdir $destdir;
> +
> +($source_file) = $source_file =~ m|^(/.*)|; # untaint
again a rather interesting untaint ;)
> +
> +run_command(['tar', '-x', '-C', $destdir, '-f', $source_file, $name]);
if $name was a symlink in the archive, you've now created a symlink
point
Re: [pve-devel] [PATCH storage 2/9] plugin: dir: implement import content type
On April 16, 2024 3:18 pm, Dominik Csapak wrote:
> in DirPlugin and not Plugin (because of cyclic dependency of
> Plugin -> OVF -> Storage -> Plugin otherwise)
>
> only ovf is currently supported (though ova will be shown in import
> listing), expects the files to not be in a subdir, and adjacent to the
> ovf file.
>
> Signed-off-by: Dominik Csapak
> ---
> src/PVE/Storage.pm | 8 ++-
> src/PVE/Storage/DirPlugin.pm | 37 +-
> src/PVE/Storage/OVF.pm | 2 ++
> src/PVE/Storage/Plugin.pm | 18 ++-
> src/test/parse_volname_test.pm | 13 +++
> src/test/path_to_volume_id_test.pm | 16 +
> 6 files changed, 91 insertions(+), 3 deletions(-)
>
> diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
> index 40314a8..f8ea93d 100755
> --- a/src/PVE/Storage.pm
> +++ b/src/PVE/Storage.pm
> @@ -114,6 +114,8 @@ our $VZTMPL_EXT_RE_1 = qr/\.tar\.(gz|xz|zst)/i;
>
> our $BACKUP_EXT_RE_2 =
> qr/\.(tgz|(?:tar|vma)(?:\.(${\PVE::Storage::Plugin::COMPRESSOR_RE}))?)/;
>
> +our $IMPORT_EXT_RE_1 = qr/\.(ov[af])/;
> +
> # FIXME remove with PVE 8.0, add versioned breaks for pve-manager
> our $vztmpl_extension_re = $VZTMPL_EXT_RE_1;
>
> @@ -612,6 +614,7 @@ sub path_to_volume_id {
> my $backupdir = $plugin->get_subdir($scfg, 'backup');
> my $privatedir = $plugin->get_subdir($scfg, 'rootdir');
> my $snippetsdir = $plugin->get_subdir($scfg, 'snippets');
> + my $importdir = $plugin->get_subdir($scfg, 'import');
>
> if ($path =~ m!^$imagedir/(\d+)/([^/\s]+)$!) {
> my $vmid = $1;
> @@ -640,6 +643,9 @@ sub path_to_volume_id {
> } elsif ($path =~ m!^$snippetsdir/([^/]+)$!) {
> my $name = $1;
> return ('snippets', "$sid:snippets/$name");
> + } elsif ($path =~ m!^$importdir/([^/]+${IMPORT_EXT_RE_1})$!) {
> + my $name = $1;
> + return ('import', "$sid:import/$name");
> }
> }
>
> @@ -2170,7 +2176,7 @@ sub normalize_content_filename {
> # If a storage provides an 'import' content type, it should be able to
> provide
> # an object implementing the import information interface.
> sub get_import_metadata {
> -my ($cfg, $volid) = @_;
> +my ($cfg, $volid, $target) = @_;
>
> my ($storeid, $volname) = parse_volume_id($volid);
>
> diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm
> index 2efa8d5..4dc7708 100644
> --- a/src/PVE/Storage/DirPlugin.pm
> +++ b/src/PVE/Storage/DirPlugin.pm
> @@ -10,6 +10,7 @@ use IO::File;
> use POSIX;
>
> use PVE::Storage::Plugin;
> +use PVE::Storage::OVF;
> use PVE::JSONSchema qw(get_standard_option);
>
> use base qw(PVE::Storage::Plugin);
> @@ -22,7 +23,7 @@ sub type {
>
> sub plugindata {
> return {
> - content => [ { images => 1, rootdir => 1, vztmpl => 1, iso => 1, backup
> => 1, snippets => 1, none => 1 },
> + content => [ { images => 1, rootdir => 1, vztmpl => 1, iso => 1, backup
> => 1, snippets => 1, none => 1, import => 1 },
>{ images => 1, rootdir => 1 }],
> format => [ { raw => 1, qcow2 => 1, vmdk => 1, subvol => 1 } , 'raw' ],
> };
> @@ -247,4 +248,38 @@ sub check_config {
> return $opts;
> }
>
> +sub get_import_metadata {
> +my ($class, $scfg, $volname, $storeid, $target) = @_;
> +
> +if ($volname !~ m!^([^/]+)/.*${PVE::Storage::IMPORT_EXT_RE_1}$!) {
> + die "volume '$volname' does not look like an importable vm config\n";
> +}
shouldn't this happen in parse_volname? or rather, why is this different
than the code there?
> +
> +my $path = $class->path($scfg, $volname, $storeid, undef);
> +
> +# NOTE: all types must be added to the return schema of the
> import-metadata API endpoint
> +my $warnings = [];
> +
> +my $res = PVE::Storage::OVF::parse_ovf($path, $isOva);
nit: $isOva doesn't yet exist in this patch, neither as variable here,
nor as parameter in parse_ovf ;)
> +my $disks = {};
> +for my $disk ($res->{disks}->@*) {
> + my $id = $disk->{disk_address};
> + my $size = $disk->{virtual_size};
> + my $path = $disk->{relative_path};
see below
> + $disks->{$id} = {
> + volid => "$storeid:import/$path",
> + defined($size) ? (size => $size) : (),
> + };
> +}
> +
> +return {
> + type => 'vm',
> + source => $volname,
> + 'create-args' => $res->{qm},
> + 'disks' => $disks,
> + warnings => $warnings,
> + net => [],
> +};
> +}
> +
> 1;
> diff --git a/src/PVE/Storage/OVF.pm b/src/PVE/Storage/OVF.pm
> index 90ca453..4a322b9 100644
> --- a/src/PVE/Storage/OVF.pm
> +++ b/src/PVE/Storage/OVF.pm
> @@ -222,6 +222,7 @@ ovf:Item[rasd:InstanceID='%s']/rasd:ResourceType",
> $controller_id);
> }
>
> ($backing_file_path) = $backing_file_path =~ m|^(/.*)|; # untaint
> + ($filepath) = $filepath =~ m|^(.*)|; # untaint
nit: that's a weird untaint ;) maybe add the `$` at least
[pve-devel] [PATCH installer v6 30/36] add proxmox-chroot utility
it is meant as a helper utility to prepare an installation for chroot
and clean up afterwards
It tries to determine the used FS from the previous installation, will
do what is necessary to mount/import the root FS to /target. It then
will set up all bind mounts.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
Cargo.toml | 1 +
Makefile | 5 +-
proxmox-chroot/Cargo.toml | 16 ++
proxmox-chroot/src/main.rs | 356 +
4 files changed, 377 insertions(+), 1 deletion(-)
create mode 100644 proxmox-chroot/Cargo.toml
create mode 100644 proxmox-chroot/src/main.rs
diff --git a/Cargo.toml b/Cargo.toml
index b694d5b..b3afc7c 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -2,6 +2,7 @@
members = [
"proxmox-auto-installer",
"proxmox-autoinst-helper",
+"proxmox-chroot",
"proxmox-fetch-answer",
"proxmox-installer-common",
"proxmox-tui-installer",
diff --git a/Makefile b/Makefile
index e32d28f..d69dc6f 100644
--- a/Makefile
+++ b/Makefile
@@ -19,6 +19,7 @@ INSTALLER_SOURCES=$(shell git ls-files) country.dat
PREFIX = /usr
BINDIR = $(PREFIX)/bin
USR_BIN := \
+ proxmox-chroot\
proxmox-tui-installer\
proxmox-autoinst-helper\
proxmox-fetch-answer\
@@ -54,6 +55,7 @@ $(BUILDDIR):
proxmox-auto-installer/ \
proxmox-autoinst-helper/ \
proxmox-fetch-answer/ \
+ proxmox-chroot \
proxmox-tui-installer/ \
proxmox-installer-common/ \
test/ \
@@ -127,7 +129,8 @@ cargo-build:
$(CARGO) build --package proxmox-tui-installer --bin
proxmox-tui-installer \
--package proxmox-auto-installer --bin proxmox-auto-installer \
--package proxmox-fetch-answer --bin proxmox-fetch-answer \
- --package proxmox-autoinst-helper --bin proxmox-autoinst-helper
$(CARGO_BUILD_ARGS)
+ --package proxmox-autoinst-helper --bin proxmox-autoinst-helper
\
+ --package proxmox-chroot --bin proxmox-chroot
$(CARGO_BUILD_ARGS)
%-banner.png: %-banner.svg
rsvg-convert -o $@ $<
diff --git a/proxmox-chroot/Cargo.toml b/proxmox-chroot/Cargo.toml
new file mode 100644
index 000..43b96ff
--- /dev/null
+++ b/proxmox-chroot/Cargo.toml
@@ -0,0 +1,16 @@
+[package]
+name = "proxmox-chroot"
+version = "0.1.0"
+edition = "2021"
+authors = [ "Aaron Lauterer " ]
+license = "AGPL-3"
+exclude = [ "build", "debian" ]
+homepage = "https://www.proxmox.com";
+
+[dependencies]
+anyhow = "1.0"
+clap = { version = "4.0", features = ["derive"] }
+nix = "0.26.1"
+proxmox-installer-common = { path = "../proxmox-installer-common" }
+regex = "1.7"
+serde_json = "1.0"
diff --git a/proxmox-chroot/src/main.rs b/proxmox-chroot/src/main.rs
new file mode 100644
index 000..c1a4785
--- /dev/null
+++ b/proxmox-chroot/src/main.rs
@@ -0,0 +1,356 @@
+use std::{fs, io, path, process::Command};
+
+use anyhow::{bail, Result};
+use clap::{Args, Parser, Subcommand, ValueEnum};
+use nix::mount::{mount, umount, MsFlags};
+use proxmox_installer_common::{
+options::FsType,
+setup::{InstallConfig, SetupInfo},
+};
+use regex::Regex;
+
+const ANSWER_MP: &str = "answer";
+static BINDMOUNTS: [&str; 4] = ["dev", "proc", "run", "sys"];
+const TARGET_DIR: &str = "/target";
+const ZPOOL_NAME: &str = "rpool";
+
+/// Helper tool to prepare eveything to `chroot` into an installation
+#[derive(Parser, Debug)]
+#[command(author, version, about, long_about = None)]
+struct Cli {
+#[command(subcommand)]
+command: Commands,
+}
+
+#[derive(Subcommand, Debug)]
+enum Commands {
+Prepare(CommandPrepare),
+Cleanup(CommandCleanup),
+}
+
+/// Mount the root file system and bind mounts in preparation to chroot into
the installation
+#[derive(Args, Debug)]
+struct CommandPrepare {
+/// Filesystem used for the installation. Will try to automatically detect
it after a
+/// successful installation.
+#[arg(short, long, value_enum)]
+filesystem: Option,
+
+/// Numerical ID of `rpool` ZFS pool to import. Needed if multiple pools
of name `rpool` are present.
+#[arg(long)]
+rpool_id: Option,
+
+/// UUID of the BTRFS file system to mount. Needed if multiple BTRFS file
systems are present.
+#[arg(long)]
+btrfs_uuid: Option,
+}
+
+/// Unmount everything. Use once done with chroot.
+#[derive(Args, Debug)]
+struct CommandCleanup {
+/// Filesystem used for the installation. Will try to automatically detect
it by default.
+#[arg(short, long, value_enum)]
+filesystem: Option,
+}
+
+#[derive(Copy, Clone, Debug, ValueEnum)]
+enum Filesystems {
+Zfs,
+Ext4,
+Xfs,
+Btrfs,
+}
+
+impl From for Filesystems {
+fn from(fs: FsType) -> Self {
+match fs {
+FsType::Xfs => Self::Xfs,
+FsType::Ext4 => Self::Ext4,
+FsType::Zfs(_) => Self::Zfs,
+FsType::Btrfs(_) => Self
[pve-devel] [PATCH installer v6 26/36] auto installer: factor out fetch-answer and autoinst-helper
Putting proxmox-fetch-answer into it's own crate, will keep the use of
OpenSSL localized to where we need it. Otherwise building other binaries
will always depend on OpenSSL as well, even without actually needing it.
Having a dedicated crate for the proxmox-autoinst-helper should make it
easier to build it independently to have it available outside of the
install environment.
The fetch plugins have been moved to the proxmox-fetch-answer crate,
except for the 'get_nic_list' function and 'sysinfo.rs'. Since both are
also needed by the proxmox-autoinst-helper, they are kept in the
proxmox-auto-installer crate.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
Cargo.toml| 2 ++
Makefile | 5 +++-
proxmox-auto-installer/Cargo.toml | 6
.../src/fetch_plugins/mod.rs | 3 --
proxmox-auto-installer/src/lib.rs | 2 +-
.../src/{fetch_plugins/utils => }/sysinfo.rs | 2 +-
proxmox-auto-installer/src/utils.rs | 25
proxmox-autoinst-helper/Cargo.toml| 21 +
.../src/main.rs | 5 ++--
proxmox-fetch-answer/Cargo.toml | 22 ++
.../src/fetch_plugins/http.rs | 3 +-
proxmox-fetch-answer/src/fetch_plugins/mod.rs | 3 ++
.../src/fetch_plugins/partition.rs| 2 +-
.../src/fetch_plugins/utils/mod.rs| 30 +--
.../src/fetch_plugins/utils/post.rs | 2 +-
.../src/main.rs | 8 ++---
16 files changed, 90 insertions(+), 51 deletions(-)
delete mode 100644 proxmox-auto-installer/src/fetch_plugins/mod.rs
rename proxmox-auto-installer/src/{fetch_plugins/utils => }/sysinfo.rs (98%)
create mode 100644 proxmox-autoinst-helper/Cargo.toml
rename proxmox-auto-installer/src/bin/proxmox-autoinst-helper.rs =>
proxmox-autoinst-helper/src/main.rs (98%)
create mode 100644 proxmox-fetch-answer/Cargo.toml
rename {proxmox-auto-installer =>
proxmox-fetch-answer}/src/fetch_plugins/http.rs (98%)
create mode 100644 proxmox-fetch-answer/src/fetch_plugins/mod.rs
rename {proxmox-auto-installer =>
proxmox-fetch-answer}/src/fetch_plugins/partition.rs (100%)
rename {proxmox-auto-installer =>
proxmox-fetch-answer}/src/fetch_plugins/utils/mod.rs (81%)
rename {proxmox-auto-installer =>
proxmox-fetch-answer}/src/fetch_plugins/utils/post.rs (99%)
rename proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs =>
proxmox-fetch-answer/src/main.rs (93%)
diff --git a/Cargo.toml b/Cargo.toml
index 7017ac5..b694d5b 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,6 +1,8 @@
[workspace]
members = [
"proxmox-auto-installer",
+"proxmox-autoinst-helper",
+"proxmox-fetch-answer",
"proxmox-installer-common",
"proxmox-tui-installer",
]
diff --git a/Makefile b/Makefile
index 197a351..e32d28f 100644
--- a/Makefile
+++ b/Makefile
@@ -52,6 +52,8 @@ $(BUILDDIR):
proxinstall \
proxmox-low-level-installer \
proxmox-auto-installer/ \
+ proxmox-autoinst-helper/ \
+ proxmox-fetch-answer/ \
proxmox-tui-installer/ \
proxmox-installer-common/ \
test/ \
@@ -124,7 +126,8 @@ $(COMPILED_BINS): cargo-build
cargo-build:
$(CARGO) build --package proxmox-tui-installer --bin
proxmox-tui-installer \
--package proxmox-auto-installer --bin proxmox-auto-installer \
- --bin proxmox-fetch-answer --bin proxmox-autoinst-helper
$(CARGO_BUILD_ARGS)
+ --package proxmox-fetch-answer --bin proxmox-fetch-answer \
+ --package proxmox-autoinst-helper --bin proxmox-autoinst-helper
$(CARGO_BUILD_ARGS)
%-banner.png: %-banner.svg
rsvg-convert -o $@ $<
diff --git a/proxmox-auto-installer/Cargo.toml
b/proxmox-auto-installer/Cargo.toml
index ac2f3a6..bb0b49c 100644
--- a/proxmox-auto-installer/Cargo.toml
+++ b/proxmox-auto-installer/Cargo.toml
@@ -18,9 +18,3 @@ toml = "0.7"
enum-iterator = "0.6.0"
log = "0.4.20"
regex = "1.7"
-ureq = { version = "2.6", features = [ "native-certs", "native-tls" ] }
-rustls = { version = "0.20", features = [ "dangerous_configuration" ] }
-rustls-native-certs = "0.6"
-native-tls = "0.2"
-sha2 = "0.10"
-hex = "0.4"
diff --git a/proxmox-auto-installer/src/fetch_plugins/mod.rs
b/proxmox-auto-installer/src/fetch_plugins/mod.rs
deleted file mode 100644
index 354fa7e..000
--- a/proxmox-auto-installer/src/fetch_plugins/mod.rs
+++ /dev/null
@@ -1,3 +0,0 @@
-pub mod http;
-pub mod partition;
-pub mod utils;
diff --git a/proxmox-auto-installer/src/lib.rs
b/proxmox-auto-installer/src/lib.rs
index 0a153b2..3bdf0b5 100644
--- a/proxmox-auto-installer/src/lib.rs
+++ b/proxmox-auto-installer/src/lib.rs
@@ -1,5 +1,5 @@
pub mod answer;
-pub mod fetch_plugins;
pub mod log;
+pub mod sysinfo;
pub mod udevinfo;
pub mod utils;
diff --git a
[pve-devel] [PATCH installer v6 31/36] auto-installer: answer: deny unknown fields
This way, serde will throw errors if fields are not known.
This can help to reduce frustration if one might think to have set an
option, but for example a small type has happened.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/src/answer.rs | 16 +++-
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/proxmox-auto-installer/src/answer.rs
b/proxmox-auto-installer/src/answer.rs
index 94cebb3..57c2602 100644
--- a/proxmox-auto-installer/src/answer.rs
+++ b/proxmox-auto-installer/src/answer.rs
@@ -10,7 +10,7 @@ use std::{collections::BTreeMap, net::IpAddr};
/// storing them in a HashMap
#[derive(Clone, Deserialize, Debug)]
-#[serde(rename_all = "kebab-case")]
+#[serde(rename_all = "kebab-case", deny_unknown_fields)]
pub struct Answer {
pub global: Global,
pub network: Network,
@@ -19,6 +19,7 @@ pub struct Answer {
}
#[derive(Clone, Deserialize, Debug)]
+#[serde(deny_unknown_fields)]
pub struct Global {
pub country: String,
pub fqdn: Fqdn,
@@ -33,6 +34,7 @@ pub struct Global {
}
#[derive(Clone, Deserialize, Debug)]
+#[serde(deny_unknown_fields)]
struct NetworkInAnswer {
#[serde(default)]
pub use_dhcp: bool,
@@ -43,7 +45,7 @@ struct NetworkInAnswer {
}
#[derive(Clone, Deserialize, Debug)]
-#[serde(try_from = "NetworkInAnswer")]
+#[serde(try_from = "NetworkInAnswer", deny_unknown_fields)]
pub struct Network {
pub network_settings: NetworkSettings,
}
@@ -97,6 +99,7 @@ pub struct NetworkManual {
}
#[derive(Clone, Debug, Deserialize)]
+#[serde(deny_unknown_fields)]
pub struct DiskSetup {
pub filesystem: Filesystem,
#[serde(default)]
@@ -109,7 +112,7 @@ pub struct DiskSetup {
}
#[derive(Clone, Debug, Deserialize)]
-#[serde(try_from = "DiskSetup")]
+#[serde(try_from = "DiskSetup", deny_unknown_fields)]
pub struct Disks {
pub fs_type: FsType,
pub disk_selection: DiskSelection,
@@ -207,14 +210,14 @@ pub enum DiskSelection {
Filter(BTreeMap),
}
#[derive(Clone, Deserialize, Debug, PartialEq, ValueEnum)]
-#[serde(rename_all = "lowercase")]
+#[serde(rename_all = "lowercase", deny_unknown_fields)]
pub enum FilterMatch {
Any,
All,
}
#[derive(Clone, Deserialize, Serialize, Debug, PartialEq)]
-#[serde(rename_all = "lowercase")]
+#[serde(rename_all = "lowercase", deny_unknown_fields)]
pub enum Filesystem {
Ext4,
Xfs,
@@ -223,6 +226,7 @@ pub enum Filesystem {
}
#[derive(Clone, Copy, Default, Deserialize, Debug)]
+#[serde(deny_unknown_fields)]
pub struct ZfsOptions {
pub raid: Option,
pub ashift: Option,
@@ -234,6 +238,7 @@ pub struct ZfsOptions {
}
#[derive(Clone, Copy, Default, Deserialize, Serialize, Debug)]
+#[serde(deny_unknown_fields)]
pub struct LvmOptions {
pub hdsize: Option,
pub swapsize: Option,
@@ -243,6 +248,7 @@ pub struct LvmOptions {
}
#[derive(Clone, Copy, Default, Deserialize, Debug)]
+#[serde(deny_unknown_fields)]
pub struct BtrfsOptions {
pub hdsize: Option,
pub raid: Option,
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager 4/4] ui: enable upload/download buttons for 'import' type storages
On April 16, 2024 3:19 pm, Dominik Csapak wrote:
> but only for non esxi ones, since that does not allow
> uploading/downloading there
what about a remove button? :)
>
> Signed-off-by: Dominik Csapak
> ---
> www/manager6/storage/Browser.js| 7 ++-
> www/manager6/window/UploadToStorage.js | 1 +
> 2 files changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/www/manager6/storage/Browser.js b/www/manager6/storage/Browser.js
> index 2123141d..77d106c1 100644
> --- a/www/manager6/storage/Browser.js
> +++ b/www/manager6/storage/Browser.js
> @@ -28,7 +28,9 @@ Ext.define('PVE.storage.Browser', {
> let res = storageInfo.data;
> let plugin = res.plugintype;
>
> - me.items = plugin !== 'esxi' ? [
> + let isEsxi = plugin === 'esxi';
> +
> + me.items = !isEsxi ? [
> {
> title: gettext('Summary'),
> xtype: 'pveStorageSummary',
> @@ -144,6 +146,9 @@ Ext.define('PVE.storage.Browser', {
> content: 'import',
> useCustomRemoveButton: true, // hide default remove button
> showColumns: ['name', 'format'],
> + enableUploadButton: enableUpload && !isEsxi,
> + enableDownloadUrlButton: enableDownloadUrl && !isEsxi,
> + useUploadButton: !isEsxi,
> itemdblclick: (view, record) =>
> createGuestImportWindow(record),
> tbar: [
> {
> diff --git a/www/manager6/window/UploadToStorage.js
> b/www/manager6/window/UploadToStorage.js
> index 3c5bba88..79a6e8a6 100644
> --- a/www/manager6/window/UploadToStorage.js
> +++ b/www/manager6/window/UploadToStorage.js
> @@ -11,6 +11,7 @@ Ext.define('PVE.window.UploadToStorage', {
> acceptedExtensions: {
> iso: ['.img', '.iso'],
> vztmpl: ['.tar.gz', '.tar.xz', '.tar.zst'],
> + 'import': ['ova'],
> },
>
> cbindData: function(initialConfig) {
> --
> 2.39.2
>
>
>
> ___
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v6 13/36] auto-installer: add tests for answer file parsing
By matching the resulting json to be passed to the low level installer
against known good ones.
The environment info was gathered from one of our AMD Epyc Rome test
servers to have a realistic starting point.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/tests/parse-answer.rs | 106 ++
.../tests/resources/iso-info.json | 1 +
.../tests/resources/locales.json | 1 +
.../resources/parse_answer/disk_match.json| 29 +
.../resources/parse_answer/disk_match.toml| 17 +++
.../parse_answer/disk_match_all.json | 26 +
.../parse_answer/disk_match_all.toml | 17 +++
.../parse_answer/disk_match_any.json | 33 ++
.../parse_answer/disk_match_any.toml | 17 +++
.../tests/resources/parse_answer/minimal.json | 17 +++
.../tests/resources/parse_answer/minimal.toml | 14 +++
.../resources/parse_answer/nic_matching.json | 17 +++
.../resources/parse_answer/nic_matching.toml | 19
.../tests/resources/parse_answer/readme | 4 +
.../resources/parse_answer/specific_nic.json | 17 +++
.../resources/parse_answer/specific_nic.toml | 19
.../tests/resources/parse_answer/zfs.json | 27 +
.../tests/resources/parse_answer/zfs.toml | 20
.../tests/resources/run-env-info.json | 1 +
.../tests/resources/run-env-udev.json | 1 +
20 files changed, 403 insertions(+)
create mode 100644 proxmox-auto-installer/tests/parse-answer.rs
create mode 100644 proxmox-auto-installer/tests/resources/iso-info.json
create mode 100644 proxmox-auto-installer/tests/resources/locales.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/disk_match.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/disk_match.toml
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.toml
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.toml
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/minimal.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/minimal.toml
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/nic_matching.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/nic_matching.toml
create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/readme
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/specific_nic.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/specific_nic.toml
create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.json
create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.toml
create mode 100644 proxmox-auto-installer/tests/resources/run-env-info.json
create mode 100644 proxmox-auto-installer/tests/resources/run-env-udev.json
diff --git a/proxmox-auto-installer/tests/parse-answer.rs
b/proxmox-auto-installer/tests/parse-answer.rs
new file mode 100644
index 000..c12520f
--- /dev/null
+++ b/proxmox-auto-installer/tests/parse-answer.rs
@@ -0,0 +1,106 @@
+use std::path::PathBuf;
+
+use serde_json::Value;
+use std::fs;
+
+use proxmox_auto_installer::answer;
+use proxmox_auto_installer::answer::Answer;
+use proxmox_auto_installer::udevinfo::UdevInfo;
+use proxmox_auto_installer::utils::parse_answer;
+
+use proxmox_installer_common::setup::{read_json, LocaleInfo, RuntimeInfo,
SetupInfo};
+
+fn get_test_resource_path() -> Result {
+Ok(std::env::current_dir()
+.expect("current dir failed")
+.join("tests/resources"))
+}
+fn get_answer(path: PathBuf) -> Result {
+let answer_raw = std::fs::read_to_string(&path).unwrap();
+let answer: answer::Answer = toml::from_str(&answer_raw)
+.map_err(|err| format!("error parsing answer.toml: {err}"))
+.unwrap();
+
+Ok(answer)
+}
+
+fn setup_test_basic(path: &PathBuf) -> (SetupInfo, LocaleInfo, RuntimeInfo,
UdevInfo) {
+let installer_info: SetupInfo = {
+let mut path = path.clone();
+path.push("iso-info.json");
+
+read_json(&path)
+.map_err(|err| format!("Failed to retrieve setup info: {err}"))
+.unwrap()
+};
+
+let locale_info = {
+let mut path = path.clone();
+path.push("locales.json");
+
+read_json(&path)
+.map_err(|err| format!("Failed to retrieve locale info: {err}"))
+.unwrap()
+};
+
+let mut runtime_info: RuntimeInfo = {
+let mut path = path.clone();
+path.push("run-env-info.json");
+
+read_json(&path)
+.map_err(|err| format!("Failed to retrieve runtime environment
Re: [pve-devel] [PATCH installer v5 00/36] add automated/unattended installation
a new v6 has been posted that includes the t-b and r-b tags as well as some smaller style fixes in the most recent patches https://lists.proxmox.com/pipermail/pve-devel/2024-April/063139.html On 2024-04-16 17:32, Aaron Lauterer wrote: patches until 31 got a [0,1] Tested-by: Christoph Heiss Reviewed-by: Christoph Heiss changes since v4: Patches 32-36 finalize how to prepare an ISO for automated installation and introduce a slight change in behavior, as it is now also possible to include the needed parameters into the ISO directly: * answer file itself * URL to fetch it from * SSL cert fingerprint The 'proxmox-autoinst-helper' tool got a new subcommand to prepare an ISO. The cover letter iself: The overall idea is to prepare an ISO for automated installation. A prepare ISO will automatically boot into the installation. The information for the installer that is usually gathered interactively from the user is provided via an `answer.toml` file. The answer file allows to select disks and the network card via filters. The installer also allows to run custom commands pre and post installation. This should give users plenty of possibilities to either further customize/prepare the installation or integrate it into a larger automated installation setup. For example, one could issue HTTP requests to signal the status and progress of the installation. When the installer is called with 'proxauto' in the kernel cmdline, the 'proxmox-fetch-answer' binary is called. It tries to find the answer file and once found, will start the 'proxmox-auto-installer' binary and pass the contents to it via stdin. The auto-installer then parses the answer file and determines what parameters need to be passed to the low-level installer. For example, which disks and NIC to use, network IP settings and so forth. The current status reporting of the actual installation is kept rather simple. Both binaries log into the tmp directory. There is a third binary, the 'proxmox-autoinst-helper'. It provides a few subcommands, from the help: prepare-iso Prepare an ISO for automated installation. validate-answer Validate if an answer file is formatted correctly device-match Test which devices the given filter matches against device-info Show device information that can be used for filters identifiers Show identifiers for the current machine. This information is part of the POST request to fetch an answer file help Print this message or the help of the given subcommand(s) The fetch-answer binary is trying to get an answer file. If included in the ISO, it will use that one. If no answer file is included, it first searches for a partition/FS labeled `proxmoxinst`, or all upper case, and an `answer.toml` in there. This could be provided by another USB flash drive. If that is not successful, the next step is to send an HTTP POST request to a URL to get the TOML contents in return. A POST request was chosen because we also send information to identify the host in JSON format. The question then is, where to get that URL from. Right now, there are three options implemented. The first is to hardcode it in the prepared iso. The second is to look for a custom DHCP option and the third is to query for a TXT record in the `proxmoxinst` subdomain of the search domain. It is possible to provide a SHA256 fingerprint of the SSL cert used by the answer server. The safest option is to include it in the ISO itself. If that is not found, then it can be provided by a second custom DHCP option or placed as TXT record in the subdomain `proxmoxinst-fp`. This patch series now also separates the 3 binaries into their own crate. The 'proxmox-fetch-answer' to keep the OpenSSL dependency as localized as possible, and the 'proxmox-autoinst-helper' to make it easy to compile just that binary. The new `proxmox-chroot` utility helps to prepare everything to chroot into a fresh installation and clean it up once done. This will be useful in the post commands when further customizing the installation. Other plans / ideas for the future: * add option to define remote SSH access (password and,or public key). This could make remote debugging in case of problems easier Regarding the patch series itself: 01-03 are needed to move some code into the common crate and make structs/functions already in the common crate accessible. I did split up the individual parts of the auto installer into their own patches as much as possible, and (hopefully) in the order they depend on each other. Patches after the `unconfigured` one (16), switch the pattern matching to the glob crate, add the helper tool and the fetching via HTTP. Patch 26 factors our the binaries into their own crates. Patches 27-30 are for the 'proxmox-chroot' utility and preparations for it to work. Patch 31 makes sure that the answer file can only contain known keys. Patches 32 - 36 finalize the ISO preparation, add the subcommand for it to the a
[pve-devel] [PATCH installer v6 11/36] auto-installer: add utils
contains several utility structs and functions.
For example: a simple pattern matcher that matches wildcards at the
beginning or end of the filter.
It currently uses a dedicated function (parse_answer) to generate the
InstallConfig struct instead of a From implementation. This is because
for now the source data is spread over several other structs in
comparison to one in the TUI installer.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/src/lib.rs | 1 +
proxmox-auto-installer/src/utils.rs | 424
2 files changed, 425 insertions(+)
create mode 100644 proxmox-auto-installer/src/utils.rs
diff --git a/proxmox-auto-installer/src/lib.rs
b/proxmox-auto-installer/src/lib.rs
index 8cda416..72884c1 100644
--- a/proxmox-auto-installer/src/lib.rs
+++ b/proxmox-auto-installer/src/lib.rs
@@ -1,2 +1,3 @@
pub mod answer;
pub mod udevinfo;
+pub mod utils;
diff --git a/proxmox-auto-installer/src/utils.rs
b/proxmox-auto-installer/src/utils.rs
new file mode 100644
index 000..ae28b1e
--- /dev/null
+++ b/proxmox-auto-installer/src/utils.rs
@@ -0,0 +1,424 @@
+use anyhow::{bail, Result};
+use log::info;
+use std::{
+collections::BTreeMap,
+process::{Command, Stdio},
+};
+
+use crate::{
+answer::{self, Answer},
+udevinfo::UdevInfo,
+};
+use proxmox_installer_common::{
+options::{FsType, NetworkOptions, ZfsChecksumOption, ZfsCompressOption},
+setup::{InstallConfig, InstallZfsOption, LocaleInfo, RuntimeInfo,
SetupInfo},
+};
+use serde::Deserialize;
+
+/// Supports the globbing character '*' at the beginning, end or both of the
pattern.
+/// Globbing within the pattern is not supported
+fn find_with_glob(pattern: &str, value: &str) -> bool {
+let globbing_symbol = '*';
+let mut start_glob = false;
+let mut end_glob = false;
+let mut pattern = pattern;
+
+if pattern.starts_with(globbing_symbol) {
+start_glob = true;
+pattern = &pattern[1..];
+}
+
+if pattern.ends_with(globbing_symbol) {
+end_glob = true;
+pattern = &pattern[..pattern.len() - 1]
+}
+
+match (start_glob, end_glob) {
+(true, true) => value.contains(pattern),
+(true, false) => value.ends_with(pattern),
+(false, true) => value.starts_with(pattern),
+_ => value == pattern,
+}
+}
+
+pub fn get_network_settings(
+answer: &Answer,
+udev_info: &UdevInfo,
+runtime_info: &RuntimeInfo,
+setup_info: &SetupInfo,
+) -> Result {
+let mut network_options = NetworkOptions::defaults_from(setup_info,
&runtime_info.network);
+
+info!("Setting network configuration");
+
+// Always use the FQDN from the answer file
+network_options.fqdn = answer.global.fqdn.clone();
+
+if let answer::NetworkSettings::Manual(settings) =
&answer.network.network_settings {
+network_options.address = settings.cidr.clone();
+network_options.dns_server = settings.dns;
+network_options.gateway = settings.gateway;
+network_options.ifname =
get_single_udev_index(settings.filter.clone(), &udev_info.nics)?;
+}
+info!("Network interface used is '{}'", &network_options.ifname);
+Ok(network_options)
+}
+
+fn get_single_udev_index(
+filter: BTreeMap,
+udev_list: &BTreeMap>,
+) -> Result {
+if filter.is_empty() {
+bail!("no filter defined");
+}
+let mut dev_index: Option = None;
+'outer: for (dev, dev_values) in udev_list {
+for (filter_key, filter_value) in &filter {
+for (udev_key, udev_value) in dev_values {
+if udev_key == filter_key && find_with_glob(filter_value,
udev_value) {
+dev_index = Some(dev.clone());
+break 'outer; // take first match
+}
+}
+}
+}
+if dev_index.is_none() {
+bail!("filter did not match any device");
+}
+
+Ok(dev_index.unwrap())
+}
+
+fn get_matched_udev_indexes(
+filter: BTreeMap,
+udev_list: &BTreeMap>,
+match_all: bool,
+) -> Result> {
+let mut matches = vec![];
+for (dev, dev_values) in udev_list {
+let mut did_match_once = false;
+let mut did_match_all = true;
+for (filter_key, filter_value) in &filter {
+for (udev_key, udev_value) in dev_values {
+if udev_key == filter_key && find_with_glob(filter_value,
udev_value) {
+did_match_once = true;
+} else if udev_key == filter_key {
+did_match_all = false;
+}
+}
+}
+if (match_all && did_match_all) || (!match_all && did_match_once) {
+matches.push(dev.clone());
+}
+}
+if matches.is_empty() {
+bail!("filter did not match any devices");
+}
+matches.sort();
+Ok(matches)
+}
+
+pub fn set_disks(
+answer: &Answer,
+udev_info: &UdevInfo,
+
[pve-devel] [PATCH installer v6 21/36] auto-installer: fetch: add gathering of system identifiers and restructure code
They will be used as payload when POSTing a request for an answer file. The
idea is, that with this information, it should be possible to identify
the system and generate a matching answer file on the fly.
Many of these properties can also be found on the machine or packaging
of the machine and could therefore be scanned into a database.
Identifiers are the following properties from `dmidecode` sections 1, 2,
and 3:
* Asset Tag
* Product Name
* Serial Number
* SKU Number
* UUID
As well as a list of the MAC addresses of all the NICs and the product
type: pve, pmg, pbs.
Since we now have more than a simple utils.rs module in the fetch
plugins, it, and the additional fetch plugin utilities are placed in
their own directory.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
.../src/fetch_plugins/mod.rs | 2 +-
.../fetch_plugins/{utils.rs => utils/mod.rs} | 43 --
.../src/fetch_plugins/utils/sysinfo.rs| 81 +++
3 files changed, 119 insertions(+), 7 deletions(-)
rename proxmox-auto-installer/src/fetch_plugins/{utils.rs => utils/mod.rs}
(72%)
create mode 100644 proxmox-auto-installer/src/fetch_plugins/utils/sysinfo.rs
diff --git a/proxmox-auto-installer/src/fetch_plugins/mod.rs
b/proxmox-auto-installer/src/fetch_plugins/mod.rs
index 11d6937..6f1e8a2 100644
--- a/proxmox-auto-installer/src/fetch_plugins/mod.rs
+++ b/proxmox-auto-installer/src/fetch_plugins/mod.rs
@@ -1,2 +1,2 @@
pub mod partition;
-mod utils;
+pub mod utils;
diff --git a/proxmox-auto-installer/src/fetch_plugins/utils.rs
b/proxmox-auto-installer/src/fetch_plugins/utils/mod.rs
similarity index 72%
rename from proxmox-auto-installer/src/fetch_plugins/utils.rs
rename to proxmox-auto-installer/src/fetch_plugins/utils/mod.rs
index f2a8e74..b3e9dad 100644
--- a/proxmox-auto-installer/src/fetch_plugins/utils.rs
+++ b/proxmox-auto-installer/src/fetch_plugins/utils/mod.rs
@@ -1,5 +1,7 @@
-use anyhow::{bail, Error, Result};
+use anyhow::{Error, Result};
use log::{info, warn};
+use serde::Deserialize;
+use serde_json;
use std::{
fs::{self, create_dir_all},
path::{Path, PathBuf},
@@ -10,6 +12,8 @@ static ANSWER_MP: &str = "/mnt/answer";
static PARTLABEL: &str = "proxmoxinst";
static SEARCH_PATH: &str = "/dev/disk/by-label";
+pub mod sysinfo;
+
/// Searches for upper and lower case existence of the partlabel in the
search_path
///
/// # Arguemnts
@@ -20,10 +24,10 @@ pub fn scan_partlabels(partlabel_source: &str, search_path:
&str) -> Result {
-info!("Found partition with label '{partlabel}'");
+info!("Found partition with label '{}'", partlabel);
return Ok(path);
}
-Ok(false) => info!("Did not detect partition with label
'{partlabel}'"),
+Ok(false) => info!("Did not detect partition with label '{}'",
partlabel),
Err(err) => info!("Encountered issue, accessing '{}': {}",
path.display(), err),
}
@@ -31,13 +35,15 @@ pub fn scan_partlabels(partlabel_source: &str, search_path:
&str) -> Result {
-info!("Found partition with label '{partlabel}'");
+info!("Found partition with label '{}'", partlabel);
return Ok(path);
}
-Ok(false) => info!("Did not detect partition with label
'{partlabel}'"),
+Ok(false) => info!("Did not detect partition with label '{}'",
partlabel),
Err(err) => info!("Encountered issue, accessing '{}': {}",
path.display(), err),
}
-bail!("Could not detect upper or lower case labels for
'{partlabel_source}'");
+Err(Error::msg(format!(
+"Could not detect upper or lower case labels for '{partlabel_source}'"
+)))
}
/// Will search and mount a partition/FS labeled proxmoxinst in lower or
uppercase to ANSWER_MP;
@@ -79,3 +85,28 @@ fn check_if_mounted(target_path: &str) -> Result {
}
Ok(false)
}
+
+#[derive(Deserialize, Debug)]
+struct IpLinksUdevInfo {
+ifname: String,
+}
+
+/// Returns vec of usable NICs
+pub fn get_nic_list() -> Result> {
+let ip_output = Command::new("/usr/sbin/ip")
+.arg("-j")
+.arg("link")
+.output()?;
+let parsed_links: Vec =
+serde_json::from_str(String::from_utf8(ip_output.stdout)?.as_str())?;
+let mut links: Vec = Vec::new();
+
+for link in parsed_links {
+if link.ifname == *"lo" {
+continue;
+}
+links.push(link.ifname);
+}
+
+Ok(links)
+}
diff --git a/proxmox-auto-installer/src/fetch_plugins/utils/sysinfo.rs
b/proxmox-auto-installer/src/fetch_plugins/utils/sysinfo.rs
new file mode 100644
index 000..8c57283
--- /dev/null
+++ b/proxmox-auto-installer/src/fetch_plugins/utils/sysinfo.rs
@@ -0,0 +1,81 @@
+use anyhow::{bail, Result};
+use proxmox_installer_common::setup::SetupInfo;
+use serde::Serialize;
+use std::{collections::HashMap, fs, io, path::Path};
+
+use super::get_nic_list;
+
+const DMI_PATH: &str = "/sys/de
[pve-devel] [PATCH installer v6 22/36] auto-installer: helper: add subcommand to view indentifiers
It will collect the information from the current system and show the
payload of identifiers that will be send.
To avoid confusion, the subcommands for the device info and filter
matching have been renamed.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
.../src/bin/proxmox-autoinst-helper.rs| 54 +--
1 file changed, 26 insertions(+), 28 deletions(-)
diff --git a/proxmox-auto-installer/src/bin/proxmox-autoinst-helper.rs
b/proxmox-auto-installer/src/bin/proxmox-autoinst-helper.rs
index 058d5ff..f0ee8f4 100644
--- a/proxmox-auto-installer/src/bin/proxmox-autoinst-helper.rs
+++ b/proxmox-auto-installer/src/bin/proxmox-autoinst-helper.rs
@@ -2,12 +2,13 @@ use anyhow::{bail, Result};
use clap::{Args, Parser, Subcommand, ValueEnum};
use glob::Pattern;
use regex::Regex;
-use serde::{Deserialize, Serialize};
+use serde::Serialize;
use std::{collections::BTreeMap, fs, io::Read, path::PathBuf,
process::Command};
use proxmox_auto_installer::{
answer::Answer,
answer::FilterMatch,
+fetch_plugins::utils::{sysinfo, get_nic_list},
utils::{get_matched_udev_indexes, get_single_udev_index},
};
@@ -23,13 +24,14 @@ struct Cli {
#[derive(Subcommand, Debug)]
enum Commands {
ValidateAnswer(CommandValidateAnswer),
-Match(CommandMatch),
-Info(CommandInfo),
+DeviceMatch(CommandDeviceMatch),
+DeviceInfo(CommandDeviceInfo),
+Identifiers(CommandIdentifiers),
}
/// Show device information that can be used for filters
#[derive(Args, Debug)]
-struct CommandInfo {
+struct CommandDeviceInfo {
/// For which device type information should be shown
#[arg(name="type", short, long, value_enum,
default_value_t=AllDeviceTypes::All)]
device: AllDeviceTypes,
@@ -52,7 +54,7 @@ struct CommandInfo {
/// proxmox-autoinst-helper match --filter-match all disk
'ID_SERIAL_SHORT=**' 'DEVNAME=*nvme*'
#[derive(Args, Debug)]
#[command(verbatim_doc_comment)]
-struct CommandMatch {
+struct CommandDeviceMatch {
/// Device type to match the filter against
r#type: Devicetype,
@@ -74,6 +76,11 @@ struct CommandValidateAnswer {
debug: bool,
}
+/// Show identifiers for the current machine. This information is part of the
POST request to fetch
+/// an answer file.
+#[derive(Args, Debug)]
+struct CommandIdentifiers {}
+
#[derive(Args, Debug)]
struct GlobalOpts {
/// Output format
@@ -109,9 +116,10 @@ struct Devs {
fn main() {
let args = Cli::parse();
let res = match &args.command {
-Commands::Info(args) => info(args),
-Commands::Match(args) => match_filter(args),
Commands::ValidateAnswer(args) => validate_answer(args),
+Commands::DeviceInfo(args) => info(args),
+Commands::DeviceMatch(args) => match_filter(args),
+Commands::Identifiers(args) => show_identifiers(args),
};
if let Err(err) = res {
eprintln!("{err}");
@@ -119,7 +127,7 @@ fn main() {
}
}
-fn info(args: &CommandInfo) -> Result<()> {
+fn info(args: &CommandDeviceInfo) -> Result<()> {
let mut devs = Devs {
disks: None,
nics: None,
@@ -141,7 +149,7 @@ fn info(args: &CommandInfo) -> Result<()> {
Ok(())
}
-fn match_filter(args: &CommandMatch) -> Result<()> {
+fn match_filter(args: &CommandDeviceMatch) -> Result<()> {
let devs: BTreeMap> = match args.r#type {
Devicetype::Disk => get_disks().unwrap(),
Devicetype::Network => get_nics().unwrap(),
@@ -205,6 +213,14 @@ fn validate_answer(args: &CommandValidateAnswer) ->
Result<()> {
Ok(())
}
+fn show_identifiers(_args: &CommandIdentifiers) -> Result<()> {
+match sysinfo::get_sysinfo(true) {
+Ok(res) => println!("{res}"),
+Err(err) => eprintln!("Error fetching system identifiers: {err}"),
+}
+Ok(())
+}
+
fn get_disks() -> Result>> {
let unwantend_block_devs = vec![
"ram[0-9]*",
@@ -275,30 +291,12 @@ fn get_disks() -> Result>> {
Ok(disks)
}
-#[derive(Deserialize, Debug)]
-struct IpLinksInfo {
-ifname: String,
-}
fn get_nics() -> Result>> {
let re_props = Regex::new(r"(?m)^E: (.*)=(.*)$")?;
let mut nics: BTreeMap> = BTreeMap::new();
-let ip_output = Command::new("/usr/sbin/ip")
-.arg("-j")
-.arg("link")
-.output()?;
-let parsed_links: Vec =
-serde_json::from_str(String::from_utf8(ip_output.stdout)?.as_str())?;
-let mut links: Vec = Vec::new();
-
-for link in parsed_links {
-if link.ifname == *"lo" {
-continue;
-}
-links.push(link.ifname);
-}
-
+let links = get_nic_list()?;
for link in links {
let path = format!("/sys/class/net/{link}");
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v6 28/36] common: add deserializer for FsType
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
proxmox-installer-common/Cargo.toml | 1 +
proxmox-installer-common/src/options.rs | 10 ++---
proxmox-installer-common/src/setup.rs | 30 ++---
3 files changed, 35 insertions(+), 6 deletions(-)
diff --git a/proxmox-installer-common/Cargo.toml
b/proxmox-installer-common/Cargo.toml
index bde5457..70f828a 100644
--- a/proxmox-installer-common/Cargo.toml
+++ b/proxmox-installer-common/Cargo.toml
@@ -8,5 +8,6 @@ exclude = [ "build", "debian" ]
homepage = "https://www.proxmox.com";
[dependencies]
+regex = "1.7"
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
diff --git a/proxmox-installer-common/src/options.rs
b/proxmox-installer-common/src/options.rs
index 1efac66..1e782f9 100644
--- a/proxmox-installer-common/src/options.rs
+++ b/proxmox-installer-common/src/options.rs
@@ -1,6 +1,6 @@
+use serde::Deserialize;
use std::net::{IpAddr, Ipv4Addr};
use std::{cmp, fmt};
-use serde::Deserialize;
use crate::setup::{
LocaleInfo, NetworkInfo, ProductConfig, ProxmoxProduct, RuntimeInfo,
SetupInfo,
@@ -32,8 +32,11 @@ pub enum ZfsRaidLevel {
Raid0,
Raid1,
Raid10,
+#[serde(rename = "raidz-1")]
RaidZ,
+#[serde(rename = "raidz-2")]
RaidZ2,
+#[serde(rename = "raidz-3")]
RaidZ3,
}
@@ -51,7 +54,8 @@ impl fmt::Display for ZfsRaidLevel {
}
}
-#[derive(Copy, Clone, Debug, Eq, PartialEq)]
+#[derive(Copy, Clone, Debug, Deserialize, Eq, PartialEq)]
+#[serde(rename_all = "lowercase")]
pub enum FsType {
Ext4,
Xfs,
@@ -226,7 +230,7 @@ pub enum AdvancedBootdiskOptions {
Btrfs(BtrfsBootdiskOptions),
}
-#[derive(Clone, Debug, PartialEq)]
+#[derive(Clone, Debug, Deserialize, PartialEq)]
pub struct Disk {
pub index: String,
pub path: String,
diff --git a/proxmox-installer-common/src/setup.rs
b/proxmox-installer-common/src/setup.rs
index 25d0e9e..c580477 100644
--- a/proxmox-installer-common/src/setup.rs
+++ b/proxmox-installer-common/src/setup.rs
@@ -155,7 +155,7 @@ pub fn installer_setup(in_test_mode: bool) ->
Result<(SetupInfo, LocaleInfo, Run
}
}
-#[derive(Serialize)]
+#[derive(Debug, Deserialize, Serialize)]
pub struct InstallZfsOption {
pub ashift: usize,
#[serde(serialize_with = "serialize_as_display")]
@@ -402,11 +402,11 @@ pub fn spawn_low_level_installer(test_mode: bool) ->
io::Result
}
/// See Proxmox::Install::Config
-#[derive(Serialize)]
+#[derive(Debug, Deserialize, Serialize)]
pub struct InstallConfig {
pub autoreboot: usize,
-#[serde(serialize_with = "serialize_fstype")]
+#[serde(serialize_with = "serialize_fstype", deserialize_with =
"deserialize_fs_type")]
pub filesys: FsType,
pub hdsize: f64,
#[serde(skip_serializing_if = "Option::is_none")]
@@ -481,3 +481,27 @@ where
serializer.collect_str(value)
}
+
+pub fn deserialize_fs_type<'de, D>(deserializer: D) -> Result
+where
+D: Deserializer<'de>,
+{
+use FsType::*;
+let de_fs: String = Deserialize::deserialize(deserializer)?;
+
+println!("deserializing fstype");
+match de_fs.as_str() {
+"ext4" => Ok(Ext4),
+"xfs" => Ok(Xfs),
+"zfs (RAID0)" => Ok(Zfs(ZfsRaidLevel::Raid0)),
+"zfs (RAID1)" => Ok(Zfs(ZfsRaidLevel::Raid1)),
+"zfs (RAID10)" => Ok(Zfs(ZfsRaidLevel::Raid10)),
+"zfs (RAIDZ-1)" => Ok(Zfs(ZfsRaidLevel::RaidZ)),
+"zfs (RAIDZ-2)" => Ok(Zfs(ZfsRaidLevel::RaidZ2)),
+"zfs (RAIDZ-3)" => Ok(Zfs(ZfsRaidLevel::RaidZ3)),
+"btrfs (RAID0)" => Ok(Btrfs(BtrfsRaidLevel::Raid0)),
+"btrfs (RAID1)" => Ok(Btrfs(BtrfsRaidLevel::Raid1)),
+"btrfs (RAID10)" => Ok(Btrfs(BtrfsRaidLevel::Raid10)),
+_ => Err(de::Error::custom("could not find file system: {de_fs}"))
+}
+}
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v6 14/36] auto-installer: add auto-installer binary
It expects the contents of an answer file via stdin. It will then be
parsed and the JSON for the low level installer is generated.
It then calls the low level installer directly.
The output of the installaton progress is kept rather simple for now.
If configured in the answer file, commands will be run pre and post the
low level installer.
It also logs everything to the logfile, currently
'/tmp/auto_installer.log'.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
Makefile | 9 +-
.../src/bin/proxmox-auto-installer.rs | 195 ++
2 files changed, 201 insertions(+), 3 deletions(-)
create mode 100644 proxmox-auto-installer/src/bin/proxmox-auto-installer.rs
diff --git a/Makefile b/Makefile
index 4f140ca..3ac5769 100644
--- a/Makefile
+++ b/Makefile
@@ -18,7 +18,9 @@ INSTALLER_SOURCES=$(shell git ls-files) country.dat
PREFIX = /usr
BINDIR = $(PREFIX)/bin
-USR_BIN := proxmox-tui-installer
+USR_BIN := \
+ proxmox-tui-installer\
+ proxmox-auto-installer
COMPILED_BINS := \
$(addprefix $(CARGO_COMPILEDIR)/,$(USR_BIN))
@@ -99,7 +101,7 @@ VARLIBDIR=$(DESTDIR)/var/lib/proxmox-installer
HTMLDIR=$(VARLIBDIR)/html/common
.PHONY: install
-install: $(INSTALLER_SOURCES) $(CARGO_COMPILEDIR)/proxmox-tui-installer
+install: $(INSTALLER_SOURCES) $(COMPILED_BINS)
$(MAKE) -C banner install
$(MAKE) -C Proxmox install
install -D -m 644 interfaces $(DESTDIR)/etc/network/interfaces
@@ -118,7 +120,8 @@ install: $(INSTALLER_SOURCES)
$(CARGO_COMPILEDIR)/proxmox-tui-installer
$(COMPILED_BINS): cargo-build
.PHONY: cargo-build
cargo-build:
- $(CARGO) build --package proxmox-tui-installer --bin
proxmox-tui-installer $(CARGO_BUILD_ARGS)
+ $(CARGO) build --package proxmox-tui-installer --bin
proxmox-tui-installer \
+ --package proxmox-auto-installer --bin proxmox-auto-installer
$(CARGO_BUILD_ARGS)
%-banner.png: %-banner.svg
rsvg-convert -o $@ $<
diff --git a/proxmox-auto-installer/src/bin/proxmox-auto-installer.rs
b/proxmox-auto-installer/src/bin/proxmox-auto-installer.rs
new file mode 100644
index 000..f43b12f
--- /dev/null
+++ b/proxmox-auto-installer/src/bin/proxmox-auto-installer.rs
@@ -0,0 +1,195 @@
+use anyhow::{anyhow, bail, Error, Result};
+use log::{error, info, LevelFilter};
+use std::{
+env,
+io::{BufRead, BufReader, Write},
+path::PathBuf,
+process::ExitCode,
+};
+
+use proxmox_installer_common::setup::{
+installer_setup, read_json, spawn_low_level_installer, LocaleInfo,
RuntimeInfo, SetupInfo,
+};
+
+use proxmox_auto_installer::{
+answer::Answer,
+log::AutoInstLogger,
+udevinfo::UdevInfo,
+utils,
+utils::{parse_answer, LowLevelMessage},
+};
+
+static LOGGER: AutoInstLogger = AutoInstLogger;
+
+pub fn init_log() -> Result<()> {
+AutoInstLogger::init("/tmp/auto_installer.log")?;
+log::set_logger(&LOGGER)
+.map(|()| log::set_max_level(LevelFilter::Info))
+.map_err(|err| anyhow!(err))
+}
+
+fn auto_installer_setup(in_test_mode: bool) -> Result<(Answer, UdevInfo)> {
+let base_path = if in_test_mode { "./testdir" } else { "/" };
+let mut path = PathBuf::from(base_path);
+
+path.push("run");
+path.push("proxmox-installer");
+
+let udev_info: UdevInfo = {
+let mut path = path.clone();
+path.push("run-env-udev.json");
+
+read_json(&path).map_err(|err| anyhow!("Failed to retrieve udev info
details: {err}"))?
+};
+
+let mut buffer = String::new();
+let lines = std::io::stdin().lock().lines();
+for line in lines {
+buffer.push_str(&line.unwrap());
+buffer.push('\n');
+}
+
+let answer: Answer =
+toml::from_str(&buffer).map_err(|err| anyhow!("Failed parsing answer
file: {err}"))?;
+
+Ok((answer, udev_info))
+}
+
+fn main() -> ExitCode {
+if let Err(err) = init_log() {
+panic!("could not initilize logging: {}", err);
+}
+
+let in_test_mode = match env::args().nth(1).as_deref() {
+Some("-t") => true,
+// Always force the test directory in debug builds
+_ => cfg!(debug_assertions),
+};
+info!("Starting auto installer");
+
+let (setup_info, locales, runtime_info) = match
installer_setup(in_test_mode) {
+Ok(result) => result,
+Err(err) => {
+error!("Installer setup error: {err}");
+return ExitCode::FAILURE;
+}
+};
+
+let (answer, udevadm_info) = match auto_installer_setup(in_test_mode) {
+Ok(result) => result,
+Err(err) => {
+error!("Autoinstaller setup error: {err}");
+return ExitCode::FAILURE;
+}
+};
+
+match utils::run_cmds("Pre", &answer.global.pre_commands) {
+Ok(_) => (),
+Err(err) => {
+error!("Error when running Pre-Commands: {}", err);
+return exit_failure(a
[pve-devel] [PATCH installer v6 33/36] auto-installer: utils: define ISO specified settings
These will be expected on the ISO itself and define the behavior of the
automated installation.
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/src/utils.rs | 20 +++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/proxmox-auto-installer/src/utils.rs
b/proxmox-auto-installer/src/utils.rs
index ff90ae8..997ab34 100644
--- a/proxmox-auto-installer/src/utils.rs
+++ b/proxmox-auto-installer/src/utils.rs
@@ -1,4 +1,5 @@
use anyhow::{bail, Result};
+use clap::ValueEnum;
use glob::Pattern;
use log::info;
use std::{
@@ -14,7 +15,7 @@ use proxmox_installer_common::{
options::{FsType, NetworkOptions, ZfsChecksumOption, ZfsCompressOption},
setup::{InstallConfig, InstallZfsOption, LocaleInfo, RuntimeInfo,
SetupInfo},
};
-use serde::Deserialize;
+use serde::{Deserialize, Serialize};
fn find_with_glob(pattern: &str, value: &str) -> Result {
let p = Pattern::new(pattern)?;
@@ -72,6 +73,23 @@ pub fn get_single_udev_index(
Ok(dev_index.unwrap())
}
+#[derive(Deserialize, Serialize, Debug, Clone, ValueEnum, PartialEq)]
+#[serde(rename_all = "lowercase", deny_unknown_fields)]
+pub enum AutoInstModes {
+Auto,
+Included,
+Http,
+Partition,
+}
+
+#[derive(Deserialize, Serialize, Debug)]
+#[serde(rename_all = "lowercase", deny_unknown_fields)]
+pub struct AutoInstSettings {
+pub mode: AutoInstModes,
+pub http_url: Option,
+pub cert_fingerprint: Option,
+}
+
#[derive(Deserialize, Debug)]
struct IpLinksUdevInfo {
ifname: String,
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v6 09/36] auto-installer: add answer file definition
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/Cargo.toml| 1 +
proxmox-auto-installer/src/answer.rs | 248 +++
proxmox-auto-installer/src/lib.rs| 1 +
3 files changed, 250 insertions(+)
create mode 100644 proxmox-auto-installer/src/answer.rs
diff --git a/proxmox-auto-installer/Cargo.toml
b/proxmox-auto-installer/Cargo.toml
index 67218dd..80de4fa 100644
--- a/proxmox-auto-installer/Cargo.toml
+++ b/proxmox-auto-installer/Cargo.toml
@@ -12,3 +12,4 @@ proxmox-installer-common = { path =
"../proxmox-installer-common" }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
toml = "0.7"
+enum-iterator = "0.6.0"
diff --git a/proxmox-auto-installer/src/answer.rs
b/proxmox-auto-installer/src/answer.rs
new file mode 100644
index 000..0add95e
--- /dev/null
+++ b/proxmox-auto-installer/src/answer.rs
@@ -0,0 +1,248 @@
+use proxmox_installer_common::{
+options::{BtrfsRaidLevel, FsType, ZfsChecksumOption, ZfsCompressOption,
ZfsRaidLevel},
+utils::{CidrAddress, Fqdn},
+};
+use serde::{Deserialize, Serialize};
+use std::{collections::BTreeMap, net::IpAddr};
+
+/// BTreeMap is used to store filters as the order of the filters will be
stable, compared to
+/// storing them in a HashMap
+
+#[derive(Clone, Deserialize, Debug)]
+#[serde(rename_all = "kebab-case")]
+pub struct Answer {
+pub global: Global,
+pub network: Network,
+#[serde(rename = "disk-setup")]
+pub disks: Disks,
+}
+
+#[derive(Clone, Deserialize, Debug)]
+pub struct Global {
+pub country: String,
+pub fqdn: Fqdn,
+pub keyboard: String,
+pub mailto: String,
+pub timezone: String,
+pub password: String,
+pub pre_commands: Option>,
+pub post_commands: Option>,
+#[serde(default)]
+pub reboot_on_error: bool,
+}
+
+#[derive(Clone, Deserialize, Debug)]
+struct NetworkInAnswer {
+#[serde(default)]
+pub use_dhcp: bool,
+pub cidr: Option,
+pub dns: Option,
+pub gateway: Option,
+pub filter: Option>,
+}
+
+#[derive(Clone, Deserialize, Debug)]
+#[serde(try_from = "NetworkInAnswer")]
+pub struct Network {
+pub network_settings: NetworkSettings,
+}
+
+impl TryFrom for Network {
+type Error = &'static str;
+
+fn try_from(source: NetworkInAnswer) -> Result {
+if !source.use_dhcp {
+if source.cidr.is_none() {
+return Err("Field 'cidr' must be set.");
+}
+if source.dns.is_none() {
+return Err("Field 'dns' must be set.");
+}
+if source.gateway.is_none() {
+return Err("Field 'gateway' must be set.");
+}
+if source.filter.is_none() {
+return Err("Field 'filter' must be set.");
+}
+
+Ok(Network {
+network_settings: NetworkSettings::Manual(NetworkManual {
+cidr: source.cidr.unwrap(),
+dns: source.dns.unwrap(),
+gateway: source.gateway.unwrap(),
+filter: source.filter.unwrap(),
+}),
+})
+} else {
+Ok(Network {
+network_settings: NetworkSettings::Dhcp(true),
+})
+}
+}
+}
+
+#[derive(Clone, Debug)]
+pub enum NetworkSettings {
+Dhcp(bool),
+Manual(NetworkManual),
+}
+
+#[derive(Clone, Debug)]
+pub struct NetworkManual {
+pub cidr: CidrAddress,
+pub dns: IpAddr,
+pub gateway: IpAddr,
+pub filter: BTreeMap,
+}
+
+#[derive(Clone, Debug, Deserialize)]
+pub struct DiskSetup {
+pub filesystem: Filesystem,
+#[serde(default)]
+pub disk_list: Vec,
+pub filter: Option>,
+pub filter_match: Option,
+pub zfs: Option,
+pub lvm: Option,
+pub btrfs: Option,
+}
+
+#[derive(Clone, Debug, Deserialize)]
+#[serde(try_from = "DiskSetup")]
+pub struct Disks {
+pub fs_type: FsType,
+pub disk_selection: DiskSelection,
+pub filter_match: Option,
+pub fs_options: FsOptions,
+}
+
+impl TryFrom for Disks {
+type Error = &'static str;
+
+fn try_from(source: DiskSetup) -> Result {
+if source.disk_list.is_empty() && source.filter.is_none() {
+return Err("Need either 'disk_list' or 'filter' set");
+}
+if !source.disk_list.is_empty() && source.filter.is_some() {
+return Err("Cannot use both, 'disk_list' and 'filter'");
+}
+
+let disk_selection = if !source.disk_list.is_empty() {
+DiskSelection::Selection(source.disk_list.clone())
+} else {
+DiskSelection::Filter(source.filter.clone().unwrap())
+};
+
+let lvm_checks = |source: &DiskSetup| -> Result<(), Self::Error> {
+if source.zfs.is_some() || source.btrfs.is_some() {
+return Err("make sure only 'lvm' options are set");
+}
+if source.disk_list
[pve-devel] [PATCH installer v6 27/36] low-level: write low level config to /tmp
This helps to know how the system was set up in steps after the
installation. For example in debug mode or when using post commands in
the automatic/unattended installation.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
proxmox-low-level-installer | 1 +
1 file changed, 1 insertion(+)
diff --git a/proxmox-low-level-installer b/proxmox-low-level-installer
index 54f689a..935bf17 100755
--- a/proxmox-low-level-installer
+++ b/proxmox-low-level-installer
@@ -69,6 +69,7 @@ sub read_and_merge_config {
Proxmox::Install::Config::merge($config);
log_info("got installation config: ".
to_json(Proxmox::Install::Config::get(), { utf8 => 1, canonical => 1 }) ."\n");
+file_write_all("/tmp/low-level-config.json",
to_json(Proxmox::Install::Config::get()));
}
sub send_reboot_ui_message {
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v6 25/36] control: update build depends for auto installer
Tested-by: Christoph Heiss Reviewed-by: Christoph Heiss Signed-off-by: Aaron Lauterer --- debian/control | 10 ++ 1 file changed, 10 insertions(+) diff --git a/debian/control b/debian/control index 3ca208b..1326400 100644 --- a/debian/control +++ b/debian/control @@ -8,10 +8,20 @@ Build-Depends: cargo:native, libgtk3-perl, libpve-common-perl, librsvg2-bin, + librust-anyhow-dev, + librust-clap-dev, librust-cursive+termion-backend-dev (>= 0.20.0), + librust-glob-dev, + librust-hex-dev, + librust-native-tls-dev, librust-regex-1+default-dev (>= 1.7~~), + librust-rustls-dev, + librust-rustls-native-certs-dev, librust-serde-1+default-dev, librust-serde-json-1+default-dev, + librust-sha2-dev, + librust-ureq-dev, + librust-toml-dev, libtest-mockmodule-perl, perl, rustc:native, -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v6 18/36] auto-installer: utils: make get_udev_index functions public
because we will need to access them directly in the future from a
separate binary
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/src/utils.rs | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/proxmox-auto-installer/src/utils.rs
b/proxmox-auto-installer/src/utils.rs
index 6101d66..ea645ad 100644
--- a/proxmox-auto-installer/src/utils.rs
+++ b/proxmox-auto-installer/src/utils.rs
@@ -47,7 +47,7 @@ pub fn get_network_settings(
Ok(network_options)
}
-fn get_single_udev_index(
+pub fn get_single_udev_index(
filter: BTreeMap,
udev_list: &BTreeMap>,
) -> Result {
@@ -72,7 +72,7 @@ fn get_single_udev_index(
Ok(dev_index.unwrap())
}
-fn get_matched_udev_indexes(
+pub fn get_matched_udev_indexes(
filter: BTreeMap,
udev_list: &BTreeMap>,
match_all: bool,
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v6 07/36] add auto-installer crate
Tested-by: Christoph Heiss Reviewed-by: Christoph Heiss Signed-off-by: Aaron Lauterer --- Cargo.toml| 1 + Makefile | 1 + proxmox-auto-installer/Cargo.toml | 10 ++ proxmox-auto-installer/src/lib.rs | 0 4 files changed, 12 insertions(+) create mode 100644 proxmox-auto-installer/Cargo.toml create mode 100644 proxmox-auto-installer/src/lib.rs diff --git a/Cargo.toml b/Cargo.toml index c1bd578..7017ac5 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,5 +1,6 @@ [workspace] members = [ +"proxmox-auto-installer", "proxmox-installer-common", "proxmox-tui-installer", ] diff --git a/Makefile b/Makefile index af33e90..4f140ca 100644 --- a/Makefile +++ b/Makefile @@ -47,6 +47,7 @@ $(BUILDDIR): interfaces \ proxinstall \ proxmox-low-level-installer \ + proxmox-auto-installer/ \ proxmox-tui-installer/ \ proxmox-installer-common/ \ test/ \ diff --git a/proxmox-auto-installer/Cargo.toml b/proxmox-auto-installer/Cargo.toml new file mode 100644 index 000..75cfb2c --- /dev/null +++ b/proxmox-auto-installer/Cargo.toml @@ -0,0 +1,10 @@ +[package] +name = "proxmox-auto-installer" +version = "0.1.0" +edition = "2021" +authors = [ "Aaron Lauterer " ] +license = "AGPL-3" +exclude = [ "build", "debian" ] +homepage = "https://www.proxmox.com"; + +[dependencies] diff --git a/proxmox-auto-installer/src/lib.rs b/proxmox-auto-installer/src/lib.rs new file mode 100644 index 000..e69de29 -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v6 08/36] auto-installer: add dependencies
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/Cargo.toml | 4
1 file changed, 4 insertions(+)
diff --git a/proxmox-auto-installer/Cargo.toml
b/proxmox-auto-installer/Cargo.toml
index 75cfb2c..67218dd 100644
--- a/proxmox-auto-installer/Cargo.toml
+++ b/proxmox-auto-installer/Cargo.toml
@@ -8,3 +8,7 @@ exclude = [ "build", "debian" ]
homepage = "https://www.proxmox.com";
[dependencies]
+proxmox-installer-common = { path = "../proxmox-installer-common" }
+serde = { version = "1.0", features = ["derive"] }
+serde_json = "1.0"
+toml = "0.7"
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v6 10/36] auto-installer: add struct to hold udev info
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/src/lib.rs | 1 +
proxmox-auto-installer/src/udevinfo.rs | 9 +
2 files changed, 10 insertions(+)
create mode 100644 proxmox-auto-installer/src/udevinfo.rs
diff --git a/proxmox-auto-installer/src/lib.rs
b/proxmox-auto-installer/src/lib.rs
index 7813b98..8cda416 100644
--- a/proxmox-auto-installer/src/lib.rs
+++ b/proxmox-auto-installer/src/lib.rs
@@ -1 +1,2 @@
pub mod answer;
+pub mod udevinfo;
diff --git a/proxmox-auto-installer/src/udevinfo.rs
b/proxmox-auto-installer/src/udevinfo.rs
new file mode 100644
index 000..a6b61b5
--- /dev/null
+++ b/proxmox-auto-installer/src/udevinfo.rs
@@ -0,0 +1,9 @@
+use serde::Deserialize;
+use std::collections::BTreeMap;
+
+#[derive(Clone, Deserialize, Debug)]
+pub struct UdevInfo {
+// use BTreeMap to have keys sorted
+pub disks: BTreeMap>,
+pub nics: BTreeMap>,
+}
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v6 01/36] tui: common: move InstallConfig struct to common crate
It describes the data structure expected by the low-level-installer.
We do this so we can use it in more than the TUI installer, for example
the planned auto installer.
Make the members public so we can easily implement a custom From method
for each dependent crate.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
proxmox-installer-common/src/setup.rs | 86 +++-
proxmox-tui-installer/src/setup.rs| 98 +--
.../src/views/install_progress.rs | 4 +-
3 files changed, 90 insertions(+), 98 deletions(-)
diff --git a/proxmox-installer-common/src/setup.rs
b/proxmox-installer-common/src/setup.rs
index 472e1f2..03beb77 100644
--- a/proxmox-installer-common/src/setup.rs
+++ b/proxmox-installer-common/src/setup.rs
@@ -12,7 +12,10 @@ use std::{
use serde::{de, Deserialize, Deserializer, Serialize, Serializer};
use crate::{
-options::{Disk, ZfsBootdiskOptions, ZfsChecksumOption, ZfsCompressOption},
+options::{
+BtrfsRaidLevel, Disk, FsType, ZfsBootdiskOptions, ZfsChecksumOption,
ZfsCompressOption,
+ZfsRaidLevel,
+},
utils::CidrAddress,
};
@@ -387,3 +390,84 @@ pub fn spawn_low_level_installer(test_mode: bool) ->
io::Result
.stdout(Stdio::piped())
.spawn()
}
+
+/// See Proxmox::Install::Config
+#[derive(Serialize)]
+pub struct InstallConfig {
+pub autoreboot: usize,
+
+#[serde(serialize_with = "serialize_fstype")]
+pub filesys: FsType,
+pub hdsize: f64,
+#[serde(skip_serializing_if = "Option::is_none")]
+pub swapsize: Option,
+#[serde(skip_serializing_if = "Option::is_none")]
+pub maxroot: Option,
+#[serde(skip_serializing_if = "Option::is_none")]
+pub minfree: Option,
+#[serde(skip_serializing_if = "Option::is_none")]
+pub maxvz: Option,
+
+#[serde(skip_serializing_if = "Option::is_none")]
+pub zfs_opts: Option,
+
+#[serde(
+serialize_with = "serialize_disk_opt",
+skip_serializing_if = "Option::is_none"
+)]
+pub target_hd: Option,
+#[serde(skip_serializing_if = "HashMap::is_empty")]
+pub disk_selection: HashMap,
+
+pub country: String,
+pub timezone: String,
+pub keymap: String,
+
+pub password: String,
+pub mailto: String,
+
+pub mngmt_nic: String,
+
+pub hostname: String,
+pub domain: String,
+#[serde(serialize_with = "serialize_as_display")]
+pub cidr: CidrAddress,
+pub gateway: IpAddr,
+pub dns: IpAddr,
+}
+
+fn serialize_disk_opt(value: &Option, serializer: S) -> Result
+where
+S: Serializer,
+{
+if let Some(disk) = value {
+serializer.serialize_str(&disk.path)
+} else {
+serializer.serialize_none()
+}
+}
+
+fn serialize_fstype(value: &FsType, serializer: S) -> Result
+where
+S: Serializer,
+{
+use FsType::*;
+let value = match value {
+// proxinstall::$fssetup
+Ext4 => "ext4",
+Xfs => "xfs",
+// proxinstall::get_zfs_raid_setup()
+Zfs(ZfsRaidLevel::Raid0) => "zfs (RAID0)",
+Zfs(ZfsRaidLevel::Raid1) => "zfs (RAID1)",
+Zfs(ZfsRaidLevel::Raid10) => "zfs (RAID10)",
+Zfs(ZfsRaidLevel::RaidZ) => "zfs (RAIDZ-1)",
+Zfs(ZfsRaidLevel::RaidZ2) => "zfs (RAIDZ-2)",
+Zfs(ZfsRaidLevel::RaidZ3) => "zfs (RAIDZ-3)",
+// proxinstall::get_btrfs_raid_setup()
+Btrfs(BtrfsRaidLevel::Raid0) => "btrfs (RAID0)",
+Btrfs(BtrfsRaidLevel::Raid1) => "btrfs (RAID1)",
+Btrfs(BtrfsRaidLevel::Raid10) => "btrfs (RAID10)",
+};
+
+serializer.collect_str(value)
+}
diff --git a/proxmox-tui-installer/src/setup.rs
b/proxmox-tui-installer/src/setup.rs
index 79421d7..e816c12 100644
--- a/proxmox-tui-installer/src/setup.rs
+++ b/proxmox-tui-installer/src/setup.rs
@@ -1,59 +1,11 @@
-use std::{collections::HashMap, fmt, net::IpAddr};
-
-use serde::{Serialize, Serializer};
+use std::collections::HashMap;
use crate::options::InstallerOptions;
use proxmox_installer_common::{
-options::{AdvancedBootdiskOptions, BtrfsRaidLevel, Disk, FsType,
ZfsRaidLevel},
-setup::InstallZfsOption,
-utils::CidrAddress,
+options::AdvancedBootdiskOptions,
+setup::InstallConfig,
};
-/// See Proxmox::Install::Config
-#[derive(Serialize)]
-pub struct InstallConfig {
-autoreboot: usize,
-
-#[serde(serialize_with = "serialize_fstype")]
-filesys: FsType,
-hdsize: f64,
-#[serde(skip_serializing_if = "Option::is_none")]
-swapsize: Option,
-#[serde(skip_serializing_if = "Option::is_none")]
-maxroot: Option,
-#[serde(skip_serializing_if = "Option::is_none")]
-minfree: Option,
-#[serde(skip_serializing_if = "Option::is_none")]
-maxvz: Option,
-
-#[serde(skip_serializing_if = "Option::is_none")]
-zfs_opts: Option,
-
-#[serde(
-serialize_with = "serialize_disk_opt",
-skip_serializing_if = "Option::is_none"
-)]
-target_hd:
[pve-devel] [PATCH installer v6 05/36] common: options: add Deserialize trait
For the Enums that will be used to deserialize an answer file.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
proxmox-installer-common/src/options.rs | 13 +
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/proxmox-installer-common/src/options.rs
b/proxmox-installer-common/src/options.rs
index 1aa8f65..1efac66 100644
--- a/proxmox-installer-common/src/options.rs
+++ b/proxmox-installer-common/src/options.rs
@@ -1,12 +1,14 @@
use std::net::{IpAddr, Ipv4Addr};
use std::{cmp, fmt};
+use serde::Deserialize;
use crate::setup::{
LocaleInfo, NetworkInfo, ProductConfig, ProxmoxProduct, RuntimeInfo,
SetupInfo,
};
use crate::utils::{CidrAddress, Fqdn};
-#[derive(Copy, Clone, Debug, Eq, PartialEq)]
+#[derive(Copy, Clone, Debug, Deserialize, Eq, PartialEq)]
+#[serde(rename_all = "lowercase")]
pub enum BtrfsRaidLevel {
Raid0,
Raid1,
@@ -24,7 +26,8 @@ impl fmt::Display for BtrfsRaidLevel {
}
}
-#[derive(Copy, Clone, Debug, Eq, PartialEq)]
+#[derive(Copy, Clone, Debug, Deserialize, Eq, PartialEq)]
+#[serde(rename_all = "lowercase")]
pub enum ZfsRaidLevel {
Raid0,
Raid1,
@@ -112,7 +115,8 @@ impl BtrfsBootdiskOptions {
}
}
-#[derive(Copy, Clone, Debug, Default, Eq, PartialEq)]
+#[derive(Copy, Clone, Debug, Default, Deserialize, Eq, PartialEq)]
+#[serde(rename_all(deserialize = "lowercase"))]
pub enum ZfsCompressOption {
#[default]
On,
@@ -141,7 +145,8 @@ pub const ZFS_COMPRESS_OPTIONS: &[ZfsCompressOption] = {
&[On, Off, Lzjb, Lz4, Zle, Gzip, Zstd]
};
-#[derive(Copy, Clone, Debug, Default, Eq, PartialEq)]
+#[derive(Copy, Clone, Debug, Default, Deserialize, Eq, PartialEq)]
+#[serde(rename_all = "kebab-case")]
pub enum ZfsChecksumOption {
#[default]
On,
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH installer v6 24/36] auto-installer: fetch: add http plugin to fetch answer
This plugin will send a HTTP POST request with identifying sysinfo to
fetch an answer file. The provided sysinfo can be used to identify the
system and generate a matching answer file on demand.
The URL to send the request to, can be defined in two ways. Via a custom
DHCP option or a TXT record on a predefined subdomain, relative to the
search domain received via DHCP.
Additionally it is possible to specify a SHA256 SSL fingerprint. This
can be useful if a self-signed certificate is used or the URL is using
an IP address instead of an FQDN. Even with a trusted cert, it can be
used to pin this specific certificate.
The certificate fingerprint can either be placed on the `proxmoxinst`
partition and needs to be called `cert_fingerprint.txt`, or it can be
provided in a second custom DHCP option or a TXT record.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
.../src/bin/proxmox-fetch-answer.rs | 11 +-
.../src/fetch_plugins/http.rs | 190 ++
.../src/fetch_plugins/mod.rs | 1 +
unconfigured.sh | 9 +
4 files changed, 208 insertions(+), 3 deletions(-)
create mode 100644 proxmox-auto-installer/src/fetch_plugins/http.rs
diff --git a/proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
b/proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
index a3681a2..6d42df2 100644
--- a/proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
+++ b/proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
@@ -1,6 +1,9 @@
use anyhow::{anyhow, Error, Result};
use log::{error, info, LevelFilter};
-use proxmox_auto_installer::{fetch_plugins::partition::FetchFromPartition,
log::AutoInstLogger};
+use proxmox_auto_installer::{
+fetch_plugins::{http::FetchFromHTTP, partition::FetchFromPartition},
+log::AutoInstLogger,
+};
use std::io::Write;
use std::process::{Command, ExitCode, Stdio};
@@ -18,8 +21,10 @@ fn fetch_answer() -> Result {
Ok(answer) => return Ok(answer),
Err(err) => info!("Fetching answer file from partition failed: {err}"),
}
-// TODO: add more options to get an answer file, e.g. download from url
where url could be
-// fetched via txt records on predefined subdomain, kernel param, dhcp
option, ...
+match FetchFromHTTP::get_answer() {
+Ok(answer) => return Ok(answer),
+Err(err) => info!("Fetching answer file via HTTP failed: {err}"),
+}
Err(Error::msg("Could not find any answer file!"))
}
diff --git a/proxmox-auto-installer/src/fetch_plugins/http.rs
b/proxmox-auto-installer/src/fetch_plugins/http.rs
new file mode 100644
index 000..4ac9afb
--- /dev/null
+++ b/proxmox-auto-installer/src/fetch_plugins/http.rs
@@ -0,0 +1,190 @@
+use anyhow::{bail, Error, Result};
+use log::info;
+use std::{
+fs::{self, read_to_string},
+path::Path,
+process::Command,
+};
+
+use crate::fetch_plugins::utils::{post, sysinfo};
+
+use super::utils;
+
+static CERT_FINGERPRINT_FILE: &str = "cert_fingerprint.txt";
+static ANSWER_SUBDOMAIN: &str = "proxmoxinst";
+static ANSWER_SUBDOMAIN_FP: &str = "proxmoxinst-fp";
+
+// It is possible to set custom DHPC options. Option numbers 224 to 254 [0].
+// To use them with dhclient, we need to configure it to request them and what
they should be
+// called.
+//
+// e.g. /etc/dhcp/dhclient.conf:
+// ```
+// option proxmoxinst-url code 250 = text;
+// option proxmoxinst-fp code 251 = text;
+// also request proxmoxinst-url, proxmoxinst-fp;
+// ```
+//
+// The results will end up in the /var/lib/dhcp/dhclient.leases file from
where we can fetch them
+//
+// [0]
https://www.iana.org/assignments/bootp-dhcp-parameters/bootp-dhcp-parameters.xhtml
+static DHCP_URL_OPTION: &str = "proxmoxinst-url";
+static DHCP_FP_OPTION: &str = "proxmoxinst-fp";
+static DHCP_LEASE_FILE: &str = "/var/lib/dhcp/dhclient.leases";
+
+pub struct FetchFromHTTP;
+
+impl FetchFromHTTP {
+/// Will try to fetch the answer.toml by sending a HTTP POST request. The
URL can be configured
+/// either via DHCP or DNS.
+/// DHCP options are checked first. The SSL certificate need to be either
trusted by the root
+/// certs or a SHA256 fingerprint needs to be provided. The SHA256 SSL
fingerprint can either
+/// be placed in a `cert_fingerprint.txt` file in the `proxmoxinst`
partition, as DHCP option,
+/// or as DNS TXT record. If provided, the `cert_fingerprint.txt` file has
preference.
+pub fn get_answer() -> Result {
+info!("Checking for certificate fingerprint in file.");
+let mut fingerprint: Option = match
Self::get_cert_fingerprint_from_file() {
+Ok(fp) => Some(fp),
+Err(err) => {
+info!("{err}");
+None
+}
+};
+
+let answer_url: String;
+
+(answer_url, fingerprint) = match
Self::fetch_dhcp(fingerprint.clone()) {
+Ok((url, fp)) => (url, fp),
+
[pve-devel] [PATCH installer v6 19/36] auto-installer: add proxmox-autoinst-helper tool
It can parse an answer file to check against syntax errors, test match
filters against the current hardware and list properties of the current
hardware to match against.
Since this tool should be able to run outside of the installer
environment, it does not rely on the device information provided by the
low-level installer. It instead fetches the list of disks and NICs by
itself.
The rules when a device is ignored, should match how the low-level
installer handles it.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
Makefile | 3 +-
proxmox-auto-installer/Cargo.toml | 2 +
proxmox-auto-installer/src/answer.rs | 3 +-
.../src/bin/proxmox-autoinst-helper.rs| 340 ++
4 files changed, 346 insertions(+), 2 deletions(-)
create mode 100644 proxmox-auto-installer/src/bin/proxmox-autoinst-helper.rs
diff --git a/Makefile b/Makefile
index e44450e..197a351 100644
--- a/Makefile
+++ b/Makefile
@@ -20,6 +20,7 @@ PREFIX = /usr
BINDIR = $(PREFIX)/bin
USR_BIN := \
proxmox-tui-installer\
+ proxmox-autoinst-helper\
proxmox-fetch-answer\
proxmox-auto-installer
@@ -123,7 +124,7 @@ $(COMPILED_BINS): cargo-build
cargo-build:
$(CARGO) build --package proxmox-tui-installer --bin
proxmox-tui-installer \
--package proxmox-auto-installer --bin proxmox-auto-installer \
- --bin proxmox-fetch-answer $(CARGO_BUILD_ARGS)
+ --bin proxmox-fetch-answer --bin proxmox-autoinst-helper
$(CARGO_BUILD_ARGS)
%-banner.png: %-banner.svg
rsvg-convert -o $@ $<
diff --git a/proxmox-auto-installer/Cargo.toml
b/proxmox-auto-installer/Cargo.toml
index 741794a..bb0b49c 100644
--- a/proxmox-auto-installer/Cargo.toml
+++ b/proxmox-auto-installer/Cargo.toml
@@ -9,6 +9,7 @@ homepage = "https://www.proxmox.com";
[dependencies]
anyhow = "1.0"
+clap = { version = "4.0", features = ["derive"] }
glob = "0.3"
proxmox-installer-common = { path = "../proxmox-installer-common" }
serde = { version = "1.0", features = ["derive"] }
@@ -16,3 +17,4 @@ serde_json = "1.0"
toml = "0.7"
enum-iterator = "0.6.0"
log = "0.4.20"
+regex = "1.7"
diff --git a/proxmox-auto-installer/src/answer.rs
b/proxmox-auto-installer/src/answer.rs
index 0add95e..94cebb3 100644
--- a/proxmox-auto-installer/src/answer.rs
+++ b/proxmox-auto-installer/src/answer.rs
@@ -1,3 +1,4 @@
+use clap::ValueEnum;
use proxmox_installer_common::{
options::{BtrfsRaidLevel, FsType, ZfsChecksumOption, ZfsCompressOption,
ZfsRaidLevel},
utils::{CidrAddress, Fqdn},
@@ -205,7 +206,7 @@ pub enum DiskSelection {
Selection(Vec),
Filter(BTreeMap),
}
-#[derive(Clone, Deserialize, Debug, PartialEq)]
+#[derive(Clone, Deserialize, Debug, PartialEq, ValueEnum)]
#[serde(rename_all = "lowercase")]
pub enum FilterMatch {
Any,
diff --git a/proxmox-auto-installer/src/bin/proxmox-autoinst-helper.rs
b/proxmox-auto-installer/src/bin/proxmox-autoinst-helper.rs
new file mode 100644
index 000..058d5ff
--- /dev/null
+++ b/proxmox-auto-installer/src/bin/proxmox-autoinst-helper.rs
@@ -0,0 +1,340 @@
+use anyhow::{bail, Result};
+use clap::{Args, Parser, Subcommand, ValueEnum};
+use glob::Pattern;
+use regex::Regex;
+use serde::{Deserialize, Serialize};
+use std::{collections::BTreeMap, fs, io::Read, path::PathBuf,
process::Command};
+
+use proxmox_auto_installer::{
+answer::Answer,
+answer::FilterMatch,
+utils::{get_matched_udev_indexes, get_single_udev_index},
+};
+
+/// This tool validates the format of an answer file. Additionally it can test
match filters and
+/// print information on the properties to match against for the current
hardware.
+#[derive(Parser, Debug)]
+#[command(author, version, about, long_about = None)]
+struct Cli {
+#[command(subcommand)]
+command: Commands,
+}
+
+#[derive(Subcommand, Debug)]
+enum Commands {
+ValidateAnswer(CommandValidateAnswer),
+Match(CommandMatch),
+Info(CommandInfo),
+}
+
+/// Show device information that can be used for filters
+#[derive(Args, Debug)]
+struct CommandInfo {
+/// For which device type information should be shown
+#[arg(name="type", short, long, value_enum,
default_value_t=AllDeviceTypes::All)]
+device: AllDeviceTypes,
+}
+
+/// Test which devices the given filter matches against
+///
+/// Filters support the following syntax:
+/// ? Match a single character
+/// * Match any number of characters
+/// [a], [0-9] Specifc character or range of characters
+/// [!a] Negate a specific character of range
+///
+/// To avoid globbing characters being interpreted by the shell, use single
quotes.
+/// Multiple filters can be defined.
+///
+/// Examples:
+/// Match disks against the serial number and device name, both must match:
+///
+/// proxmox-autoinst-helper match --filter-match all disk
'ID_SERIAL_SHORT=**' 'DEVNAME=*nvme*'
+
[pve-devel] [PATCH installer v6 15/36] auto-installer: add fetch answer binary
it is supposed to be run first and fetch an answer file.
The initial implementation searches for a partition/filesystem called
'proxmoxinst' or 'PROXMOXINST' with an 'answer.toml' file in the root
directory.
Once it has an answer file, it will call the 'proxmox-auto-installer'
and pipe in the contents via stdin.
Tested-by: Christoph Heiss
Reviewed-by: Christoph Heiss
Signed-off-by: Aaron Lauterer
---
Makefile | 4 +-
.../src/bin/proxmox-fetch-answer.rs | 71
.../src/fetch_plugins/mod.rs | 2 +
.../src/fetch_plugins/partition.rs| 32
.../src/fetch_plugins/utils.rs| 81 +++
proxmox-auto-installer/src/lib.rs | 1 +
6 files changed, 190 insertions(+), 1 deletion(-)
create mode 100644 proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
create mode 100644 proxmox-auto-installer/src/fetch_plugins/mod.rs
create mode 100644 proxmox-auto-installer/src/fetch_plugins/partition.rs
create mode 100644 proxmox-auto-installer/src/fetch_plugins/utils.rs
diff --git a/Makefile b/Makefile
index 3ac5769..e44450e 100644
--- a/Makefile
+++ b/Makefile
@@ -20,6 +20,7 @@ PREFIX = /usr
BINDIR = $(PREFIX)/bin
USR_BIN := \
proxmox-tui-installer\
+ proxmox-fetch-answer\
proxmox-auto-installer
COMPILED_BINS := \
@@ -121,7 +122,8 @@ $(COMPILED_BINS): cargo-build
.PHONY: cargo-build
cargo-build:
$(CARGO) build --package proxmox-tui-installer --bin
proxmox-tui-installer \
- --package proxmox-auto-installer --bin proxmox-auto-installer
$(CARGO_BUILD_ARGS)
+ --package proxmox-auto-installer --bin proxmox-auto-installer \
+ --bin proxmox-fetch-answer $(CARGO_BUILD_ARGS)
%-banner.png: %-banner.svg
rsvg-convert -o $@ $<
diff --git a/proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
b/proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
new file mode 100644
index 000..a3681a2
--- /dev/null
+++ b/proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
@@ -0,0 +1,71 @@
+use anyhow::{anyhow, Error, Result};
+use log::{error, info, LevelFilter};
+use proxmox_auto_installer::{fetch_plugins::partition::FetchFromPartition,
log::AutoInstLogger};
+use std::io::Write;
+use std::process::{Command, ExitCode, Stdio};
+
+static LOGGER: AutoInstLogger = AutoInstLogger;
+
+pub fn init_log() -> Result<()> {
+AutoInstLogger::init("/tmp/fetch_answer.log")?;
+log::set_logger(&LOGGER)
+.map(|()| log::set_max_level(LevelFilter::Info))
+.map_err(|err| anyhow!(err))
+}
+
+fn fetch_answer() -> Result {
+match FetchFromPartition::get_answer() {
+Ok(answer) => return Ok(answer),
+Err(err) => info!("Fetching answer file from partition failed: {err}"),
+}
+// TODO: add more options to get an answer file, e.g. download from url
where url could be
+// fetched via txt records on predefined subdomain, kernel param, dhcp
option, ...
+
+Err(Error::msg("Could not find any answer file!"))
+}
+
+fn main() -> ExitCode {
+if let Err(err) = init_log() {
+panic!("could not initialize logging: {err}");
+}
+
+info!("Fetching answer file");
+let answer = match fetch_answer() {
+Ok(answer) => answer,
+Err(err) => {
+error!("Aborting: {}", err);
+return ExitCode::FAILURE;
+}
+};
+
+let mut child = match Command::new("proxmox-auto-installer")
+.stdout(Stdio::inherit())
+.stdin(Stdio::piped())
+.stderr(Stdio::null())
+.spawn()
+{
+Ok(child) => child,
+Err(err) => panic!("Failed to start automatic installation: {err}"),
+};
+
+let mut stdin = child.stdin.take().expect("Failed to open stdin");
+std::thread::spawn(move || {
+stdin
+.write_all(answer.as_bytes())
+.expect("Failed to write to stdin");
+});
+
+match child.wait() {
+Ok(status) => {
+if status.success() {
+ExitCode::SUCCESS
+} else {
+ExitCode::FAILURE // Will be trapped
+}
+}
+Err(err) => {
+error!("Auto installer exited: {err}");
+ExitCode::FAILURE
+}
+}
+}
diff --git a/proxmox-auto-installer/src/fetch_plugins/mod.rs
b/proxmox-auto-installer/src/fetch_plugins/mod.rs
new file mode 100644
index 000..11d6937
--- /dev/null
+++ b/proxmox-auto-installer/src/fetch_plugins/mod.rs
@@ -0,0 +1,2 @@
+pub mod partition;
+mod utils;
diff --git a/proxmox-auto-installer/src/fetch_plugins/partition.rs
b/proxmox-auto-installer/src/fetch_plugins/partition.rs
new file mode 100644
index 000..0c47a62
--- /dev/null
+++ b/proxmox-auto-installer/src/fetch_plugins/partition.rs
@@ -0,0 +1,32 @@
+use anyhow::{Error, Result};
+use std::{fs::read_to_string, path::Path};
+use log::info;
+
+use crate::fetch_plugins::u
[pve-devel] [PATCH installer v6 36/36] autoinst-helper: add prepare-iso subcommand
This new subcommand makes it possible to prepare an ISO to use it for an
automated installation.
It is possible to control the behavior of the resulting automated ISO
with optional parameters.
If no target file is specified, the new ISO will be named with suffixes
to indicate it as automated and additional information. This should help
to distinct between the different options that were chosen to create it.
The code for parsing an answer file is moved to its own function.
Signed-off-by: Aaron Lauterer
---
proxmox-autoinst-helper/Cargo.toml | 1 +
proxmox-autoinst-helper/src/main.rs | 269 +---
2 files changed, 248 insertions(+), 22 deletions(-)
diff --git a/proxmox-autoinst-helper/Cargo.toml
b/proxmox-autoinst-helper/Cargo.toml
index 2a88c0f..75399e0 100644
--- a/proxmox-autoinst-helper/Cargo.toml
+++ b/proxmox-autoinst-helper/Cargo.toml
@@ -19,3 +19,4 @@ serde_json = "1.0"
toml = "0.7"
log = "0.4.20"
regex = "1.7"
+which = "4.2.5"
diff --git a/proxmox-autoinst-helper/src/main.rs
b/proxmox-autoinst-helper/src/main.rs
index fe1cbec..81bb54d 100644
--- a/proxmox-autoinst-helper/src/main.rs
+++ b/proxmox-autoinst-helper/src/main.rs
@@ -3,16 +3,29 @@ use clap::{Args, Parser, Subcommand, ValueEnum};
use glob::Pattern;
use regex::Regex;
use serde::Serialize;
-use std::{collections::BTreeMap, fs, io::Read, path::PathBuf,
process::Command};
+use std::{
+collections::BTreeMap,
+fs,
+io::Read,
+path::{Path, PathBuf},
+process::{Command, Stdio},
+};
+use which::which;
use proxmox_auto_installer::{
answer::Answer,
answer::FilterMatch,
sysinfo,
-utils::{get_matched_udev_indexes, get_nic_list, get_single_udev_index},
+utils::{
+get_matched_udev_indexes, get_nic_list, get_single_udev_index,
AutoInstModes,
+AutoInstSettings,
+},
};
-/// This tool validates the format of an answer file. Additionally it can test
match filters and
+static PROXMOX_ISO_FLAG: &str = "/autoinst-capable";
+
+/// This tool can be used to prepare a Proxmox installation ISO for automated
installations.
+/// Additional uses are to validate the format of an answer file or to test
match filters and
/// print information on the properties to match against for the current
hardware.
#[derive(Parser, Debug)]
#[command(author, version, about, long_about = None)]
@@ -23,6 +36,7 @@ struct Cli {
#[derive(Subcommand, Debug)]
enum Commands {
+PrepareIso(CommandPrepareISO),
ValidateAnswer(CommandValidateAnswer),
DeviceMatch(CommandDeviceMatch),
DeviceInfo(CommandDeviceInfo),
@@ -76,6 +90,62 @@ struct CommandValidateAnswer {
debug: bool,
}
+/// Prepare an ISO for automated installation.
+///
+/// The final ISO will try to fetch an answer file automatically. It will
first search for a
+/// partition / file-system called "PROXMOXINST" (or lowercase) and a file in
the root named
+/// "answer.toml".
+///
+/// If that is not found, it will try to fetch an answer file via an HTTP Post
request. The URL for
+/// it can be defined for the ISO with the '--url', '-u' argument. If not
present, it will try to
+/// get a URL from a DHCP option (250, TXT) or as a DNS TXT record at
'proxmoxinst.{search
+/// domain}'.
+///
+/// The TLS certificate fingerprint can either be defined via the
'--cert-fingerprint', '-c'
+/// argument or alternatively via the custom DHCP option (251, TXT) or in a
DNS TXT record located
+/// at 'proxmoxinst-fp.{search domain}'.
+///
+/// The latter options to provide the TLS fingerprint will only be used if the
same method was used
+/// to retrieve the URL. For example, the DNS TXT record for the fingerprint
will only be used, if
+/// no one was configured with the '--cert-fingerprint' parameter and if the
URL was retrieved via
+/// the DNS TXT record.
+///
+/// The behavior of how to fetch an answer file can be overridden with the
'--install-mode', '-i'
+/// parameter. The answer file can be{n}
+/// * integrated into the ISO itself ('included'){n}
+/// * needs to be present in a partition / file-system called 'PROXMOXINST'
('partition'){n}
+/// * only be requested via an HTTP Post request ('http').
+#[derive(Args, Debug)]
+struct CommandPrepareISO {
+/// Path to the source ISO
+source: PathBuf,
+
+/// Path to store the final ISO to.
+#[arg(short, long)]
+target: Option,
+
+/// Where to fetch the answer file from.
+#[arg(short, long, value_enum, default_value_t=AutoInstModes::Auto)]
+install_mode: AutoInstModes,
+
+/// Include the specified answer file in the ISO. Requires the
'--install-mode', '-i' parameter
+/// to be set to 'included'.
+#[arg(short, long)]
+answer_file: Option,
+
+/// Specify URL for fetching the answer file via HTTP
+#[arg(short, long)]
+url: Option,
+
+/// Pin the ISO to the specified SHA256 TLS certificate fingerprint.
+#[arg(short, long)]
+cert_fingerprint: Option,
+
+/// Tmp directory to use.
+
[pve-devel] [PATCH installer v6 34/36] fetch-answer: use ISO specified configurations
This patch switches the behavior to use the settings that can be
specified in the ISO.
This means, that it is possible to control how the answer file should be
fetched:
* auto - as usually, go through the options until one works (partition,
http)
* included - the answer file is included in the ISO
* partition - only check for an answer file in a partition called
'proxmoxinst' in lower or uppercase
* http - only fetch the answer file via an HTTP POST request.
Additionally it is possible to specify the HTTP URL directly in the ISO.
Placing the SSL fingerprint on a partition is not possible anymore. If
one wants to provide it right away (besides DHCP or DNS), it must be
incluced in the ISO itself. This reduced the need for another USB flash
drive.
Signed-off-by: Aaron Lauterer
---
proxmox-fetch-answer/Cargo.toml | 1 +
.../src/fetch_plugins/http.rs | 65 +++
proxmox-fetch-answer/src/main.rs | 64 ++
3 files changed, 77 insertions(+), 53 deletions(-)
diff --git a/proxmox-fetch-answer/Cargo.toml b/proxmox-fetch-answer/Cargo.toml
index fbcca46..797c185 100644
--- a/proxmox-fetch-answer/Cargo.toml
+++ b/proxmox-fetch-answer/Cargo.toml
@@ -17,6 +17,7 @@ log = "0.4.20"
ureq = { version = "2.6", features = [ "native-certs", "native-tls" ] }
rustls = { version = "0.20", features = [ "dangerous_configuration" ] }
rustls-native-certs = "0.6"
+toml = "0.7"
native-tls = "0.2"
sha2 = "0.10"
hex = "0.4"
diff --git a/proxmox-fetch-answer/src/fetch_plugins/http.rs
b/proxmox-fetch-answer/src/fetch_plugins/http.rs
index 5772c42..4093131 100644
--- a/proxmox-fetch-answer/src/fetch_plugins/http.rs
+++ b/proxmox-fetch-answer/src/fetch_plugins/http.rs
@@ -2,16 +2,12 @@ use anyhow::{bail, Error, Result};
use log::info;
use std::{
fs::{self, read_to_string},
-path::Path,
process::Command,
};
use crate::fetch_plugins::utils::post;
-use proxmox_auto_installer::sysinfo;
+use proxmox_auto_installer::{sysinfo, utils::AutoInstSettings};
-use super::utils;
-
-static CERT_FINGERPRINT_FILE: &str = "cert_fingerprint.txt";
static ANSWER_SUBDOMAIN: &str = "proxmoxinst";
static ANSWER_SUBDOMAIN_FP: &str = "proxmoxinst-fp";
@@ -37,30 +33,33 @@ pub struct FetchFromHTTP;
impl FetchFromHTTP {
/// Will try to fetch the answer.toml by sending a HTTP POST request. The
URL can be configured
-/// either via DHCP or DNS.
-/// DHCP options are checked first. The SSL certificate need to be either
trusted by the root
-/// certs or a SHA256 fingerprint needs to be provided. The SHA256 SSL
fingerprint can either
-/// be placed in a `cert_fingerprint.txt` file in the `proxmoxinst`
partition, as DHCP option,
-/// or as DNS TXT record. If provided, the `cert_fingerprint.txt` file has
preference.
-pub fn get_answer() -> Result {
-info!("Checking for certificate fingerprint in file.");
-let mut fingerprint: Option = match
Self::get_cert_fingerprint_from_file() {
-Ok(fp) => Some(fp),
-Err(err) => {
-info!("{err}");
-None
+/// either via DHCP or DNS or preconfigured in the ISO.
+/// If the URL is not defined in the ISO, it will first check DHCP
options. The SSL certificate
+/// needs to be either trusted by the root certs or a SHA256 fingerprint
needs to be provided.
+/// The SHA256 SSL fingerprint can either be defined in the ISO, as DHCP
option, or as DNS TXT
+/// record. If provided, the fingerprint provided in the ISO has
preference.
+pub fn get_answer(settings: &AutoInstSettings) -> Result {
+let mut fingerprint: Option = match
settings.cert_fingerprint.clone() {
+Some(fp) => {
+info!("SSL fingerprint provided through ISO.");
+Some(fp)
}
+None => None,
};
let answer_url: String;
-
-(answer_url, fingerprint) = match
Self::fetch_dhcp(fingerprint.clone()) {
-Ok((url, fp)) => (url, fp),
-Err(err) => {
-info!("{err}");
-Self::fetch_dns(fingerprint.clone())?
-}
-};
+if let Some(url) = settings.http_url.clone() {
+info!("URL specified in ISO");
+answer_url = url;
+} else {
+(answer_url, fingerprint) = match
Self::fetch_dhcp(fingerprint.clone()) {
+Ok((url, fp)) => (url, fp),
+Err(err) => {
+info!("{err}");
+Self::fetch_dns(fingerprint.clone())?
+}
+};
+}
if fingerprint.is_some() {
let fp = fingerprint.clone();
@@ -74,22 +73,6 @@ impl FetchFromHTTP {
Ok(answer)
}
-/// Reads certificate fingerprint from file
-pub fn get_cert_fingerprint_from_file() -> Result {
-let mount_path = utils::mount_proxmoxinst_part()?;
-let
