Re: [pve-devel] [PATCH manager] report: filter comments in VM/CT configs
Am 28/12/2022 um 15:18 schrieb Stefan Sterz: > maybe somewhat off-topic for the patch at hand, but it might be nice to > modularize the pve report. so that `pvereport` gives you a default set > of information, but you could also use `pvereport ha` to give you more > information specifically about the state of the ha manager or `pvereport > ceph` for information about ceph etc. > > maybe paired with a verbose flag so that you could request more detailed > info. e.g. `pvereport zfs` gives the zfs information currently in the > report, but `pvereport zfs -v` could also include `arc_summary` and `cat > /sys/module/zfs/parameters/zfs_arc_max` and other less often needed but > sometimes useful information. > > this might be handy especially in cases where you need information that > spans several files/commands that aren't always needed. it might make > the `pvereport` more useful in the forum too, where we currently can't > use it at all because it discloses too much information. > I'd rather keep it simple and not blow it up too much with a massive amount of switches, lots of topics are very intertwined, e.g., ha issues often need most information to have a full picture, i.e., it never hurts to have more information. Also note that the forum is the community one and so IMO the wrong place for any generic enterprise support report request, publicly readable for all compared to the subscription contract and private enterprise support portal. If, a generic (public) IP hashing might be sensible, and if that's done such that it's still relatively easy to trace the different used one (i.e., a more human readable/comparable hash than md5/SHA/...) it can be just always turned on too. ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager] report: filter comments in VM/CT configs
On 12/16/22 13:15, Fiona Ebner wrote:
> Am 15.12.22 um 17:57 schrieb Mira Limbeck:
>> diff --git a/PVE/Report.pm b/PVE/Report.pm
>> index 90b7cb1c..7ebe98f7 100644
>> --- a/PVE/Report.pm
>> +++ b/PVE/Report.pm
>> @@ -5,16 +5,34 @@ use warnings;
>>
>> use PVE::Tools;
>>
>> +my sub file2text {
>> +my ($file, $filter) = @_;
>> +my $text = "\n# cat $file\n";
>> +
>> +my $contents = PVE::Tools::file_get_contents($file);
>> +if ($filter) {
>> +foreach my $line (split('\n', $contents)) {
>> +next if $line =~ m/^\s*#/;
>> +next if $line =~ m/^cipassword/;
>> +
> Should we rather mask the value instead of dropping the whole line, so
> that we can see that /some/ cipassword is set? Might be more relevant
> for future filtered options.
>
>
> ___
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
maybe somewhat off-topic for the patch at hand, but it might be nice to
modularize the pve report. so that `pvereport` gives you a default set
of information, but you could also use `pvereport ha` to give you more
information specifically about the state of the ha manager or `pvereport
ceph` for information about ceph etc.
maybe paired with a verbose flag so that you could request more detailed
info. e.g. `pvereport zfs` gives the zfs information currently in the
report, but `pvereport zfs -v` could also include `arc_summary` and `cat
/sys/module/zfs/parameters/zfs_arc_max` and other less often needed but
sometimes useful information.
this might be handy especially in cases where you need information that
spans several files/commands that aren't always needed. it might make
the `pvereport` more useful in the forum too, where we currently can't
use it at all because it discloses too much information.
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager] report: filter comments in VM/CT configs
Am 15.12.22 um 17:57 schrieb Mira Limbeck:
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 90b7cb1c..7ebe98f7 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -5,16 +5,34 @@ use warnings;
>
> use PVE::Tools;
>
> +my sub file2text {
> +my ($file, $filter) = @_;
> +my $text = "\n# cat $file\n";
> +
> +my $contents = PVE::Tools::file_get_contents($file);
> +if ($filter) {
> + foreach my $line (split('\n', $contents)) {
> + next if $line =~ m/^\s*#/;
> + next if $line =~ m/^cipassword/;
> +
Should we rather mask the value instead of dropping the whole line, so
that we can see that /some/ cipassword is set? Might be more relevant
for future filtered options.
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager] report: filter comments in VM/CT configs
On 12/16/22 11:31, Thomas Lamprecht wrote:
On 15/12/2022 17:57, Mira Limbeck wrote:
Since some users keep their passwords in the VM/CT configs as comments
and those are most of the time unnecessary when looking through the
report, filter those.
I'd rather have this optional, opt-out is fine, where the CLI gets a switch
and the web interface gets a dialogue with a checkbox for filtering comments
in guest configs. A lot of sane admins don't sprinkle plain text PWs into
www-data readable plain text configs but possible relevant setup info.
Yes, I'd say most of our users that open tickets in the enterprise
support portal don't do that.
I'll rework it to make it optional then.
If you want to get this faster in it's fine to focus on CLI only for now,
but I think some more options might be relevant for the report in the near
term, be it filtering other stuff (e.g., public IP addresses) or also
adding extra info (e.g., (parts of) journal/syslogs), so laying out a bit
foundations for that now could be reused for that; but as said, no need
to do all that now.
Also please separate adding and using file2text and adding the filter part
into two commits, those are different things.
Sure, will do.
In addition to the comments, also filter the `cipassword` option
since it contains the hash of the password.
To facilitate the filtering, a new sub 'file2text' is introduced that
can filter the file contents if required.
This sub replaces the 'cat ...' commands.
Signed-off-by: Mira Limbeck
---
I did not add print to STDERR in file2text for now since it got quite
chatty.
If this is wanted, I'll send a v2 adding it. But since file2text is also
called by dir2text the 'OK' at the end won't always align, especially
when dir2text is used.
PVE/Report.pm | 48 +---
1 file changed, 33 insertions(+), 15 deletions(-)
diff --git a/PVE/Report.pm b/PVE/Report.pm
index 90b7cb1c..7ebe98f7 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -5,16 +5,34 @@ use warnings;
use PVE::Tools;
+my sub file2text {
+my ($file, $filter) = @_;
+my $text = "\n# cat $file\n";
I agree with the sentiment of Stefan's comment but comment something else
that we do is IMO not ideal, tends to get out of date soon and might
be hard to output a grep invocation that behaves really 1:1 as the code here
does, or will do sometimes.
So maybe just output something like:
my $text = "\n# ". ($filter ? 'filtered' : '') ." cat $file\n";
I wasn't sure what to do here, and dir2text already used
file_get_contents instead of cat, so the comment wasn't exactly true in
the first place for some of the files. Although none of those were filtered.
+
+my $contents = PVE::Tools::file_get_contents($file);
+if ($filter) {
+ foreach my $line (split('\n', $contents)) {
+ next if $line =~ m/^\s*#/;
+ next if $line =~ m/^cipassword/;
not sure if it's a good idea to just plainly filter, I mean only guest configs
are setting $filter anyway, but IMO it's still a bid odd coupling; maybe switch
to a $code-ref and calling that, with a filter_guest_config sub passed then on
the call sites?
I didn't want to make it more complex than necessary, since asking
colleagues what else should be filtered didn't result in any other
suggestions than the VM/CT configs (for now). But I'll rework it to make
it more flexible.
Thanks!
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager] report: filter comments in VM/CT configs
On 15/12/2022 17:57, Mira Limbeck wrote:
> Since some users keep their passwords in the VM/CT configs as comments
> and those are most of the time unnecessary when looking through the
> report, filter those.
I'd rather have this optional, opt-out is fine, where the CLI gets a switch
and the web interface gets a dialogue with a checkbox for filtering comments
in guest configs. A lot of sane admins don't sprinkle plain text PWs into
www-data readable plain text configs but possible relevant setup info.
If you want to get this faster in it's fine to focus on CLI only for now,
but I think some more options might be relevant for the report in the near
term, be it filtering other stuff (e.g., public IP addresses) or also
adding extra info (e.g., (parts of) journal/syslogs), so laying out a bit
foundations for that now could be reused for that; but as said, no need
to do all that now.
Also please separate adding and using file2text and adding the filter part
into two commits, those are different things.
>
> In addition to the comments, also filter the `cipassword` option
> since it contains the hash of the password.
>
> To facilitate the filtering, a new sub 'file2text' is introduced that
> can filter the file contents if required.
> This sub replaces the 'cat ...' commands.
>
> Signed-off-by: Mira Limbeck
> ---
> I did not add print to STDERR in file2text for now since it got quite
> chatty.
> If this is wanted, I'll send a v2 adding it. But since file2text is also
> called by dir2text the 'OK' at the end won't always align, especially
> when dir2text is used.
>
> PVE/Report.pm | 48 +---
> 1 file changed, 33 insertions(+), 15 deletions(-)
>
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 90b7cb1c..7ebe98f7 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -5,16 +5,34 @@ use warnings;
>
> use PVE::Tools;
>
> +my sub file2text {
> +my ($file, $filter) = @_;
> +my $text = "\n# cat $file\n";
I agree with the sentiment of Stefan's comment but comment something else
that we do is IMO not ideal, tends to get out of date soon and might
be hard to output a grep invocation that behaves really 1:1 as the code here
does, or will do sometimes.
So maybe just output something like:
my $text = "\n# ". ($filter ? 'filtered' : '') ." cat $file\n";
> +
> +my $contents = PVE::Tools::file_get_contents($file);
> +if ($filter) {
> + foreach my $line (split('\n', $contents)) {
> + next if $line =~ m/^\s*#/;
> + next if $line =~ m/^cipassword/;
not sure if it's a good idea to just plainly filter, I mean only guest configs
are setting $filter anyway, but IMO it's still a bid odd coupling; maybe switch
to a $code-ref and calling that, with a filter_guest_config sub passed then on
the call sites?
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager] report: filter comments in VM/CT configs
On 12/15/22 17:57, Mira Limbeck wrote:
> Since some users keep their passwords in the VM/CT configs as comments
> and those are most of the time unnecessary when looking through the
> report, filter those.
>
> In addition to the comments, also filter the `cipassword` option
> since it contains the hash of the password.
>
> To facilitate the filtering, a new sub 'file2text' is introduced that
> can filter the file contents if required.
> This sub replaces the 'cat ...' commands.
>
> Signed-off-by: Mira Limbeck
> ---
> I did not add print to STDERR in file2text for now since it got quite
> chatty.
> If this is wanted, I'll send a v2 adding it. But since file2text is also
> called by dir2text the 'OK' at the end won't always align, especially
> when dir2text is used.
>
> PVE/Report.pm | 48 +---
> 1 file changed, 33 insertions(+), 15 deletions(-)
>
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 90b7cb1c..7ebe98f7 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -5,16 +5,34 @@ use warnings;
>
> use PVE::Tools;
>
> +my sub file2text {
> +my ($file, $filter) = @_;
> +my $text = "\n# cat $file\n";
> +
maybe this should be something like `cat $file | grep -v
"\(\(^\s*#\)\|\(^cipassword\)\)"` or `filtered contents of $file`? just
to make it clearer to someone that just looks at the report that we
don't *just* do a `cat` anymore, but do filter sensitive information.
although, i do think that the amount of users that know what is in the
config files and don't check what is actually in the report is probably
rather small.
> +my $contents = PVE::Tools::file_get_contents($file);
> +if ($filter) {
> + foreach my $line (split('\n', $contents)) {
> + next if $line =~ m/^\s*#/;
> + next if $line =~ m/^cipassword/;
> +
> + $text .= "$line\n";
> + }
> +} else {
> + $text .= $contents;
> +}
> +
> +return $text;
> +}
> +
> # output the content of all the files of a directory
> my sub dir2text {
> -my ($target_dir, $regexp) = @_;
> +my ($target_dir, $regexp, $filter) = @_;
>
> print STDERR "dir2text '${target_dir}${regexp}'...";
> my $text = '';
> PVE::Tools::dir_glob_foreach($target_dir, $regexp, sub {
> my ($file) = @_;
> - $text .= "\n# cat $target_dir$file\n";
> - $text .= PVE::Tools::file_get_contents($target_dir.$file)."\n";
> + $text .= file2text($target_dir.$file, $filter)."\n";
> });
> return $text;
> }
> @@ -30,9 +48,9 @@ my $init_report_cmds = sub {
> cmds => [
> 'hostname',
> 'pveversion --verbose',
> - 'cat /etc/hosts',
> + sub { file2text('/etc/hosts') },
> 'pvesubscription get',
> - 'cat /etc/apt/sources.list',
> + sub { file2text('/etc/apt/sources.list') },
> sub { dir2text('/etc/apt/sources.list.d/', '.*list') },
> sub { dir2text('/etc/apt/sources.list.d/', '.*sources') },
> 'lscpu',
> @@ -50,9 +68,9 @@ my $init_report_cmds = sub {
> storage => {
> order => 30,
> cmds => [
> - 'cat /etc/pve/storage.cfg',
> + sub { file2text('/etc/pve/storage.cfg') },
> 'pvesm status',
> - 'cat /etc/fstab',
> + sub { file2text('/etc/fstab') },
> 'findmnt --ascii',
> 'df --human -T',
> 'proxmox-boot-tool status',
> @@ -62,9 +80,9 @@ my $init_report_cmds = sub {
> order => 40,
> cmds => [
> 'qm list',
> - sub { dir2text('/etc/pve/qemu-server/', '\d.*conf') },
> + sub { dir2text('/etc/pve/qemu-server/', '\d.*conf', 1) },
> 'pct list',
> - sub { dir2text('/etc/pve/lxc/', '\d.*conf') },
> + sub { dir2text('/etc/pve/lxc/', '\d.*conf', 1) },
> ],
> },
> network => {
> @@ -73,14 +91,14 @@ my $init_report_cmds = sub {
> 'ip -details -statistics address',
> 'ip -details -4 route show',
> 'ip -details -6 route show',
> - 'cat /etc/network/interfaces',
> + sub { file2text('/etc/network/interfaces') },
> ],
> },
> firewall => {
> order => 50,
> cmds => [
> sub { dir2text('/etc/pve/firewall/', '.*fw') },
> - 'cat /etc/pve/local/host.fw',
> + sub { file2text('/etc/pve/local/host.fw') },
> 'iptables-save',
> ],
> },
> @@ -89,7 +107,7 @@ my $init_report_cmds = sub {
> cmds => [
> 'pvecm nodes',
> 'pvecm status',
> - 'cat /etc/pve/corosync.conf 2>/dev/null',
> + sub { file2text('/etc/pve/corosync.conf') },
> 'ha-manager status',
> ],
> },
> @@ -135,7 +153,7 @@ my $init_report_cmds = sub {
> 'ceph df',
[pve-devel] [PATCH manager] report: filter comments in VM/CT configs
Since some users keep their passwords in the VM/CT configs as comments
and those are most of the time unnecessary when looking through the
report, filter those.
In addition to the comments, also filter the `cipassword` option
since it contains the hash of the password.
To facilitate the filtering, a new sub 'file2text' is introduced that
can filter the file contents if required.
This sub replaces the 'cat ...' commands.
Signed-off-by: Mira Limbeck
---
I did not add print to STDERR in file2text for now since it got quite
chatty.
If this is wanted, I'll send a v2 adding it. But since file2text is also
called by dir2text the 'OK' at the end won't always align, especially
when dir2text is used.
PVE/Report.pm | 48 +---
1 file changed, 33 insertions(+), 15 deletions(-)
diff --git a/PVE/Report.pm b/PVE/Report.pm
index 90b7cb1c..7ebe98f7 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -5,16 +5,34 @@ use warnings;
use PVE::Tools;
+my sub file2text {
+my ($file, $filter) = @_;
+my $text = "\n# cat $file\n";
+
+my $contents = PVE::Tools::file_get_contents($file);
+if ($filter) {
+ foreach my $line (split('\n', $contents)) {
+ next if $line =~ m/^\s*#/;
+ next if $line =~ m/^cipassword/;
+
+ $text .= "$line\n";
+ }
+} else {
+ $text .= $contents;
+}
+
+return $text;
+}
+
# output the content of all the files of a directory
my sub dir2text {
-my ($target_dir, $regexp) = @_;
+my ($target_dir, $regexp, $filter) = @_;
print STDERR "dir2text '${target_dir}${regexp}'...";
my $text = '';
PVE::Tools::dir_glob_foreach($target_dir, $regexp, sub {
my ($file) = @_;
- $text .= "\n# cat $target_dir$file\n";
- $text .= PVE::Tools::file_get_contents($target_dir.$file)."\n";
+ $text .= file2text($target_dir.$file, $filter)."\n";
});
return $text;
}
@@ -30,9 +48,9 @@ my $init_report_cmds = sub {
cmds => [
'hostname',
'pveversion --verbose',
- 'cat /etc/hosts',
+ sub { file2text('/etc/hosts') },
'pvesubscription get',
- 'cat /etc/apt/sources.list',
+ sub { file2text('/etc/apt/sources.list') },
sub { dir2text('/etc/apt/sources.list.d/', '.*list') },
sub { dir2text('/etc/apt/sources.list.d/', '.*sources') },
'lscpu',
@@ -50,9 +68,9 @@ my $init_report_cmds = sub {
storage => {
order => 30,
cmds => [
- 'cat /etc/pve/storage.cfg',
+ sub { file2text('/etc/pve/storage.cfg') },
'pvesm status',
- 'cat /etc/fstab',
+ sub { file2text('/etc/fstab') },
'findmnt --ascii',
'df --human -T',
'proxmox-boot-tool status',
@@ -62,9 +80,9 @@ my $init_report_cmds = sub {
order => 40,
cmds => [
'qm list',
- sub { dir2text('/etc/pve/qemu-server/', '\d.*conf') },
+ sub { dir2text('/etc/pve/qemu-server/', '\d.*conf', 1) },
'pct list',
- sub { dir2text('/etc/pve/lxc/', '\d.*conf') },
+ sub { dir2text('/etc/pve/lxc/', '\d.*conf', 1) },
],
},
network => {
@@ -73,14 +91,14 @@ my $init_report_cmds = sub {
'ip -details -statistics address',
'ip -details -4 route show',
'ip -details -6 route show',
- 'cat /etc/network/interfaces',
+ sub { file2text('/etc/network/interfaces') },
],
},
firewall => {
order => 50,
cmds => [
sub { dir2text('/etc/pve/firewall/', '.*fw') },
- 'cat /etc/pve/local/host.fw',
+ sub { file2text('/etc/pve/local/host.fw') },
'iptables-save',
],
},
@@ -89,7 +107,7 @@ my $init_report_cmds = sub {
cmds => [
'pvecm nodes',
'pvecm status',
- 'cat /etc/pve/corosync.conf 2>/dev/null',
+ sub { file2text('/etc/pve/corosync.conf') },
'ha-manager status',
],
},
@@ -135,7 +153,7 @@ my $init_report_cmds = sub {
'ceph df',
'ceph osd df tree',
'ceph device ls',
- 'cat /etc/ceph/ceph.conf',
+ sub { file2text('/etc/ceph/ceph.conf') },
'ceph config dump',
'pveceph pool ls',
'ceph versions',
@@ -144,8 +162,8 @@ my $init_report_cmds = sub {
if (cmd_exists('multipath')) {
push @{$report_def->{disks}->{cmds}},
- 'cat /etc/multipath.conf',
- 'cat /etc/multipath/wwids',
+ sub { file2text('/etc/multipath.conf') },
+ sub { file2text('/etc/multipath/wwids') },
