RE: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Roberto Sassu via devel]

> From: Brandon Nielsen [mailto:niels...@jetfuse.net]
> Sent: Wednesday, January 26, 2022 5:14 PM
> On 1/26/22 3:25 AM, Roberto Sassu via devel wrote:
> 
> [Snip]
> 
> >
> > - web servers or other kind of servers where you, as client, would
> >like the guarantee that your data is processed only if the software
> >running in the server is not compromised
> >
> 
> For what it's worth, I, and several people I work with, have been hoping
> for something to enable this kind of functionality for quite some time
> now. Hope to see more about it soon!

Thanks Brandon! That is very encouraging.

All the pieces have been already developed and demonstrated
at the final review meeting of the FutureTPM European project.

Here, at the bottom of the email, you can find some links:

https://www.spinics.net/lists/fedora-devel/msg295543.html

I planned to create a dedicated Fedora Change for the remote
attestation part, which would depend on:

https://fedoraproject.org/wiki/Changes/DIGLIM

I'm still looking for some support from people interested like
you, to comment in the kernel mailing list and help for the
acceptance of the kernel patches in the upstream kernel.

Thanks

Roberto

HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063
Managing Director: Li Peng, Zhong Ronghua
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Brandon Nielsen]

On 1/26/22 3:25 AM, Roberto Sassu via devel wrote:

[Snip]



- web servers or other kind of servers where you, as client, would
   like the guarantee that your data is processed only if the software
   running in the server is not compromised



For what it's worth, I, and several people I work with, have been hoping 
for something to enable this kind of functionality for quite some time 
now. Hope to see more about it soon!

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Miroslav Suchý]

Dne 21. 01. 22 v 17:08 Roberto Sassu via devel napsal(a):

(note for the infrastructure mailing list: please check if the changes
I'm proposing could be tested in the Fedora infrastructure, like Copr)


Copr does not use `rpmsign` at all. Copr uses `obs-sign` [1]. The benefits of obs-sign is that you do not need to send 
whole package to the signing machine and then back. You send just hash there and signature back.


So to test this in Copr the --signverity has to be first implemented in 
obs-sign.

[1] https://github.com/openSUSE/obs-sign

Miroslav
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Stephen Snow]

On Wed, 2022-01-26 at 09:25 +, Roberto Sassu via devel wrote:
> > ...snip

> 
> If the users often make changes on their system, with high
> privileges,
> I agree that DIGLIM would simply cause too much overhead for
> the configuration (every time the users make a change, they have
> to whitelist their software).
> 
Yeah that will definitely be a problem in my use cases. You know, if I
wanted to have something interfere with my work, like Windows, I would
just install Windows.


Stephen
> Roberto
> 
> HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063
> Managing Director: Li Peng, Zhong Ronghua
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

RE: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Roberto Sassu via devel]

> From: Kevin Fenzi [mailto:ke...@scrye.com]
> Sent: Tuesday, January 25, 2022 7:30 PM
> On Fri, Jan 21, 2022 at 04:08:04PM +, Roberto Sassu via devel wrote:
> > Hi everyone
> >
> > (note for the infrastructure mailing list: please check if the changes
> > I'm proposing could be tested in the Fedora infrastructure, like Copr)
> 
> copr uses a different signing setup... so probibly won't work there.
> 
> We could perhaps work with you to try and get it in our staging env to
> test, but I am not sure how involved that might be. I guess the first
> step would be filing a infratructure ticket and explain what you would
> need to change/test.

Thanks, Kevin. Will do.

> So, question for the change owners:
> 
> This is not enabled by default, I assume you wish to enable it for
> something, can you expand on what you will use it for and what that
> helps you with?
> 
> Also, is there any plan to enable it by default? Is there some way this
> could be useful to all our users? or more of our users?
> 
> This change and the DIGLIM change both feel to me like we are spending
> time and effort enabling something that only helps a tiny fraction of
> our users. Is there some way this could be more generally useful to our
> users?

I guess not many would use the remote attestation capability
offered by DIGLIM (sealing a TPM key to the software running
in the machine). The two scenarios in which that could be used
are:

- web servers or other kind of servers where you, as client, would
  like the guarantee that your data is processed only if the software
  running in the server is not compromised

- closed networks where you, as network administrator, would like
  the guarantee that your clients can access those networks only if
  their software is not compromised


Probably, the secure boot functionality at application level is much
more appealing, and less invasive. It should have almost zero impact
if the users use their system for web browsing or writing. Although
it is unlikely to happen, this feature prevents arbitrary execution of
software, even by root. The trust anchor will be the kernel, and one
would not have to rely on the correct execution of the package
manager (and on its verification of the package signature).

If the users often make changes on their system, but confined to
their environment, DIGLIM could still be helpful, as the damage of
running arbitrary software will be limited to the users environment
(no arbitrary software execution as root).

If the users often make changes on their system, with high privileges,
I agree that DIGLIM would simply cause too much overhead for
the configuration (every time the users make a change, they have
to whitelist their software).

Roberto

HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063
Managing Director: Li Peng, Zhong Ronghua
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Kevin Fenzi]

On Fri, Jan 21, 2022 at 04:08:04PM +, Roberto Sassu via devel wrote:
> Hi everyone
> 
> (note for the infrastructure mailing list: please check if the changes
> I'm proposing could be tested in the Fedora infrastructure, like Copr)

copr uses a different signing setup... so probibly won't work there. 

We could perhaps work with you to try and get it in our staging env to
test, but I am not sure how involved that might be. I guess the first
step would be filing a infratructure ticket and explain what you would
need to change/test.

So, question for the change owners:

This is not enabled by default, I assume you wish to enable it for
something, can you expand on what you will use it for and what that
helps you with?

Also, is there any plan to enable it by default? Is there some way this
could be useful to all our users? or more of our users?

This change and the DIGLIM change both feel to me like we are spending
time and effort enabling something that only helps a tiny fraction of
our users. Is there some way this could be more generally useful to our
users?

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

RE: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Roberto Sassu via devel]

Hi everyone

(note for the infrastructure mailing list: please check if the changes
I'm proposing could be tested in the Fedora infrastructure, like Copr)

I made the first version of the rpm extension to sign fsverity
digests with a GPG key. The patch set (with some bug fixes)
is available here:

https://github.com/robertosassu/rpm/commits/fsverity-gpg-v1

I tested it locally with my own GPG key. I took an existing
Fedora 34 package and signed it with rpmsign:

$ usr/bin/rpmsign --define "%_gpg_name testhost " \
 --define "%_file_signing_key _GPG_" \
 --define "%_file_signing_cert _GPG_" \
 --addsign --signverity 
tmux-3.1c-2.fc34.x86_64.rpm


I then checked that the package has now fsverity signatures:

$ usr/bin/rpm -qp tmux-3.1c-2.fc34.x86_64.rpm \
  --queryformat '[%{RPMTAG_FILENAMES} 
%{RPMTAG_VERITYSIGNATURES}\n]'
[...]
/usr/bin/tmux iQHHBAABCgAxFiEEEiFa0dGZVYzTrIN+rxtXRMfK0McFAmHq0+4THHRlc3Rob3N0
QHRlc3QudGVzdAAKCRCvG1dEx8rQx81nC/42NW9xJx3rcTiR6/5oL55GPkan+OIq
t2dW1clJUOrxOGVy/5JQTQf0MQXA7gzH1yPgcrskkahjSfWlp4pt7oOw3rukUyaO
zVZxue4XE6XESYtolczK4VEhpc8lbm4hj0e4NCg/dKri/+L5wIdJvmqWNeCfl7uZ
[...]

In a VM I tried to install the modified package. The root filesystem
is ext4 and has the fsverity feature enabled.

The fsverity rpm plugin is also enabled and hasn't been modified
to work with the new PGP signatures.

The kernel includes the patch set I recently sent to the kernel
mailing lists to add support for PGP keys and signatures:

https://lore.kernel.org/linux-integrity/2022080318.591029-1-roberto.sa...@huawei.com/

and another patch that calls verify_pgp_signature() in
fs/verity/signature.c.

The first installation attempt fails, due to the missing key
in the .fs-verity keyring:

# usr/bin/rpm -Uhvi ../tmux-3.1c-2.fc34.x86_64.rpm --debug
[...]
D: Plugin: calling hook fsm_file_prepare in fsverity plugin
D: applying signature: 
[...]
D: failed to enable verity (errno 126) for /usr/bin/tmux;61ead62d

Then, I added the required GPG key to the .fs-verity keyring:

# cat /mnt/repos/linux/certs/pubring.gpg | keyctl padd asymmetric test 
%keyring:.fs-verity
76292211

The key is now loaded:

# keyctl show %keyring:.fs-verity
Keyring
  66741466 --a-swrv  0 0  keyring: .fs-verity
  76292211 --als--v  0 0   \_ asymmetric: test

I retried the tmux installation:

# usr/bin/rpm -Uhvi ../tmux-3.1c-2.fc34.x86_64.rpm --debug
[...]
D: Plugin: calling hook fsm_file_prepare in fsverity plugin
D: applying signature: 
[...]
D: fsverity enabled signature for: path /usr/bin/tmux;61ead713 dest 
/usr/bin/tmux

This time the installation is successful, which means that the PGP
signature has been successfully verified.

Roberto

HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063
Managing Director: Li Peng, Zhong Ronghua
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Neal Gompa]

On Fri, Dec 17, 2021 at 4:59 PM Colin Walters  wrote:
>
>
>
> On Mon, Dec 13, 2021, at 5:21 PM, Tom Stellard wrote:
> >
> > Did you test the impact this has on package build times?  Particularly
> > packages like llvm, clang, webkit2gtk3, etc. that have very large
> > debuginfo files?
>
> I think far too often the culture here is "make $change for all RPMs".  But 
> this "everything is an RPM" mindset can lead to outcomes and methodology that 
> is at best weird.
>
> For e.g. "let's try building with newer gcc", it would seem far better to me 
> to e.g. start with the things that are in Fedora CoreOS or Workstation or 
> whatever.  (And, optionally their build dependencies)
>

Subset validation is very useful, yes, but fundamentally the Rawhide
corpus is used to shake out GCC in the first place. It's a big part of
how GCC new stable releases become so good. If we don't do it,
basically nobody will.

> For *this* particular change, the value of pre-signing the -debug RPMs 
> seems...weak.  Or even the `-devel` RPMs.  Now, obviously choosing *which* 
> binaries to sign would require some thought.
> But I think that's worth doing instead of blindly doing everything.

I'm not sure I agree. Outputs become inputs for other processes, and I
think people generally prefer that the integrity of those inputs could
be assured. If we'd want the integrity of the filesystem to be assured
for runtime components, I see no reason that we wouldn't want it for
development components and build-time components too. Moreover, as
infrastructure for granular, network-based access for those things
becomes a reality, having signed blobs for those things becomes more
valuable so that privileged processes can be reasonably assured that
they aren't tampered with.



-- 
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

RE: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Roberto Sassu via devel]

> From: Neal Gompa [mailto:ngomp...@gmail.com]
> Sent: Friday, December 17, 2021 11:17 AM
> On Fri, Dec 17, 2021 at 5:14 AM Roberto Sassu via devel
>  wrote:
> >
> > > In Fedora, we use a new package signing key for each Fedora release.
> > > What key would be used for the fs-verity signatures: the same key,
> > > a separate key? Edit: I see that the Change page says a dedicated key is 
> > > used.
> >
> > Hi all
> >
> > I'm doing related work in this area. I'll provide some additional
> > thoughts.
> >
> > Probably, it could be possible to use the package signing key
> > for the fsverity signatures. However, this would require the
> > kernel to be able to load the PGP keys in the builtin keyring
> > and support PGP signatures.
> >
> > David Howells did that some time ago, and I adapted his
> > patches for the latest kernel. Without going into too much
> > detail, I've modified the kernel build to take the Fedora
> > keys. They are available to verify PGP signatures (I use
> > this feature to verify RPM headers).
> >
> > If there is interest, I could propose the patch set to the kernel
> > mailing list.
> >
> 
> That would be phenomenal. That would allow us to tie things to our
> existing process of key rotation for each Fedora release and leverage
> the existing infrastructure for package signing for fsverity
> signatures too.

Great. I will propose the patch set again. For another Fedora
feature I'm proposing (related to this one):

https://fedoraproject.org/wiki/Changes/DIGLIM

I went as far as to be able to appraise with IMA existing
RPM headers (I added support for PGP appended signatures).

I'm using this information in a different way: loading
the file digests from the RPM header to the kernel and
querying them with IMA during appraisal.

That would work with fsverity digests too. If the concern
is the increased package size, adding fsverity digests to the
RPM header instead of signatures might be more acceptable.

I made a small demo:

https://lore.kernel.org/linux-integrity/a16a628b9e21433198c490500a987...@huawei.com/

where loading of kernel modules is granted by IPE if their
fsverity digest is found in the DIGLIM hash table.

Roberto

HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063
Managing Director: Li Peng, Zhong Ronghua

> > > IIUC, I need to do some steps at each boot:
> > > 1. add all the keys to the keyring
> > > 2. set sysctl fs.verity.require_signatures=1
> > >
> > > So… in 1., do I always load all the keys that Fedora has used for this
> > > purpose, in case there are still some files on disk? Or is there some
> > > mechanism to say that e.g. keys older than F(N-2) are now not necessary?
> > >
> > > Who does 2.?
> > >
> > > I think it'd be hard to enable this during a system upgrade: one would 
> > > need
> > > to reinstall all rpms (with new rpms carrying the fsverity metadata)
> > > or some files become unreadable once 2. is done. This brings a question:
> > > is there some rpm virtual Provides/Requires to specify that the fs-verity
> > > stuff is present? I assume the user would want to triple-check that they
> > > don't have any rpms without the metadata before enabling verification.
> >
> > The kernel does not enforce signature verification unless fsverity
> > is explicitly enabled on a file. I guess the rpm plugin can be configured
> > to enable fsverity only if it finds a signature in the RPM header. That
> > would allow a mixed configuration where some files are protected,
> > other not.
> >
> > This does not seem ideal for mandatory protection, where you want
> > to be sure that integrity is checked, even if only on a subset of files
> > (e.g. executables and shared libraries), regardless of whether fsverity
> > was enabled or not on a file.
> >
> > It could be task of the security subsystem to do this type of enforcement.
> > At the moment, IMA and IPE (Integrity Protection Enforcement) are
> > planning to support fsverity and do the enforcement based on a policy.
> >
> 
> So that means there *will* be a policy control mechanism for this
> enforced by the kernel, cool!
> 
> 
> 
> 
> --
> 真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Neal Gompa]

On Fri, Dec 17, 2021 at 5:14 AM Roberto Sassu via devel
 wrote:
>
> > In Fedora, we use a new package signing key for each Fedora release.
> > What key would be used for the fs-verity signatures: the same key,
> > a separate key? Edit: I see that the Change page says a dedicated key is 
> > used.
>
> Hi all
>
> I'm doing related work in this area. I'll provide some additional
> thoughts.
>
> Probably, it could be possible to use the package signing key
> for the fsverity signatures. However, this would require the
> kernel to be able to load the PGP keys in the builtin keyring
> and support PGP signatures.
>
> David Howells did that some time ago, and I adapted his
> patches for the latest kernel. Without going into too much
> detail, I've modified the kernel build to take the Fedora
> keys. They are available to verify PGP signatures (I use
> this feature to verify RPM headers).
>
> If there is interest, I could propose the patch set to the kernel
> mailing list.
>

That would be phenomenal. That would allow us to tie things to our
existing process of key rotation for each Fedora release and leverage
the existing infrastructure for package signing for fsverity
signatures too.

> > IIUC, I need to do some steps at each boot:
> > 1. add all the keys to the keyring
> > 2. set sysctl fs.verity.require_signatures=1
> >
> > So… in 1., do I always load all the keys that Fedora has used for this
> > purpose, in case there are still some files on disk? Or is there some
> > mechanism to say that e.g. keys older than F(N-2) are now not necessary?
> >
> > Who does 2.?
> >
> > I think it'd be hard to enable this during a system upgrade: one would need
> > to reinstall all rpms (with new rpms carrying the fsverity metadata)
> > or some files become unreadable once 2. is done. This brings a question:
> > is there some rpm virtual Provides/Requires to specify that the fs-verity
> > stuff is present? I assume the user would want to triple-check that they
> > don't have any rpms without the metadata before enabling verification.
>
> The kernel does not enforce signature verification unless fsverity
> is explicitly enabled on a file. I guess the rpm plugin can be configured
> to enable fsverity only if it finds a signature in the RPM header. That
> would allow a mixed configuration where some files are protected,
> other not.
>
> This does not seem ideal for mandatory protection, where you want
> to be sure that integrity is checked, even if only on a subset of files
> (e.g. executables and shared libraries), regardless of whether fsverity
> was enabled or not on a file.
>
> It could be task of the security subsystem to do this type of enforcement.
> At the moment, IMA and IPE (Integrity Protection Enforcement) are
> planning to support fsverity and do the enforcement based on a policy.
>

So that means there *will* be a policy control mechanism for this
enforced by the kernel, cool!




--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Demi Marie Obenour]

On 12/12/21 8:26 AM, Florian Weimer wrote:
> * Zbigniew Jędrzejewski-Szmek:
> 
>> Some more questions about how the verification happens… IIUC, I need to
>> load some keys to the kernel keyring, and then set 
>> fs.verity.require_signatures.
>>
>> Where do the keys come from? How are the keys themselves verified?
>> At what time are they loaded and by whom?
>>
>> (Let's say that I'm an attacker with access to the file system. If
>> the keys are loaded from the file system, can I just drop in a rogue key,
>> similarly to what happens when new keys are distributed as part of
>> distro upgrades?)
>>
>> If fs-verity verification prevents me from successfully modifying or
>> replacing /usr/bin/foo or /usr/lib/systemd/system/foo.service, is
>> there anything which hinders just adding /etc/systemd/system/foo.service
>> that does whatever I want?

There is not.  dm-verity is the only solution to that that I am aware of in
Linux.  dm-verity also solves the problem that someone could tamper with
the block device and exploit a vulnerability in the kernel's filesystem
drivers.

> Even if we do that, we can't rule out two scenarios:
> 
> * The attacker builds a (perhaps unrelated) Fedora package and lets
>   Fedora sign it with the official key.  The file signature is as good
>   as any other produced by Fedora.
> 
>   (Alternatively, the attacker reuses file contents signed elsewhere in
>   Fedora because it has an unintended side effect when used in a
>   different context.  This attack requires finding a suitable gadget in
>   some Fedora package, but a dodgy Fedora package build is not needed.)

NetBSD’s Veriexec has a neat solution to this problem, but it is
significantly more complicated.  It uses additional metadata to validate a
file.

> * The attacker (who could be the legitimate user) changes the system
>   configuration so that the feature is disabled during boot.
> 
> Neither we or hardware suppliers installing Fedora on their devices are
> permitted to plug the second hole because it's required for software
> freedom (or Fedora would have to share its private signing keys).

The obvious solution to the second problem is some form of measured boot:
if the feature is disabled, it shows up in the system measurements, and
access to various secrets may be denied by the TPM.  Android uses a similar
solution: at least on Google Pixel devices, one can unlock the bootloader,
but doing so erases all data on the device.

> The combination of these two unsolvable issues suggests to me that
> anyone who wants to deploy this is better off with their own trust root,
> and that approach will include their particular version of key
> management as well.  But this also means that pre-computed file
> signatures are not particular useful to them.  They would have to
> discard them anyway before deployment.  Their own signing process might
> as well check the RPM header signature instead.
+1 on this.  There have also been bugs in RPM's handling of IMA signatures,
and fs-verity signature handling could have similar issues.  Since IMA and 
fs-verity signatures are currently stored in the signature header, they can
be modified without changing the package signature, so they increase the
attack surface of RPM.

-- 
Sincerely,
Demi Marie Obenour (she/her/hers)


OpenPGP_0xB288B55FFF9C22C1.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Luca Boccassi]

> On Wed, Dec 15, 2021, at 1:45 PM, Luca Boccassi wrote:
> 
> Hmm.  Some interesting stuff going on there but I would have started with a 
> new SELinux
> access vector.  That'd allow fine-grained constraints, e.g. disallowing 
> `init_t` but
> allowing `unconfined_service_t`.  Possibly also landlock should be able to 
> hook into this.
>  IOW it's not clear to me that a new LSM is the best thing for the ecosystem 
> here.
> 
> But bigger picture I'd agree that fs-verity is significantly stronger when 
> coupled
> with such a policy - strong enough to block exploits like the runc one:
> https://unit42.paloaltonetworks.com/breaking-docker-via-runc-explaining-c...

We use this in production, and those kind of constraints are just not enough, 
because there's too many obvious ways around them. What we needed is for the 
kernel to enforce that only signed code shall run, and thus IPE was born, so 
that when coupled with dm-verity/fs-verity it allows to enforce a policy like 
that.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Colin Walters]

On Mon, Dec 13, 2021, at 5:21 PM, Tom Stellard wrote:
> 
> Did you test the impact this has on package build times?  Particularly
> packages like llvm, clang, webkit2gtk3, etc. that have very large 
> debuginfo files?

I think far too often the culture here is "make $change for all RPMs".  But 
this "everything is an RPM" mindset can lead to outcomes and methodology that 
is at best weird.

For e.g. "let's try building with newer gcc", it would seem far better to me to 
e.g. start with the things that are in Fedora CoreOS or Workstation or 
whatever.  (And, optionally their build dependencies)

For *this* particular change, the value of pre-signing the -debug RPMs 
seems...weak.  Or even the `-devel` RPMs.  Now, obviously choosing *which* 
binaries to sign would require some thought.
But I think that's worth doing instead of blindly doing everything.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Colin Walters]

On Wed, Dec 15, 2021, at 1:45 PM, Luca Boccassi wrote:
>> On Fri, Dec 10, 2021 at 10:47:52AM +0100, Vít Ondruch wrote:
>> 
>> Any file covered by fs-verity is immutable after installation. So you
>> cannot modify the contents, the kernel refuses. But you can just
>> replace the file (like during an upgrade), and of course copy and edit
>> in a different location. If replaced, no fs-verity checking is done
>> any more by the kernel. There was some talk about high-level solution
>> to prevent such files from being executed, e.g. an LSM module, but no
>> details... (Thinking about this, it would be pretty hard, because the
>> LSM would need to be smart enough to know which files are installed
>> through rpm, and which files are not. I would love to hear more details
>> about what is planned here.)
>> 
>> Zbyszek
>
> There is such an LSM that supports fs-verity (and dm-verity), being 
> reviewed right now: IPE
>
> https://lore.kernel.org/lkml/81d5e825-1ee2-8f6b-cd9d-07b0f8bd3...@linux.microsoft.com/T/
> https://microsoft.github.io/ipe/

Hmm.  Some interesting stuff going on there but I would have started with a new 
SELinux access vector.  That'd allow fine-grained constraints, e.g. disallowing 
`init_t` but allowing `unconfined_service_t`.  Possibly also landlock should be 
able to hook into this.  IOW it's not clear to me that a new LSM is the best 
thing for the ecosystem here.

But bigger picture I'd agree that fs-verity is significantly stronger when 
coupled with such a policy - strong enough to block exploits like the runc one:
https://unit42.paloaltonetworks.com/breaking-docker-via-runc-explaining-cve-2019-5736/

(Of course that one was *also* stopped by ostree's default read-only bind 
mount, which
 I keep saying is not really primarily about security, but hey it worked there)

I said this about the IMA-for-Fedora proposal too - to me the approach of "sign 
all the RPMs" is not a good idea.  Instead the focus should be on ensuring the 
capability works, and is usable from tools to build custom systems.  And this 
of course runs into a bigger picture question of whether we should emphasize 
the entry point into Fedora being Anaconda/FCOS like (provision and configure a 
"pre-built" system) or more like Image Builder and tools like that.  I think 
the answer has to be "we do both" really - it's just hard.


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Luca Boccassi]

> We don't have a proof of concept for the LSM module. I agree with you that in 
> practice
> it would probably need to implement some kind of "list of files we care 
> about",
> but I do not have an intelligent opinion about that.
> 
> Based on Roberto's comment in a different sub-thread, there could be some 
> ongoing work
> integrating with IMA that might be past the hypothetical stage?

IPE supports fs-verity and dm-verity, and allows to write a policy such as 
"only allow execution of binaries/libraries from fs-verity verified files or 
dm-verity verified volumes):

https://lore.kernel.org/lkml/81d5e825-1ee2-8f6b-cd9d-07b0f8bd3...@linux.microsoft.com/T/
https://microsoft.github.io/ipe/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Luca Boccassi]

> On Fri, Dec 10, 2021 at 10:47:52AM +0100, Vít Ondruch wrote:
> 
> Any file covered by fs-verity is immutable after installation. So you
> cannot modify the contents, the kernel refuses. But you can just
> replace the file (like during an upgrade), and of course copy and edit
> in a different location. If replaced, no fs-verity checking is done
> any more by the kernel. There was some talk about high-level solution
> to prevent such files from being executed, e.g. an LSM module, but no
> details... (Thinking about this, it would be pretty hard, because the
> LSM would need to be smart enough to know which files are installed
> through rpm, and which files are not. I would love to hear more details
> about what is planned here.)
> 
> Zbyszek

There is such an LSM that supports fs-verity (and dm-verity), being reviewed 
right now: IPE

https://lore.kernel.org/lkml/81d5e825-1ee2-8f6b-cd9d-07b0f8bd3...@linux.microsoft.com/T/
https://microsoft.github.io/ipe/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Chris Murphy]

On Tue, Dec 14, 2021 at 4:20 PM Zbigniew Jędrzejewski-Szmek
 wrote:
>
> On Tue, Dec 14, 2021 at 08:08:19PM +0100, Fabio Valentini wrote:

> > I thought fsverity was about determining at runtime that the system
> > has not been tampered with? But if somebody who has (physical) access
> > to the device can just ... move verified files out of the way and put
> > their own (unverified) files there (which then apparently does not
> > trigger red warning signs?) - doesn't that defeat the whole purpose of
> > enabling fsverity?
>
> That's a good question. fs-verity and dm-verity share the same
> underlying concept (merkle trees and signature verification by the
> kernel). So this raises the question that you asked… which can also be
> phrased as "why would you even use fs-verity, if you can do dm-verity"?

dm-verity presents a read-only block device. It can't be modified once created.

fs-verity works at the VFS layer. The block device and file system
remain read-write capable, only the fs-verity enabled file contents
are read-only. Also, fs-verity only really provides integrity checking
in an efficient manner, with an optional mechanism for user space to
enforce authentication - the kernel doesn't do this. A further option
for user space is auditing.

There is this dance in computer security where you're constantly
plugging holes, looking for holes, but also assuming certain holes are
definitely filled. It might seem like a contradiction. But of course
we say physical access means the system is compromised, so you have to
assume that hole is filled or all bets are off. Here you have to
assume enough of user space is not compromised enough that you can
trust that the fs-verity mechanisms can be used (create and verify),
and the reason why you're using this regime is to make certain
modifications impossible without detection so that the system doesn't
become compromised, or at least compromised in silent fashion.
fs-verity does not completely provide a secure and closed system, it's
just one aspect of creating a more secure system.

I'd go so far as to say it takes some esoteric knowledge to understand
that this is only a "probably important small thing", and not a
"critical big thing", to do. This is a few years old, 2018, but it
discusses some of the confusing points when fs-verity was merged.
https://lwn.net/Articles/763729/

Chris Murphy


> My understanding it the following: fs-verity originated in the Android
> world where you can have an unprivileged process downloading a file,
> e.g. a jar. This unprivileged process manages the download, but the
> file is only trusted and executed when it has a matching signature
> from some central authority. The file contains the whole app,
> including all resources, so there is no question of other unverified
> files being used by the app. And the file can be large enough that
> it's practical to do chunked verification, since checksumming the whole
> file on first use would be slow.
>
> We don't really have the same considerations: the download process
> has full privileges, and the download is exploded into individual files,
> and more importantly, unpackaged files are also used.



--
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Matthew Miller]

On Tue, Dec 14, 2021 at 09:18:20PM +, Zbigniew Jędrzejewski-Szmek wrote:
> My understanding it the following: fs-verity originated in the Android
> world where you can have an unprivileged process downloading a file,
> e.g. a jar. This unprivileged process manages the download, but the
> file is only trusted and executed when it has a matching signature
> from some central authority. The file contains the whole app,
> including all resources, so there is no question of other unverified
> files being used by the app. And the file can be large enough that
> it's practical to do chunked verification, since checksumming the whole
> file on first use would be slow.

This does seem rather reminiscent of our LiveCD squashfs situation

-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Zbigniew Jędrzejewski-Szmek]

On Tue, Dec 14, 2021 at 08:08:19PM +0100, Fabio Valentini wrote:
> On Tue, Dec 14, 2021 at 1:45 AM Davide Cavalca via devel
>  wrote:
> >
> > On Mon, 2021-12-13 at 16:00 +0100, Vít Ondruch wrote:
> > > Would it be possible to document the editing of protected file in the
> > > change proposal, probably including example of the best way to do it
> > > (is
> > > it possible to replace the file by symlink?) Or is there a way to
> > > temporary enable the editing with some overlay? Is there any other way
> > > to restore the original file except "dnf reinstall"?
> >
> > I've added this to the wiki:
> > https://fedoraproject.org/wiki/Changes/FsVerityRPM#Can_the_user_modify_a_file_shipped_by_a_package_.28e.g._to_edit_a_script_while_debugging.29_.3F
> >
> > You could restore the original file via "dnf reinstall", or by moving
> > it back into place (rename() and unlink() are allowed on fs-verity
> > enabled files).
> 
> I thought fsverity was about determining at runtime that the system
> has not been tampered with? But if somebody who has (physical) access
> to the device can just ... move verified files out of the way and put
> their own (unverified) files there (which then apparently does not
> trigger red warning signs?) - doesn't that defeat the whole purpose of
> enabling fsverity?

That's a good question. fs-verity and dm-verity share the same
underlying concept (merkle trees and signature verification by the
kernel). So this raises the question that you asked… which can also be
phrased as "why would you even use fs-verity, if you can do dm-verity"? 

My understanding it the following: fs-verity originated in the Android
world where you can have an unprivileged process downloading a file,
e.g. a jar. This unprivileged process manages the download, but the
file is only trusted and executed when it has a matching signature
from some central authority. The file contains the whole app,
including all resources, so there is no question of other unverified
files being used by the app. And the file can be large enough that
it's practical to do chunked verification, since checksumming the whole
file on first use would be slow.

We don't really have the same considerations: the download process
has full privileges, and the download is exploded into individual files,
and more importantly, unpackaged files are also used.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Fabio Valentini]

On Tue, Dec 14, 2021 at 1:45 AM Davide Cavalca via devel
 wrote:
>
> On Mon, 2021-12-13 at 16:00 +0100, Vít Ondruch wrote:
> > Would it be possible to document the editing of protected file in the
> > change proposal, probably including example of the best way to do it
> > (is
> > it possible to replace the file by symlink?) Or is there a way to
> > temporary enable the editing with some overlay? Is there any other way
> > to restore the original file except "dnf reinstall"?
>
> I've added this to the wiki:
> https://fedoraproject.org/wiki/Changes/FsVerityRPM#Can_the_user_modify_a_file_shipped_by_a_package_.28e.g._to_edit_a_script_while_debugging.29_.3F
>
> You could restore the original file via "dnf reinstall", or by moving
> it back into place (rename() and unlink() are allowed on fs-verity
> enabled files).

I thought fsverity was about determining at runtime that the system
has not been tampered with? But if somebody who has (physical) access
to the device can just ... move verified files out of the way and put
their own (unverified) files there (which then apparently does not
trigger red warning signs?) - doesn't that defeat the whole purpose of
enabling fsverity?

Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Boris Burkov via devel]

I don't believe we systematically tested this. We will collect that along with 
the detailed size increase data.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Davide Cavalca via devel]

On Mon, 2021-12-13 at 16:00 +0100, Vít Ondruch wrote:
> Would it be possible to document the editing of protected file in the
> change proposal, probably including example of the best way to do it
> (is 
> it possible to replace the file by symlink?) Or is there a way to 
> temporary enable the editing with some overlay? Is there any other way 
> to restore the original file except "dnf reinstall"?

I've added this to the wiki:
https://fedoraproject.org/wiki/Changes/FsVerityRPM#Can_the_user_modify_a_file_shipped_by_a_package_.28e.g._to_edit_a_script_while_debugging.29_.3F

You could restore the original file via "dnf reinstall", or by moving
it back into place (rename() and unlink() are allowed on fs-verity
enabled files).

Cheers
Davide
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Tom Stellard]

On 12/2/21 11:36, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/FsVerityRPM

== Summary ==

Enable the use of fsverity for installed RPM files validation.

== Owners ==

* Name: [[User:Dcavalca|Davide Cavalca]], [[User:Borisb|Boris
Burkov]], [[User:Filbranden|Filipe Brandenburger]],
[[User:Salimma|Michel Alexandre Salim]], [[User:Malmond|Matthew
Almond]]
* Email: dcava...@fb.com, bor...@fb.com, filbran...@fb.com,
mic...@fb.com, malm...@fb.com


== Detailed description ==

fs-verity is a [https://www.kernel.org/doc/html/latest/filesystems/fsverity.html
Linux kernel feature] that does transparent on-demand
integrity/authenticity verification of the contents of read-only
files, using a hidden Merkle tree (hash tree) associated with the
file. The mechanism is similar to dm-verity, but implemented at the
file level rather than at the block device level.

When fsverity is enabled for a file, the kernel reads every block and
generates a hash tree for it, which is stored within the filesystem.
On subsequent reads, the kernel computes the block hash and compares
it with the one stored in the tree, protecting against alterations and
corruption. Because this happens at the filesystem data block read
layer, it encompasses all file operations (open,
mmap,exec, etc.).

In the context of rpm, there are two parts to this:
* at build time, we compute the Merkle tree for the files within a
package, then sign it and ship it as part of the rpm metadata;
* at run time, if the fsverity rpm plugin is enabled, rpm will install
the fsverity signature key and enable fsverity on files that are
installed.


This proposal is primarily concerned with the first part, which will
make it possible for users to leverage fs-verity for RPM if they so
desire. Specifically, installing and enabling the fs-verity rpm plugin
by default is explicitly considered out of scope here.

=== Caveats ===

 Merkle tree cost 

The Merkle tree used by fsverity needs to be generated (once at build
time, once when the package is installed) and stored on disk. The
generation process involves reading all blocks and computing the hash,
which has a non-trivial cost; however, it does not appear to
meaningfully slow down package installs during empirical testing. Once


Did you test the impact this has on package build times?  Particularly
packages like llvm, clang, webkit2gtk3, etc. that have very large debuginfo 
files?

-Tom
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Zbigniew Jędrzejewski-Szmek]

On Mon, Dec 13, 2021 at 07:32:34PM -, Boris Burkov via devel wrote:
> Sorry this wasn't clear.
> 
> The rpm carries just 'c' (as well as some small, fixed-size metadata for 
> interpreting it, like hash algorithm)
> 
> Just to explain that comment which suggested 'a': we have to compute the 
> Merkle tree at build time in order to get a root hash to sign. The Merkle 
> tree is then fully re-computed by the fs-verity code in the kernel for 
> storage by the local filesystem when the rpm is ultimately installed.

OK, so please adjust the change page not to say "it", but "the
signature" or something like that.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Boris Burkov via devel]

We don't have a proof of concept for the LSM module. I agree with you that in 
practice it would probably need to implement some kind of "list of files we 
care about", but I do not have an intelligent opinion about that.

Based on Roberto's comment in a different sub-thread, there could be some 
ongoing work integrating with IMA that might be past the hypothetical stage?
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Boris Burkov via devel]

Sorry this wasn't clear.

The rpm carries just 'c' (as well as some small, fixed-size metadata for 
interpreting it, like hash algorithm)

Just to explain that comment which suggested 'a': we have to compute the Merkle 
tree at build time in order to get a root hash to sign. The Merkle tree is then 
fully re-computed by the fs-verity code in the kernel for storage by the local 
filesystem when the rpm is ultimately installed.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Vít Ondruch]

Dne 12. 12. 21 v 12:33 Zbigniew Jędrzejewski-Szmek napsal(a):

On Fri, Dec 10, 2021 at 10:47:52AM +0100, Vít Ondruch wrote:

Dne 10. 12. 21 v 0:08 Davide Cavalca via devel napsal(a):

On Fri, 2021-12-03 at 22:08 +, Richard W.M. Jones wrote:

I'm unclear about the threat model - this is an attacker who is
someone able to overwrite single files (eg. /bin/ls) but cannot turn
off the fs-verity system as a whole?

Also if RPM can update /bin/ls then surely an attacker who can widely
compromise system files must also be able to update /bin/ls in the
same way?

Once fs-verity is enabled for a given file (which, in the RPM case,
happens at package installation time), it cannot be disabled, and the
file becomes immutable. One can still rename() or unlink() it (and this
is indeed how rpm is able to replace files when upgrading packages),
but the actual contents cannot be altered.


Trying to debug some issue in shell/ruby/python script, will it be possible
to modify such file?

Any file covered by fs-verity is immutable after installation. So you
cannot modify the contents, the kernel refuses. But you can just
replace the file (like during an upgrade), and of course copy and edit
in a different location. If replaced, no fs-verity checking is done
any more by the kernel. There was some talk about high-level solution
to prevent such files from being executed, e.g. an LSM module, but no
details... (Thinking about this, it would be pretty hard, because the
LSM would need to be smart enough to know which files are installed
through rpm, and which files are not. I would love to hear more details
about what is planned here.)



Thx for explanation.

Would it be possible to document the editing of protected file in the 
change proposal, probably including example of the best way to do it (is 
it possible to replace the file by symlink?) Or is there a way to 
temporary enable the editing with some overlay? Is there any other way 
to restore the original file except "dnf reinstall"?


I am asking because I am torn between the security and OTOH convenient 
debugging if something goes wrong. Don't take me wrong I am far from 
editing system file on daily basis, but I have certainly did that.



Vít



OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Roberto Sassu via devel]

> In Fedora, we use a new package signing key for each Fedora release.
> What key would be used for the fs-verity signatures: the same key,
> a separate key? Edit: I see that the Change page says a dedicated key is used.

Hi all

I'm doing related work in this area. I'll provide some additional
thoughts.

Probably, it could be possible to use the package signing key
for the fsverity signatures. However, this would require the
kernel to be able to load the PGP keys in the builtin keyring
and support PGP signatures.

David Howells did that some time ago, and I adapted his
patches for the latest kernel. Without going into too much
detail, I've modified the kernel build to take the Fedora
keys. They are available to verify PGP signatures (I use
this feature to verify RPM headers).

If there is interest, I could propose the patch set to the kernel
mailing list.

> IIUC, I need to do some steps at each boot:
> 1. add all the keys to the keyring
> 2. set sysctl fs.verity.require_signatures=1
> 
> So… in 1., do I always load all the keys that Fedora has used for this
> purpose, in case there are still some files on disk? Or is there some
> mechanism to say that e.g. keys older than F(N-2) are now not necessary?
> 
> Who does 2.?
> 
> I think it'd be hard to enable this during a system upgrade: one would need
> to reinstall all rpms (with new rpms carrying the fsverity metadata)
> or some files become unreadable once 2. is done. This brings a question:
> is there some rpm virtual Provides/Requires to specify that the fs-verity
> stuff is present? I assume the user would want to triple-check that they
> don't have any rpms without the metadata before enabling verification.

The kernel does not enforce signature verification unless fsverity
is explicitly enabled on a file. I guess the rpm plugin can be configured
to enable fsverity only if it finds a signature in the RPM header. That
would allow a mixed configuration where some files are protected,
other not.

This does not seem ideal for mandatory protection, where you want
to be sure that integrity is checked, even if only on a subset of files
(e.g. executables and shared libraries), regardless of whether fsverity
was enabled or not on a file.

It could be task of the security subsystem to do this type of enforcement.
At the moment, IMA and IPE (Integrity Protection Enforcement) are
planning to support fsverity and do the enforcement based on a policy.

Roberto

HUAWEI TECHNOLOGIES Duesseldorf GmbH, HRB 56063
Managing Director: Li Peng, Zhong Ronghua
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Lennart Poettering]

On Do, 09.12.21 23:55, Fedora Development ML (devel@lists.fedoraproject.org) 
wrote:

> > On Do, 02.12.21 14:36, Ben Cotton (bcotton(a)redhat.com) wrote:
> >
> > Hmm, so what I am really missing on the feature page: what's the
> > attack scenario here? Usually security features come with an attack
> > scenario they are supposed to address. But there's no discussion about
> > that.
> >
> Good point, we were skimpy on that. In case you missed it, Davide
> just laid it out in his reply to the same question from Richard
> Jones higher in the thread. The short answer is that just this
> change as-is has no practical threat model, we viewed it as an
> enablement step that we believe has a realistic path to broadly
> authenticated rpm contents. Alternatively, it does give a
> sophisticated user an opportunity to build something like that for
> themselves as well.

This sounds strange to me. I mean, it's fine if new features initially
only implement a subset of what they eventually are supposed to
accomplish, but it appears to me you add only infrastructure without
*any* use? Infrastructure that has no consuming feature typically
doesn't work. Experience tells us that. Hence: I think it's fine if
you don't go from 0 to 100 right away but there should be at least
*some* benefit if you add such complex infra to the distro.

> > This protects file contents, not the metada, right? So what about the
> > metadata? if I see a fs-verity enabled inode with libssl.so data in it
> > today, and it's a vulnerably version, and I make a hardlink to it, and
> > then it gets replaced by a fixed version (with a slightly updated
> > name) — how you intend to make sure, that i can't fool you into loading
> > my copy of the old file but under the new name?
>
> If the file is still properly signed with a trusted matching
> certificate in the kernel keyring we can't/won't do anything about
> it.

But that's an extremely weak property then? If I can have "ping" and
"poweroff" binaries both signed by Fedora, but then swap them without
the signing being able to detect that, why do this at all?

> > What about signing anyway? What is this precisely signed with? The
> > keys for that when are they rolled over? Who manages those keys? The
> > tea for that, who's doing that in Fedora? Is there any protection
> > against downgrades between RPM package versions? Does this in any way
> > protect combinations of binaries/libraries? I mean, pretty much all
> > programs we ship consist of a large number of ELF objects, and you
> > probably need to sign the combination of them, but this model doesn't
> > look like it offers that at all? If a security bug is found in library
> > X, how is it taken out of equation? What would the workflow for that
> > look like on the Fedora side of things?
> >
> I don't have a particularly specific answer to all of these
> questions, unfortunately. Perhaps someone with more knowledge about
> Fedora's existing key management can weigh in. But imagining we have
> the LSM hooks setup so that we never load anything executable
> without it being signed by a trusted key, then in my mind, banishing
> a bad shared library would require revoking the certificate.

Uff. That might work in focussed deployments, but across a
distribution like Fedora? Everytime there's an exploitable bug in
*any* of the software Fedora ships you want to invalidate basically
the *whole* distro? That sounds absolutely unmanagable and unrealistic
to me.

> I think one could draw a reasonable parallel between package signing
> and file signing here, though. Users assemble meaningful system
> functionality out of a collection of packages, which are each
> individually signed. One of those packages could then be found to be
> compromised, and need to be removed, otherwise a user could download
> and install a perfectly valid signed package and have a bad time,
> which undermines the value of the signing. Is there something about
> composing executables out of files that is meaningfully different in
> this sense from composing systems out of packages?

RPMs sign metadata. You can't trick RPM that easily into accepting
"ping" as "poweroff" binary or vice versa. I mean, RPM's security
model is from the 1990's, and it only does validation on request, not
on access. So it's not great, but it *did* get right to authenticate a
whole package in combination with all its metadata. And since
versioned deps are part of that metadata you can express relationships.

Still, this feature appears totally not throught to the end. I don't
get the usecase?

Lennart

--
Lennart Poettering, Berlin
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Florian Weimer]

* Zbigniew Jędrzejewski-Szmek:

> Some more questions about how the verification happens… IIUC, I need to
> load some keys to the kernel keyring, and then set 
> fs.verity.require_signatures.
>
> Where do the keys come from? How are the keys themselves verified?
> At what time are they loaded and by whom?
>
> (Let's say that I'm an attacker with access to the file system. If
> the keys are loaded from the file system, can I just drop in a rogue key,
> similarly to what happens when new keys are distributed as part of
> distro upgrades?)
>
> If fs-verity verification prevents me from successfully modifying or
> replacing /usr/bin/foo or /usr/lib/systemd/system/foo.service, is
> there anything which hinders just adding /etc/systemd/system/foo.service
> that does whatever I want?

Even if we do that, we can't rule out two scenarios:

* The attacker builds a (perhaps unrelated) Fedora package and lets
  Fedora sign it with the official key.  The file signature is as good
  as any other produced by Fedora.

  (Alternatively, the attacker reuses file contents signed elsewhere in
  Fedora because it has an unintended side effect when used in a
  different context.  This attack requires finding a suitable gadget in
  some Fedora package, but a dodgy Fedora package build is not needed.)

* The attacker (who could be the legitimate user) changes the system
  configuration so that the feature is disabled during boot.

Neither we or hardware suppliers installing Fedora on their devices are
permitted to plug the second hole because it's required for software
freedom (or Fedora would have to share its private signing keys).

The combination of these two unsolvable issues suggests to me that
anyone who wants to deploy this is better off with their own trust root,
and that approach will include their particular version of key
management as well.  But this also means that pre-computed file
signatures are not particular useful to them.  They would have to
discard them anyway before deployment.  Their own signing process might
as well check the RPM header signature instead.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Zbigniew Jędrzejewski-Szmek]

On Fri, Dec 10, 2021 at 10:47:52AM +0100, Vít Ondruch wrote:
> 
> Dne 10. 12. 21 v 0:08 Davide Cavalca via devel napsal(a):
> >On Fri, 2021-12-03 at 22:08 +, Richard W.M. Jones wrote:
> >>I'm unclear about the threat model - this is an attacker who is
> >>someone able to overwrite single files (eg. /bin/ls) but cannot turn
> >>off the fs-verity system as a whole?
> >>
> >>Also if RPM can update /bin/ls then surely an attacker who can widely
> >>compromise system files must also be able to update /bin/ls in the
> >>same way?
> >Once fs-verity is enabled for a given file (which, in the RPM case,
> >happens at package installation time), it cannot be disabled, and the
> >file becomes immutable. One can still rename() or unlink() it (and this
> >is indeed how rpm is able to replace files when upgrading packages),
> >but the actual contents cannot be altered.
> 
> 
> Trying to debug some issue in shell/ruby/python script, will it be possible
> to modify such file?

Any file covered by fs-verity is immutable after installation. So you
cannot modify the contents, the kernel refuses. But you can just
replace the file (like during an upgrade), and of course copy and edit
in a different location. If replaced, no fs-verity checking is done
any more by the kernel. There was some talk about high-level solution
to prevent such files from being executed, e.g. an LSM module, but no
details... (Thinking about this, it would be pretty hard, because the
LSM would need to be smart enough to know which files are installed
through rpm, and which files are not. I would love to hear more details
about what is planned here.)

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Zbigniew Jędrzejewski-Szmek]

Some more questions about how the verification happens… IIUC, I need to
load some keys to the kernel keyring, and then set fs.verity.require_signatures.

Where do the keys come from? How are the keys themselves verified?
At what time are they loaded and by whom?

(Let's say that I'm an attacker with access to the file system. If
the keys are loaded from the file system, can I just drop in a rogue key,
similarly to what happens when new keys are distributed as part of
distro upgrades?)

If fs-verity verification prevents me from successfully modifying or
replacing /usr/bin/foo or /usr/lib/systemd/system/foo.service, is
there anything which hinders just adding /etc/systemd/system/foo.service
that does whatever I want?

On Thu, Dec 09, 2021 at 11:08:53PM +, Davide Cavalca via devel wrote:
> - you could use a LSM to enforce that exec() can only happen on files
> with valid fs-verity signatures; this would protect any binary
> - you could use a launcher booted from secure storage (say, a dm-verity

Is there some LSM module like this ready for use?

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Zbigniew Jędrzejewski-Szmek]

> * at run time, if the fsverity rpm plugin is enabled, rpm will install
> the fsverity signature key and enable fsverity on files that are
> installed.

This requires CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y. Currently our
kernels are built without that. It seems like a simple addition (the
amount of code guarded by this is very small), but this should be added
to the Scope.

> In the context of rpm, there are two parts to this:
> * at build time, we compute the Merkle tree for the files within a
> package, then sign it and ship it as part of the rpm metadata;

There was already some chatter about signatures and verification in
the thread, but I didn't see a definite answer, and I think this is the
part that is currently the most underspecified.

In Fedora, we use a new package signing key for each Fedora release.
What key would be used for the fs-verity signatures: the same key,
a separate key? Edit: I see that the Change page says a dedicated key is used.

OK, so is this key rotated one per release or some different schedule?
How would the public part of the key be delivered to users?

How does one actually enable signature verification? Let's say I have
a kernel with CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y and rpm-plugin-fsverity
installed, and I installed some rpms, and the files from those rpms now
have now had the FS_IOC_ENABLE_VERITY ioctl done.

IIUC, I need to do some steps at each boot:
1. add all the keys to the keyring
2. set sysctl fs.verity.require_signatures=1

So… in 1., do I always load all the keys that Fedora has used for this
purpose, in case there are still some files on disk? Or is there some
mechanism to say that e.g. keys older than F(N-2) are now not necessary?

Who does 2.?

I think it'd be hard to enable this during a system upgrade: one would need
to reinstall all rpms (with new rpms carrying the fsverity metadata)
or some files become unreadable once 2. is done. This brings a question:
is there some rpm virtual Provides/Requires to specify that the fs-verity
stuff is present? I assume the user would want to triple-check that they
don't have any rpms without the metadata before enabling verification.

Zbyszek

P.S. Those questions are about the mechanism, not policy or the threat model.
I have some questions about that, but I'll ask separately, since this mail
is lengthy already.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Zbigniew Jędrzejewski-Szmek]

On Fri, Dec 03, 2021 at 05:31:21PM -, Boris Burkov via devel wrote:
> The top-level hash is calculated for each file, then that hash is signed with 
> the inputted rsa key pair and the signed hash is appended to the array of 
> signed hashes in the rpm metadata. I am guessing the way we worded the 
> proposal is a little unclear because we call it "the" signature when it's one 
> rpm metadata item that's an array of the signatures.
> 
> fs-verity the kernel feature operates on a per-file basis, and since the 
> ultimate goal is to deliver fs-verity enabled files on the installer's 
> system, we need each file's signature in the rpm. At install, we call the 
> fs-verity enable ioctl for each file, passing in its signature to make use of 
> the kernel authentication functionality.

What exactly is appended to the rpm:
a) the merkle tree
b) the top-level hash from the merkle tree
c) the signature for b
d) some combination of a, b, c?

Above you seem to say "b+c" ("signed hash").

In the Change page:
> at build time, we compute the Merkle tree for the files within a
> package, then sign it and ship it as part of the rpm metadata;

…which is "a".

"c" would make sense to me.

Please clarify.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Stephen John Smoogen]

On Thu, 2 Dec 2021 at 14:37, Ben Cotton  wrote:
>
> https://fedoraproject.org/wiki/Changes/FsVerityRPM
>
> == Summary ==
>
> Enable the use of fsverity for installed RPM files validation.
>
> == Owners ==
>
> * Name: [[User:Dcavalca|Davide Cavalca]], [[User:Borisb|Boris
> Burkov]], [[User:Filbranden|Filipe Brandenburger]],
> [[User:Salimma|Michel Alexandre Salim]], [[User:Malmond|Matthew
> Almond]]
> * Email: dcava...@fb.com, bor...@fb.com, filbran...@fb.com,
> mic...@fb.com, malm...@fb.com


> ** koji integration: koji will need to add the fs-verity metadata to
> packages when signing them

I am going to say that this is a bigger sticking point than people are
realising and this is not a F36 but a F37 change.

The changes to koji, signing infrastructure and testing have to be put
in place before January 19 2022 (2022-01-19) for the mass rebuild[1].
They need to have been in staging and testing before then so that bugs
could be worked out and various groups could work things out.  Due to
various end of year holidays and 'use your PTO before its lost'
schedules, many of the developers who could sign off on the changes to
koji/bodhi/whatever-testing/sigul/etc depending on the changes needed
are not going to be available. Even if the changes are trivial ones,
this is a very short window to land things.

At best for F36, I could see the changes getting put into Fedora
before the beta and some packages signed.

If you want this for F37, then all that work needs to be ready and
done by (2022-07-20)[2].

[1] https://fedorapeople.org/groups/schedule/f-36/f-36-key-tasks.html
[2] https://fedorapeople.org/groups/schedule/f-37/f-37-key-tasks.html

-- 
Stephen J Smoogen.
Let us be kind to one another, for most of us are fighting a hard
battle. -- Ian MacClaren
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Vít Ondruch]

Dne 10. 12. 21 v 0:08 Davide Cavalca via devel napsal(a):

On Fri, 2021-12-03 at 22:08 +, Richard W.M. Jones wrote:

I'm unclear about the threat model - this is an attacker who is
someone able to overwrite single files (eg. /bin/ls) but cannot turn
off the fs-verity system as a whole?

Also if RPM can update /bin/ls then surely an attacker who can widely
compromise system files must also be able to update /bin/ls in the
same way?

Once fs-verity is enabled for a given file (which, in the RPM case,
happens at package installation time), it cannot be disabled, and the
file becomes immutable. One can still rename() or unlink() it (and this
is indeed how rpm is able to replace files when upgrading packages),
but the actual contents cannot be altered.



Trying to debug some issue in shell/ruby/python script, will it be 
possible to modify such file?



Vít



OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Boris Burkov via devel]

> On Do, 02.12.21 14:36, Ben Cotton (bcotton(a)redhat.com) wrote:
> 
> Hmm, so what I am really missing on the feature page: what's the
> attack scenario here? Usually security features come with an attack
> scenario they are supposed to address. But there's no discussion about
> that.
> 
Good point, we were skimpy on that. In case you missed it, Davide just laid it 
out in his reply to the same question from Richard Jones higher in the thread. 
The short answer is that just this change as-is has no practical threat model, 
we viewed it as an enablement step that we believe has a realistic path to 
broadly authenticated rpm contents. Alternatively, it does give a sophisticated 
user an opportunity to build something like that for themselves as well.

> This protects file contents, not the metada, right? So what about the
> metadata? if I see a fs-verity enabled inode with libssl.so data in it
> today, and it's a vulnerably version, and I make a hardlink to it, and
> then it gets replaced by a fixed version (with a slightly updated
> name) — how you intend to make sure, that i can't fool you into loading
> my copy of the old file but under the new name?
> 

If the file is still properly signed with a trusted matching certificate in the 
kernel keyring we can't/won't do anything about it.

> What about signing anyway? What is this precisely signed with? The
> keys for that when are they rolled over? Who manages those keys? The
> tea for that, who's doing that in Fedora? Is there any protection
> against downgrades between RPM package versions? Does this in any way
> protect combinations of binaries/libraries? I mean, pretty much all
> programs we ship consist of a large number of ELF objects, and you
> probably need to sign the combination of them, but this model doesn't
> look like it offers that at all? If a security bug is found in library
> X, how is it taken out of equation? What would the workflow for that
> look like on the Fedora side of things?
> 
I don't have a particularly specific answer to all of these questions, 
unfortunately. Perhaps someone with more knowledge about Fedora's existing key 
management can weigh in. But imagining we have the LSM hooks setup so that we 
never load anything executable without it being signed by a trusted key, then 
in my mind, banishing a bad shared library would require revoking the 
certificate.

I think one could draw a reasonable parallel between package signing and file 
signing here, though. Users assemble meaningful system functionality out of a 
collection of packages, which are each individually signed. One of those 
packages could then be found to be compromised, and need to be removed, 
otherwise a user could download and install a perfectly valid signed package 
and have a bad time, which undermines the value of the signing. Is there 
something about composing executables out of files that is meaningfully 
different in this sense from composing systems out of packages?

> The security model seems really strange to me I must say: so much
> infrastructure, and it doesn't protect at all how our programs are
> usually combined?
> 
> Puzzled,
> 
> Lennart
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Davide Cavalca via devel]

On Sat, 2021-12-04 at 23:46 +0100, Kevin Kofler via devel wrote:
> Davide Cavalca via devel wrote:
> > To clarify: RPM does support files validation, but fs-verity is
> > more
> > than just that. With RPM, the validation only happens on install
> > time,
> > and when one runs rpm -V manually. With fs-verity, the validation
> > happens on-demand whenever a block of a file that originated from
> > an
> > RPM is accessed. This means, for example, that if an attacker
> > replaces
> > /bin/ls on disk with a compromised one, the next time it's read
> > from
> > disk (e.g. because you ran it) you will see a validation failure
> > and
> > the syscall will be blocked, preventing the compromised code from
> > being
> > executed.
> 
> This means that there is a performance cost in addition to the disk
> space 
> cost (because something has to compute those checksums each time the
> file is 
> acessed).

There's only a performance cost if fs-verity is actually enabled (which
is not in scope for this proposal). The checksums are computed on-
demand on a per-block basis, so you only end up checksumming the pages
you're actually accessing. 

>  It also means that it is harder for users to exercise their right 
> to modify the Free Software (because replacing or patching RPM-
> installed 
> binaries will lead to them failing to execute).

There's nothing stopping the user from loading their own key in the
kernel keyring and then installing their own locally-build RPM that has
been verity-signed with their own key. And again, this only becomes a
concern if one is actually enabling fs-verity in the first place.

> 
> Since the change also adds to the metadata in the RPM, that means
> that it 
> also increases the size of the RPMs. With keepcache=1, this also
> translates 
> to increased disk space use. But even if the user does not keep
> cached RPMs, 
> the download sizes will increase, which can cost time and for some
> users 
> even money.

That's correct, there will be (modest) increase in the size of the
RPMs. We're going to collect some more data to quantify this more
concretely.

Cheers
Davide
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Boris Burkov via devel]

> If I enable FS-verity and later find that I need to patch a file to fix
> some problem, how do I as the sysadmin tell Linux that this change is
> authorized? Do I disable FS-verity for that specific file? Disable
> FS-verity globally? Add my own key to the kernel's keyring? Build and
> sign my own RPM package?
> 
> What prevents an attacker from doing the same?

I think this is a good, fair point and is a serious tradeoff in authenticating 
distributed files. However, I believe it should be possible for the user to 
securely configure a keypair and load the certificate in the fs-verity keyring 
s.t. they can sign the files they craft themselves, without allowing an 
attacker to, just like they would to normally sign things. So you could copy 
the file, modify it to your liking, (or just rebuild the rpm locally) then 
enable verity with your own signature.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Davide Cavalca via devel]

On Sat, 2021-12-04 at 09:37 -0500, Stephen John Smoogen wrote:
> 
> Or just pad /usr/bin/rpm with some null characters at the end to break
> its signature and also stop updates from happening. [Or the fs-verity
> daemon which will report that these problems are occuring. ]

If the attacker has filesystem access, this wouldn't work, as fs-verity
makes the file immutable. If they have block level access to the
underlying storage, they can alter the data blocks of the rpm binary,
and that would indeed result in failure on the next exec as the
signature wouldn't match.

Cheers
Davide
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Davide Cavalca via devel]

On Fri, 2021-12-03 at 22:08 +, Richard W.M. Jones wrote:
> I'm unclear about the threat model - this is an attacker who is
> someone able to overwrite single files (eg. /bin/ls) but cannot turn
> off the fs-verity system as a whole?
> 
> Also if RPM can update /bin/ls then surely an attacker who can widely
> compromise system files must also be able to update /bin/ls in the
> same way?

Once fs-verity is enabled for a given file (which, in the RPM case,
happens at package installation time), it cannot be disabled, and the
file becomes immutable. One can still rename() or unlink() it (and this
is indeed how rpm is able to replace files when upgrading packages),
but the actual contents cannot be altered.

Where is this useful? For example, fs-verity can help in the scenario
where an attacker has out-of-band access to the storage device (say,
they pull a hard drive from a colo'd server or a sdcard from an
embedded device, or they boot into a liveusb, or they access a VM image
directly from a host).

Let's say that happens, and the attacker changes a few blocks of
/bin/ls on the device to make it run nefarious code. When you boot your
system again, it would fail at exec() time because the Merkle tree
wouldn't match.

Let's say that instead the attacker mounts (or gains access to) your
filesystem, unlinks /bin/ls and replaces it wholesale with a new copy
(hence creating a new inode). The attacker doesn't have your signing
key, so they can't resign the file and enable fs-verity on it (they
could resign it with their own key, but unless they can then find a way
to load its cert into the kernel keyring it won't do much good). To
protect against this, you now have a few options:

- you could use a LSM to enforce that exec() can only happen on files
with valid fs-verity signatures; this would protect any binary
- you could use a launcher booted from secure storage (say, a dm-verity
volume, which could even be the initrd), and have this launcher perform
the verification; this of course only protects against binaries
executed from the launcher, but depending on your threat model it might
be enough

Like most security solutions, this isn’t a silver bullet and it’s not
something that in and of itself would necessarily prevent all possible
attacks. However, fs-verity can be a useful building block in a
defense-in-depth approach against specific attacks, depending on your
threat model.

Cheers
Davide

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Boris Burkov via devel]

You could reinstall the rpm, the fs will reclaim the verity metadata along with 
the rest of the old file.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Lennart Poettering]

On Do, 02.12.21 14:36, Ben Cotton (bcot...@redhat.com) wrote:

Hmm, so what I am really missing on the feature page: what's the
attack scenario here? Usually security features come with an attack
scenario they are supposed to address. But there's no discussion about
that.

This protects file contents, not the metada, right? So what about the
metadata? if I see a fs-verity enabled inode with libssl.so data in it
today, and it's a vulnerably version, and I make a hardlink to it, and
then it gets replaced by a fixed version (with a slightly updated
name) — how you intend to make sure, that i can't fool you into loading
my copy of the old file but under the new name?

What about signing anyway? What is this precisely signed with? The
keys for that when are they rolled over? Who manages those keys? The
tea for that, who's doing that in Fedora? Is there any protection
against downgrades between RPM package versions? Does this in any way
protect combinations of binaries/libraries? I mean, pretty much all
programs we ship consist of a large number of ELF objects, and you
probably need to sign the combination of them, but this model doesn't
look like it offers that at all? If a security bug is found in library
X, how is it taken out of equation? What would the workflow for that
look like on the Fedora side of things?

The security model seems really strange to me I must say: so much
infrastructure, and it doesn't protect at all how our programs are
usually combined?

Puzzled,

Lennart
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Matthew Miller]

On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote:
> Enable the use of fsverity for installed RPM files validation.

Can we use this to validate the install media at runtime rather than as a
separate boot step?

-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Kevin Kofler via devel]

Davide Cavalca via devel wrote:
> To clarify: RPM does support files validation, but fs-verity is more
> than just that. With RPM, the validation only happens on install time,
> and when one runs rpm -V manually. With fs-verity, the validation
> happens on-demand whenever a block of a file that originated from an
> RPM is accessed. This means, for example, that if an attacker replaces
> /bin/ls on disk with a compromised one, the next time it's read from
> disk (e.g. because you ran it) you will see a validation failure and
> the syscall will be blocked, preventing the compromised code from being
> executed.

This means that there is a performance cost in addition to the disk space 
cost (because something has to compute those checksums each time the file is 
acessed). It also means that it is harder for users to exercise their right 
to modify the Free Software (because replacing or patching RPM-installed 
binaries will lead to them failing to execute).

> About filesystem usage: unless you install rpm-plugin-fsverity (which
> is not and will not be installed by default), there is no disk space
> increase for verity-signed RPM packages. If you do install rpm-plugin-
> fsverity, some disk space will be used for the Merkle tree as described
> in the Change.

Since the change also adds to the metadata in the RPM, that means that it 
also increases the size of the RPMs. With keepcache=1, this also translates 
to increased disk space use. But even if the user does not keep cached RPMs, 
the download sizes will increase, which can cost time and for some users 
even money.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Kevin Kofler via devel]

Vitaly Zaitsev via devel wrote:
> -1. RPM already supports files validation and this feature will waste
> file system space.

I agree with you.

This is yet another "feature" that increases both the size of RPMs and, if 
enabled by default, the size of default installations. We need to stop 
tolerating creeping bloat!

So -1 to the Change proposal.

Kevin Kofler
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Stephen John Smoogen]

On Fri, 3 Dec 2021 at 17:09, Richard W.M. Jones  wrote:
>
> On Fri, Dec 03, 2021 at 06:08:49PM +, Davide Cavalca via devel wrote:
> > Broadly speaking, fs-verity makes it possible to ensure that files that
> > were installed via an RPM have not been modified. It is useful in
> > environments where an attacker might be able to modify system files
> > (say, replace /bin/ls with a compromised version) and you want to
> > protect against that. For example, consider an appliance-like system
> > placed in an untrusted location where you may not be able to control
> > who has physical access (this could be a server, but it could also be a
> > kiosk in an internet point or a school). In this scenario, fs-verity
> > can be one of the building blocks to ensure and maintain system trust.
>
> I'm unclear about the threat model - this is an attacker who is
> someone able to overwrite single files (eg. /bin/ls) but cannot turn
> off the fs-verity system as a whole?
>
> Also if RPM can update /bin/ls then surely an attacker who can widely
> compromise system files must also be able to update /bin/ls in the
> same way?
>

Or just pad /usr/bin/rpm with some null characters at the end to break
its signature and also stop updates from happening. [Or the fs-verity
daemon which will report that these problems are occuring. ]





-- 
Stephen J Smoogen.
Let us be kind to one another, for most of us are fighting a hard
battle. -- Ian MacClaren
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Dominik 'Rathann' Mierzejewski]

On Friday, 03 December 2021 at 18:49, Davide Cavalca via devel wrote:
[...]
> About filesystem usage: unless you install rpm-plugin-fsverity (which
> is not and will not be installed by default), there is no disk space
> increase for verity-signed RPM packages. If you do install rpm-plugin-
> fsverity, some disk space will be used for the Merkle tree as described
> in the Change.

How do I reclaim that space after removing rpm-plugin-fsverity?

Regards,
Dominik
-- 
Fedora   https://getfedora.org  |  RPM Fusion  http://rpmfusion.org
There should be a science of discontent. People need hard times and
oppression to develop psychic muscles.
-- from "Collected Sayings of Muad'Dib" by the Princess Irulan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Björn Persson]

> * at build time, we compute the Merkle tree for the files within a
> package, then sign it and ship it as part of the rpm metadata;

[...]

> Note that the Merkle tree
> is ''not'' shipped with the RPM itself (only its signature is)

In that case, "ship it" above should be changed to "ship the signature",
unless this is some distinction between "the RPM metadata" and "the RPM
itself".

If I enable FS-verity and later find that I need to patch a file to fix
some problem, how do I as the sysadmin tell Linux that this change is
authorized? Do I disable FS-verity for that specific file? Disable
FS-verity globally? Add my own key to the kernel's keyring? Build and
sign my own RPM package?

What prevents an attacker from doing the same?

Will files under /etc be covered, or will local configuration still be
possible?

Björn Persson


pgpbMIhTbl4rJ.pgp
Description: OpenPGP digital signatur
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Richard W.M. Jones]

On Fri, Dec 03, 2021 at 06:08:49PM +, Davide Cavalca via devel wrote:
> Broadly speaking, fs-verity makes it possible to ensure that files that
> were installed via an RPM have not been modified. It is useful in
> environments where an attacker might be able to modify system files
> (say, replace /bin/ls with a compromised version) and you want to
> protect against that. For example, consider an appliance-like system
> placed in an untrusted location where you may not be able to control
> who has physical access (this could be a server, but it could also be a
> kiosk in an internet point or a school). In this scenario, fs-verity
> can be one of the building blocks to ensure and maintain system trust.

I'm unclear about the threat model - this is an attacker who is
someone able to overwrite single files (eg. /bin/ls) but cannot turn
off the fs-verity system as a whole?

Also if RPM can update /bin/ls then surely an attacker who can widely
compromise system files must also be able to update /bin/ls in the
same way?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Josh Boyer]

On Fri, Dec 3, 2021 at 1:15 PM Davide Cavalca  wrote:
>
> On Thu, 2021-12-02 at 19:10 -0500, Josh Boyer wrote:
> > On Thu, Dec 2, 2021, 5:33 PM Davide Cavalca via devel
> >  wrote:
> >
> > > Correct, XFS doesn't support fs-verity at the moment (though it
> > > could
> > > be implemented if one wanted to).
> >
> > That means it would exclude Fedora Server and ELN as they are XFS
> > based.
>
> There isn't any impact on XFS-based editions by default. Building and
> signing RPMs with fs-verity works fine, as there's no filesystem
> dependency for that. Installing RPMs also works fine, with one caveat:
> if you're running a system with an unsupported filesystem and have rpm-
> plugin-fsverity (which is not installed by default), fs-verity will not
> be enabled for the installed files so there will be no verification
> (but the RPM installation itself should succeed).

Yes, thank you.  I meant it would render those Editions/projects
unable to use this Change.  It's not the end of the world, just an
observation.

josh
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Boris Burkov via devel]

Errors at installation time should be fully diagnosable, and even if the output 
today doesn't make it totally obvious what happened, it would be easy to fix in 
rpm.

The errors post-install are a bit trickier. Imagine you install your rpm, and 
kick off some long running daemon from it. A month later, a block gets 
corrupted in a way fs checksums don't catch (e.g. ext4, btrfs nodatasum, evil 
maid), and suddenly that daemon receives a SIGBUS and crashes. You would be 
able to see clearly that it was a verity issue in dmesg, but I don't think the 
binary could reasonably know what happened or write a meaningful log. In that 
sense, I think it's actually pretty similar to the experience if you have 
corruption in your disk and start getting btrfs checksum errors on a 
file--you'd have to look in dmesg to know why your file is broken.

The middle ground is when opening/exec-ing the file fails. In that case, you 
might get a sufficiently specific error code you could figure out it's verity, 
and the full error would be in dmesg as well.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Tom Seewald]

> I think there are two cases of interest:
> 
> 1) a file or signature in the rpm is corrupted, the signature doesn't have a 
> matching
> cert installed, etc...
> in this case, if the plugin is present, when you attempt to install the rpm 
> the verity
> enable ioctl will explicitly fail, and presumably so will the rpm install. 
> You can see
> most of the details for this sort of case here in the rpm plugin code:
> https://github.com/rpm-software-management/rpm/blob/master/plugins/fsveri
> 
> 2) after installation, a file from an fs-verity enabled rpm gets one or more 
> blocks
> corrupted
> The first read of a corrupted block from disk (the good uncorrupted page 
> might survive in
> page cache for a while) will result in EIO for read-like system calls and 
> SIGBUS if the
> file is mapped (executables, mmap).

Thanks for the information. So in the event of an error, it's expected that the 
user would be informed that the error was due to a rpm-fsverity check and they 
could potentially reinstall the rpm from a trusted source to resolve the 
problem? Of course there's the underlying issue of _why_ it happened, but from 
the standpoint of wanting an immediate path forward.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Boris Burkov via devel]

I omitted one more interesting case.

If the verity metadata (signature, root hash) is corrupted after installation 
but before the file is opened, then opening/exec-ing the file can fail. Also, 
if pages from a binary read in during the exec itself are corrupted, the system 
call itself could fail (rather than the process getting sigbus like for a 
random page during execution)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Davide Cavalca via devel]

On Thu, 2021-12-02 at 19:10 -0500, Josh Boyer wrote:
> On Thu, Dec 2, 2021, 5:33 PM Davide Cavalca via devel
>  wrote:
> 
> > Correct, XFS doesn't support fs-verity at the moment (though it
> > could
> > be implemented if one wanted to).
> 
> That means it would exclude Fedora Server and ELN as they are XFS
> based.

There isn't any impact on XFS-based editions by default. Building and
signing RPMs with fs-verity works fine, as there's no filesystem
dependency for that. Installing RPMs also works fine, with one caveat:
if you're running a system with an unsupported filesystem and have rpm-
plugin-fsverity (which is not installed by default), fs-verity will not
be enabled for the installed files so there will be no verification
(but the RPM installation itself should succeed).

Cheers
Davide
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Davide Cavalca via devel]

On Thu, 2021-12-02 at 20:05 -0500, Josh Boyer wrote:
> Yes, I saw that and I appreciate it.  That's a comparison between the
> two implementations.  I am asking about what benefits and use cases
> fs-verity solves in Fedora.  Right now, the change simply says:
> 
> "The main benefit is the ability to do block-level verification of
> RPM-installed files. In turn, this can be used to implement
> usecase-specific validation and verification policies depending on
> the
> environment requirements."
> 
> which is also largely true of IMA.  The IMA change went into more
> detailed use cases, which perhaps may have been it's downfall.  So
> can
> you describe what most Fedora users would use this for or benefit
> from
> it?  Or if "most users" is not an applicable qualifier, can you at
> least give some more detailed use cases that you would expect people
> to use it for?

Broadly speaking, fs-verity makes it possible to ensure that files that
were installed via an RPM have not been modified. It is useful in
environments where an attacker might be able to modify system files
(say, replace /bin/ls with a compromised version) and you want to
protect against that. For example, consider an appliance-like system
placed in an untrusted location where you may not be able to control
who has physical access (this could be a server, but it could also be a
kiosk in an internet point or a school). In this scenario, fs-verity
can be one of the building blocks to ensure and maintain system trust.

This Change is mostly about putting in place the necessary plumbing for
this to be at all possible. We'll try to expand the Change proposal and
flesh out potential usecases a bit more.

> OK.  I guess I was looking for some side-by-side data comparisons in
> the overhead between IMA metadata and fs-verity.  "1/127th of the
> original Merkel tree size" doesn't tell me much.
> 
> Are there some test runs with numbers to show before/after data for
> both the RPM size and installed FS usage?  Perhaps with an example
> install.  The IMA change attempted to document this and seeing a 1.1%
> average increase in RPM size was easier to understand.

We've done some empirical testing on this (showing neglibible
increases), but still need to gather more formal data. We'll try to
prioritize that and add it to the Change once it's available.

Cheers
Davide
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Boris Burkov via devel]

I think there are two cases of interest:

1) a file or signature in the rpm is corrupted, the signature doesn't have a 
matching cert installed, etc...
in this case, if the plugin is present, when you attempt to install the rpm the 
verity enable ioctl will explicitly fail, and presumably so will the rpm 
install. You can see most of the details for this sort of case here in the rpm 
plugin code: 
https://github.com/rpm-software-management/rpm/blob/master/plugins/fsverity.c#L114.

2) after installation, a file from an fs-verity enabled rpm gets one or more 
blocks corrupted
The first read of a corrupted block from disk (the good uncorrupted page might 
survive in page cache for a while) will result in EIO for read-like system 
calls and SIGBUS if the file is mapped (executables, mmap).
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Davide Cavalca via devel]

On Fri, 2021-12-03 at 12:21 +0100, Vitaly Zaitsev via devel wrote:
> On 02/12/2021 20:36, Ben Cotton wrote:
> > Enable the use of fsverity for installed RPM files validation.
> 
> -1. RPM already supports files validation and this feature will waste
> file system space.

To clarify: RPM does support files validation, but fs-verity is more
than just that. With RPM, the validation only happens on install time,
and when one runs rpm -V manually. With fs-verity, the validation
happens on-demand whenever a block of a file that originated from an
RPM is accessed. This means, for example, that if an attacker replaces
/bin/ls on disk with a compromised one, the next time it's read from
disk (e.g. because you ran it) you will see a validation failure and
the syscall will be blocked, preventing the compromised code from being
executed.

About filesystem usage: unless you install rpm-plugin-fsverity (which
is not and will not be installed by default), there is no disk space
increase for verity-signed RPM packages. If you do install rpm-plugin-
fsverity, some disk space will be used for the Merkle tree as described
in the Change.

Cheers
Davide
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Boris Burkov via devel]

The top-level hash is calculated for each file, then that hash is signed with 
the inputted rsa key pair and the signed hash is appended to the array of 
signed hashes in the rpm metadata. I am guessing the way we worded the proposal 
is a little unclear because we call it "the" signature when it's one rpm 
metadata item that's an array of the signatures.

fs-verity the kernel feature operates on a per-file basis, and since the 
ultimate goal is to deliver fs-verity enabled files on the installer's system, 
we need each file's signature in the rpm. At install, we call the fs-verity 
enable ioctl for each file, passing in its signature to make use of the kernel 
authentication functionality.

I'm happy to explain the exact flow at signing and at install in more detail if 
that would be helpful.

Boris
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Tom Seewald]

Perhaps I glossed over it in the description, but what is the expected user 
experience in the event of a RPM fs-verity mismatch/error?
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Vitaly Zaitsev via devel]

On 02/12/2021 20:36, Ben Cotton wrote:

Enable the use of fsverity for installed RPM files validation.


-1. RPM already supports files validation and this feature will waste 
file system space.


--
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Zbigniew Jędrzejewski-Szmek]

> The signature size is roughly proportional to the number of files in
> the package.
Can you explain how the signature is performed? I assume that the verity
top-level hash is calculated for each file and then … ?
And if you have all the per-file hashes, why not do one more level of
Merkle and calculate a single hash for all the files in the rpm and sign
that (so that the signature overhead is of fixed size)? 

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Josh Boyer]

On Thu, Dec 2, 2021 at 7:27 PM Michel Alexandre Salim
 wrote:
>
> Hello,
>
> On Thu, Dec 02, 2021 at 07:10:32PM -0500, Josh Boyer wrote:
> > On Thu, Dec 2, 2021, 5:33 PM Davide Cavalca via devel <
> > devel@lists.fedoraproject.org> wrote:
> >
> > > On Thu, 2021-12-02 at 13:09 -0800, Kevin Fenzi wrote:
> > > > On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote:
> > > > ...snip...
> > > > >
> > > > > In the context of rpm, there are two parts to this:
> > > > > * at build time, we compute the Merkle tree for the files within a
> > > > > package, then sign it and ship it as part of the rpm metadata;
> > > >
> > > > This is some kind of seperate signing that happens at build time?
> > > >
> > > > Or it's added to the rpm metadata and covered by the normal package
> > > > signing if/when the package is signed?
> > >
> > > As part of the signing flow (e.g. via rpmsign), the Merkle tree is
> > > generated and a signature is computed from it, which is then added to
> > > the rpm metadata.
> > >
> >
> > IMA received significant pushback on the impact to RPMs and signing
> > implications in Fedora.  How does fs-verity compare here?
> We wrote up a comparison here: 
> https://fedoraproject.org/wiki/Changes/FsVerityRPM#Relationship_with_IMA

Yes, I saw that and I appreciate it.  That's a comparison between the
two implementations.  I am asking about what benefits and use cases
fs-verity solves in Fedora.  Right now, the change simply says:

"The main benefit is the ability to do block-level verification of
RPM-installed files. In turn, this can be used to implement
usecase-specific validation and verification policies depending on the
environment requirements."

which is also largely true of IMA.  The IMA change went into more
detailed use cases, which perhaps may have been it's downfall.  So can
you describe what most Fedora users would use this for or benefit from
it?  Or if "most users" is not an applicable qualifier, can you at
least give some more detailed use cases that you would expect people
to use it for?

> > Alternatively/additionally, why is fs-verity worth the hit for Fedora where
> > IMA was not.
>
> The hit is much smaller; per 
> https://fedoraproject.org/wiki/Changes/FsVerityRPM#Merkle_tree_cost
> - if the plugin is installed, the Merkle tree is stored but it's 1/127th
>   the original file size
> - the RPM only ships the signature, not the tree itself; per
> https://fedoraproject.org/wiki/Changes/FsVerityRPM#Signature_overhead_cost
>   in practice we see a minimal to no increase in the size of the RPM
>
> So as proposed in this Change, users can opt in by installing the
> plugin, otherwise they will be mostly unimpacted.

OK.  I guess I was looking for some side-by-side data comparisons in
the overhead between IMA metadata and fs-verity.  "1/127th of the
original Merkel tree size" doesn't tell me much.

Are there some test runs with numbers to show before/after data for
both the RPM size and installed FS usage?  Perhaps with an example
install.  The IMA change attempted to document this and seeing a 1.1%
average increase in RPM size was easier to understand.

josh

> As discussed in the write-up - IMA does have a richer policy system, and
> could potentially be integrated (so we use IMA but with a fsverity
> backend) to get the best of both worlds.
>
> Best regards,
>
> --
> Michel Alexandre Salim
> profile: https://keyoxide.org/mic...@michel-slm.name
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Michael Cronenworth]

On 12/2/21 6:46 PM, Michael Cronenworth wrote:


Could this be directly added to rpm instead of an external tool set? 


I see you did. It helps to read the Change link... Sorry. :)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Michael Cronenworth]

On 12/2/21 4:32 PM, Davide Cavalca via devel wrote:

There's support in robosignatory to ask to sign files (used for the
short lived IMA stuff), but I suspect it would need a new ability for
this.

Finally who is going to write this? Change owners?
Or do you expect robosignatory maintainers to do so?

Thanks for clarifying! Yes, I think robosignatory is likely what we
want here. We (the Change owners) expect to do the work, though we'll
likely need some advice/help around code review, testing and
deployment.



Could this be directly added to rpm instead of an external tool set?

rpm --addsign ./$RPM
to
rpm --addsign --addfsverity ./$RPM

I'm thinking about those who might also want to use it... end-users and third-party 
repos such as RPMFusion.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Michel Alexandre Salim]

Hello,

On Thu, Dec 02, 2021 at 07:10:32PM -0500, Josh Boyer wrote:
> On Thu, Dec 2, 2021, 5:33 PM Davide Cavalca via devel <
> devel@lists.fedoraproject.org> wrote:
> 
> > On Thu, 2021-12-02 at 13:09 -0800, Kevin Fenzi wrote:
> > > On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote:
> > > ...snip...
> > > >
> > > > In the context of rpm, there are two parts to this:
> > > > * at build time, we compute the Merkle tree for the files within a
> > > > package, then sign it and ship it as part of the rpm metadata;
> > >
> > > This is some kind of seperate signing that happens at build time?
> > >
> > > Or it's added to the rpm metadata and covered by the normal package
> > > signing if/when the package is signed?
> >
> > As part of the signing flow (e.g. via rpmsign), the Merkle tree is
> > generated and a signature is computed from it, which is then added to
> > the rpm metadata.
> >
> 
> IMA received significant pushback on the impact to RPMs and signing
> implications in Fedora.  How does fs-verity compare here?
We wrote up a comparison here: 
https://fedoraproject.org/wiki/Changes/FsVerityRPM#Relationship_with_IMA

> Alternatively/additionally, why is fs-verity worth the hit for Fedora where
> IMA was not.

The hit is much smaller; per 
https://fedoraproject.org/wiki/Changes/FsVerityRPM#Merkle_tree_cost
- if the plugin is installed, the Merkle tree is stored but it's 1/127th
  the original file size
- the RPM only ships the signature, not the tree itself; per
https://fedoraproject.org/wiki/Changes/FsVerityRPM#Signature_overhead_cost
  in practice we see a minimal to no increase in the size of the RPM

So as proposed in this Change, users can opt in by installing the
plugin, otherwise they will be mostly unimpacted.

As discussed in the write-up - IMA does have a richer policy system, and
could potentially be integrated (so we use IMA but with a fsverity
backend) to get the best of both worlds.

Best regards,

-- 
Michel Alexandre Salim
profile: https://keyoxide.org/mic...@michel-slm.name


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Josh Boyer]

On Thu, Dec 2, 2021, 5:33 PM Davide Cavalca via devel <
devel@lists.fedoraproject.org> wrote:

> On Thu, 2021-12-02 at 13:09 -0800, Kevin Fenzi wrote:
> > On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote:
> > ...snip...
> > >
> > > In the context of rpm, there are two parts to this:
> > > * at build time, we compute the Merkle tree for the files within a
> > > package, then sign it and ship it as part of the rpm metadata;
> >
> > This is some kind of seperate signing that happens at build time?
> >
> > Or it's added to the rpm metadata and covered by the normal package
> > signing if/when the package is signed?
>
> As part of the signing flow (e.g. via rpmsign), the Merkle tree is
> generated and a signature is computed from it, which is then added to
> the rpm metadata.
>

IMA received significant pushback on the impact to RPMs and signing
implications in Fedora.  How does fs-verity compare here?
Alternatively/additionally, why is fs-verity worth the hit for Fedora where
IMA was not.
I'm not looking to create a false equivalence or compare apples to oranges,
but I want to better understand why this is a more acceptable approach for
Fedora.

As an aside, a few people have asked why RHEL didn't get IMA into Fedora
first.  Frankly, we tried and got feedback that it didn't apply to Fedora's
needs and use cases.  As of right now, we have no plans to force something
into Fedora that the community doesn't want.  If fs-verity is more
acceptable or is described in a way that is better understood with enough
similarities to IMA, we may try again.

> > * at run time, if the fsverity rpm plugin is enabled, rpm will
> > > install
> > > the fsverity signature key and enable fsverity on files that are
> > > installed.
> >
> > Is this signature key the fedora rpm package signing key?
> > Or some seperate fsverity key?
>
> fs-verity needs a RSA key/cert pair for file signing at package
> signature time. At package install time, the cert needs to be loaded in
> the appropriate kernel keyring. We've always used a dedicated keypair
> during testing -- I'm not actually sure if the package signing key
> could be reused for this, as it's a GPG key, but this is something we
> should follow up on.
>
> > > fs-verity works by using a Merkle tree to generate a checksum for
> > > every data block in the system, and reads will fail if a single
> > > data
> >
> > every data block? or every packaged in rpms data block?
>
> fs-verity only operates on files where it has been enabled via its
> ioctl (which, if you install the RPM plugin, is taken care of by RPM on
> your behalf). For those, fs-verity will checksum every data block
> whenever it's accessed and validate it still matches.
>
> > > block read fails it’s checksum. The signature of the the file is
> > > validated against a public key loaded into the kernel keyring.
> > > Because
> > > fsverity operates on block reads, its runtime cost is small (as it
> > > only needs to verify the block that is being accessed), and it can
> > > protect from alterations at any point in time.
> >
> > Is this via dm_verity kernel module? Or thats a completely seperate
> > thing?
>
> It's somewhat inspired by dm-verity, but it's a separate
> implementation, the only shared logic is the hash computation code in
> the kernel.
>
> > > == Scope ==
> > >
> > > * Proposal owners
> > > ** btrfs kernel enablement work
> > > ([
> > >
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=146054090b0859b28fc39015c7704ccc3c3a347f
> > > landed in 5.15]); see this
> > > [
> > >
> https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-btrfs/
> > > blog post] for more details
> >
> > What does this mean for all the other filesystems? They would just be
> > slower since btrfs is computing the trees as part of it's normal
> > checksumming?
>
> fs-verity requires support in the underlying filesystem. If you're
> using a filesystem that doesn't support it and attempt to enable fs-
> verity on a file, the ioctl will fail. Note that this is only a concern
> at runtime, not at build time.
>
> > > ** koji integration: koji will need to add the fs-verity metadata
> > > to
> > > packages when signing them
> >
> > Well, koji doesn't sign packages. robosignatory listens for messages
> > on
> > the message bus for koji tagging and based on it's config, it asks
> > sigul
> > to sign rpms and import the signatures into koji.
> >
> > There's support in robosignatory to ask to sign files (used for the
> > short lived IMA stuff), but I suspect it would need a new ability for
> > this.
> >
> > Finally who is going to write this? Change owners?
> > Or do you expect robosignatory maintainers to do so?
>
> Thanks for clarifying! Yes, I think robosignatory is likely what we
> want here. We (the Change owners) expect to do the work, though we'll
> likely need some advice/help around code review, testing and
> deployment.
>
> > > * fs-verity requires filesystem support; currently 

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Davide Cavalca via devel]

On Thu, 2021-12-02 at 13:09 -0800, Kevin Fenzi wrote:
> On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote:
> ...snip...
> > 
> > In the context of rpm, there are two parts to this:
> > * at build time, we compute the Merkle tree for the files within a
> > package, then sign it and ship it as part of the rpm metadata;
> 
> This is some kind of seperate signing that happens at build time?
> 
> Or it's added to the rpm metadata and covered by the normal package
> signing if/when the package is signed?

As part of the signing flow (e.g. via rpmsign), the Merkle tree is
generated and a signature is computed from it, which is then added to
the rpm metadata.

> > * at run time, if the fsverity rpm plugin is enabled, rpm will
> > install
> > the fsverity signature key and enable fsverity on files that are
> > installed.
> 
> Is this signature key the fedora rpm package signing key? 
> Or some seperate fsverity key?

fs-verity needs a RSA key/cert pair for file signing at package
signature time. At package install time, the cert needs to be loaded in
the appropriate kernel keyring. We've always used a dedicated keypair
during testing -- I'm not actually sure if the package signing key
could be reused for this, as it's a GPG key, but this is something we
should follow up on.

> > fs-verity works by using a Merkle tree to generate a checksum for
> > every data block in the system, and reads will fail if a single
> > data
> 
> every data block? or every packaged in rpms data block?

fs-verity only operates on files where it has been enabled via its
ioctl (which, if you install the RPM plugin, is taken care of by RPM on
your behalf). For those, fs-verity will checksum every data block
whenever it's accessed and validate it still matches.

> > block read fails it’s checksum. The signature of the the file is
> > validated against a public key loaded into the kernel keyring.
> > Because
> > fsverity operates on block reads, its runtime cost is small (as it
> > only needs to verify the block that is being accessed), and it can
> > protect from alterations at any point in time.
> 
> Is this via dm_verity kernel module? Or thats a completely seperate
> thing?

It's somewhat inspired by dm-verity, but it's a separate
implementation, the only shared logic is the hash computation code in
the kernel.

> > == Scope ==
> > 
> > * Proposal owners
> > ** btrfs kernel enablement work
> > ([
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=146054090b0859b28fc39015c7704ccc3c3a347f
> > landed in 5.15]); see this
> > [
> > https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-btrfs/
> > blog post] for more details
> 
> What does this mean for all the other filesystems? They would just be
> slower since btrfs is computing the trees as part of it's normal
> checksumming?

fs-verity requires support in the underlying filesystem. If you're
using a filesystem that doesn't support it and attempt to enable fs-
verity on a file, the ioctl will fail. Note that this is only a concern
at runtime, not at build time.

> > ** koji integration: koji will need to add the fs-verity metadata
> > to
> > packages when signing them
> 
> Well, koji doesn't sign packages. robosignatory listens for messages
> on
> the message bus for koji tagging and based on it's config, it asks
> sigul
> to sign rpms and import the signatures into koji. 
> 
> There's support in robosignatory to ask to sign files (used for the
> short lived IMA stuff), but I suspect it would need a new ability for
> this. 
> 
> Finally who is going to write this? Change owners?
> Or do you expect robosignatory maintainers to do so?

Thanks for clarifying! Yes, I think robosignatory is likely what we
want here. We (the Change owners) expect to do the work, though we'll
likely need some advice/help around code review, testing and
deployment.

> > * fs-verity requires filesystem support; currently support for ext4
> > and f2fs is already available; support for btrfs landed in 5.15
> 
> No xfs support?

Correct, XFS doesn't support fs-verity at the moment (though it could
be implemented if one wanted to).

Cheers
Davide
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Kevin Fenzi]

On Thu, Dec 02, 2021 at 02:36:51PM -0500, Ben Cotton wrote:
...snip...
> 
> In the context of rpm, there are two parts to this:
> * at build time, we compute the Merkle tree for the files within a
> package, then sign it and ship it as part of the rpm metadata;

This is some kind of seperate signing that happens at build time?

Or it's added to the rpm metadata and covered by the normal package
signing if/when the package is signed?

> * at run time, if the fsverity rpm plugin is enabled, rpm will install
> the fsverity signature key and enable fsverity on files that are
> installed.

Is this signature key the fedora rpm package signing key? 
Or some seperate fsverity key?

...snip...

> fs-verity works by using a Merkle tree to generate a checksum for
> every data block in the system, and reads will fail if a single data

every data block? or every packaged in rpms data block?

> block read fails it’s checksum. The signature of the the file is
> validated against a public key loaded into the kernel keyring. Because
> fsverity operates on block reads, its runtime cost is small (as it
> only needs to verify the block that is being accessed), and it can
> protect from alterations at any point in time.

Is this via dm_verity kernel module? Or thats a completely seperate
thing?

...snip...

> == Scope ==
> 
> * Proposal owners
> ** btrfs kernel enablement work
> ([https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=146054090b0859b28fc39015c7704ccc3c3a347f
> landed in 5.15]); see this
> [https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-btrfs/
> blog post] for more details

What does this mean for all the other filesystems? They would just be
slower since btrfs is computing the trees as part of it's normal
checksumming?

> ** koji integration: koji will need to add the fs-verity metadata to
> packages when signing them

Well, koji doesn't sign packages. robosignatory listens for messages on
the message bus for koji tagging and based on it's config, it asks sigul
to sign rpms and import the signatures into koji. 

There's support in robosignatory to ask to sign files (used for the
short lived IMA stuff), but I suspect it would need a new ability for
this. 

Finally who is going to write this? Change owners?
Or do you expect robosignatory maintainers to do so?

> * Other developers:
> ** deploy the koji integration changes to production
> * Release engineering: tbd
> * Policies and guidelines: N/A
> * Trademark approval: N/A
> 
> == Upgrade/compatibility impact ==
> 
> None
> 
> == How to test ==
> 
> Install the fs-verity RPM plugin to validate package contents:
> 
> $ sudo dnf install rpm-plugin-fsverity
> Note that this will only be useful if the packages being installed
> contain the appropriate fs-verity metadata (which, for Fedora upstream
> packages, requires Koji integration that is part of this Change).
> However, you should still be able to test this if you locally sign a
> package with rpmsign --addverity.
> 
> == User experience ==
> 
> This Change is fully transparent and there is no user impact by
> default. If the user chooses to enable the fs-verity RPM plugin, they
> can then leverage the additional verification features provided by
> fs-verity.
> 
> == Dependencies ==
> 
> * fs-verity support is available in RPM as of 4.17, which is available
> as of Fedora 35 and is already enabled in rpm-4.17.0-0.rc1.0.fc36
> * CONFIG_FS_VERITY in the kernel config; this is already enabled
> * fs-verity requires filesystem support; currently support for ext4
> and f2fs is already available; support for btrfs landed in 5.15

No xfs support?

Thanks in advance for answers. 

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Davide Cavalca via devel]

On Thu, 2021-12-02 at 15:08 -0500, Frank Ch. Eigler wrote:
> 
> > === Relationship with IMA ===
> > 
> > [https://sourceforge.net/p/linux-ima/wiki/Home/ IMA] is another
> > technology meant to provide detection of file alterations. IMA and
> > fsverity operate very differently, and are somewhat complementary.
> > [...]
> 
> Do these two systems use the same per-file signature metadata in the
> RPMs?

Both fs-verity and IMA use file signatures, but they each have their
own dedicated flags and signing flows in RPM (e.g. see
https://github.com/rpm-software-management/rpm/blob/4afe2d14d33db82ccb41c0a8d5eb1a4db90762fc/rpmsign.c
for the signing implementation). The signatures themselves are not
interchangeable -- fs-verity's signature is based off the Merkle tree
(which itself is block-based), while IMA measures the file as a whole.

Cheers
Davide
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Frank Ch. Eigler]

> === Relationship with IMA ===
>
> [https://sourceforge.net/p/linux-ima/wiki/Home/ IMA] is another
> technology meant to provide detection of file alterations. IMA and
> fsverity operate very differently, and are somewhat complementary.
> [...]

Do these two systems use the same per-file signature metadata in the RPMs?

- FChE
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/FsVerityRPM

== Summary ==

Enable the use of fsverity for installed RPM files validation.

== Owners ==

* Name: [[User:Dcavalca|Davide Cavalca]], [[User:Borisb|Boris
Burkov]], [[User:Filbranden|Filipe Brandenburger]],
[[User:Salimma|Michel Alexandre Salim]], [[User:Malmond|Matthew
Almond]]
* Email: dcava...@fb.com, bor...@fb.com, filbran...@fb.com,
mic...@fb.com, malm...@fb.com


== Detailed description ==

fs-verity is a [https://www.kernel.org/doc/html/latest/filesystems/fsverity.html
Linux kernel feature] that does transparent on-demand
integrity/authenticity verification of the contents of read-only
files, using a hidden Merkle tree (hash tree) associated with the
file. The mechanism is similar to dm-verity, but implemented at the
file level rather than at the block device level.

When fsverity is enabled for a file, the kernel reads every block and
generates a hash tree for it, which is stored within the filesystem.
On subsequent reads, the kernel computes the block hash and compares
it with the one stored in the tree, protecting against alterations and
corruption. Because this happens at the filesystem data block read
layer, it encompasses all file operations (open,
mmap,exec, etc.).

In the context of rpm, there are two parts to this:
* at build time, we compute the Merkle tree for the files within a
package, then sign it and ship it as part of the rpm metadata;
* at run time, if the fsverity rpm plugin is enabled, rpm will install
the fsverity signature key and enable fsverity on files that are
installed.


This proposal is primarily concerned with the first part, which will
make it possible for users to leverage fs-verity for RPM if they so
desire. Specifically, installing and enabling the fs-verity rpm plugin
by default is explicitly considered out of scope here.

=== Caveats ===

 Merkle tree cost 

The Merkle tree used by fsverity needs to be generated (once at build
time, once when the package is installed) and stored on disk. The
generation process involves reading all blocks and computing the hash,
which has a non-trivial cost; however, it does not appear to
meaningfully slow down package installs during empirical testing. Once
generated, the Merkle tree will use up some disk space for its storage
(about 1/127th of the original file size). Note that the Merkle tree
is ''not'' shipped with the RPM itself (only its signature is) and is
only generated and stored at install time if the fsverity rpm plugin
is enabled. Hence, there is no cost (neither in generation time nor in
disk space usage) if the plugin is disabled.

 Signature overhead cost 

To leverage fsverity every rpm needs to include the hash signature as
part of its metadata, which will increase its size. The signature size
is roughly proportional to the number of files in the package. From
empirical testing, in the vast majority of cases we expect to see
minimal to no size increase thanks to RPM header packing.

=== Relationship with IMA ===

[https://sourceforge.net/p/linux-ima/wiki/Home/ IMA] is another
technology meant to provide detection of file alterations. IMA and
fsverity operate very differently, and are somewhat complementary.

fs-verity works by using a Merkle tree to generate a checksum for
every data block in the system, and reads will fail if a single data
block read fails it’s checksum. The signature of the the file is
validated against a public key loaded into the kernel keyring. Because
fsverity operates on block reads, its runtime cost is small (as it
only needs to verify the block that is being accessed), and it can
protect from alterations at any point in time.

IMA works by measuring a file as a whole and comparing its signature
whenever it’s read of executed. It has a higher runtime cost than
fsverity (as it needs to verify the whole file at once) and it cannot
detect subsequent alternations. IMA provides a much more rich and
complex policy system, allowing one to define system-wide policies
around trusted files that tie into LSMs such as SELinux.

IMA and fsverity could potentially be integrated (meaning, an fsverity
backend for IMA could be implemented to leverage its policy controls),
but this is not currently planned or being worked on.

=== Relationship with native checksums ===

By default, btrfs already checksums each file extent, which could
potentially be leveraged to implement a HMAC solution. This currently
exists as a 
[https://lore.kernel.org/linux-btrfs/sn4pr0401mb3598ddef9bf9baca71a1041d9b...@sn4pr0401mb3598.namprd04.prod.outlook.com/T/
patch series] but it hasn’t been merged yet. Similarly to IMA, we see
this approach as complementary to fs-verity. The
[https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-btrfs/
blog post] goes into more details of the tradeoffs involved.

== Feedback ==

Pending the devel discussion

== Benefit to Fedora ==

The main benefit is the ability to do block-level verification of

F36 Change: Enable fs-verity in RPM (System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/FsVerityRPM

== Summary ==

Enable the use of fsverity for installed RPM files validation.

== Owners ==

* Name: [[User:Dcavalca|Davide Cavalca]], [[User:Borisb|Boris
Burkov]], [[User:Filbranden|Filipe Brandenburger]],
[[User:Salimma|Michel Alexandre Salim]], [[User:Malmond|Matthew
Almond]]
* Email: dcava...@fb.com, bor...@fb.com, filbran...@fb.com,
mic...@fb.com, malm...@fb.com


== Detailed description ==

fs-verity is a [https://www.kernel.org/doc/html/latest/filesystems/fsverity.html
Linux kernel feature] that does transparent on-demand
integrity/authenticity verification of the contents of read-only
files, using a hidden Merkle tree (hash tree) associated with the
file. The mechanism is similar to dm-verity, but implemented at the
file level rather than at the block device level.

When fsverity is enabled for a file, the kernel reads every block and
generates a hash tree for it, which is stored within the filesystem.
On subsequent reads, the kernel computes the block hash and compares
it with the one stored in the tree, protecting against alterations and
corruption. Because this happens at the filesystem data block read
layer, it encompasses all file operations (open,
mmap,exec, etc.).

In the context of rpm, there are two parts to this:
* at build time, we compute the Merkle tree for the files within a
package, then sign it and ship it as part of the rpm metadata;
* at run time, if the fsverity rpm plugin is enabled, rpm will install
the fsverity signature key and enable fsverity on files that are
installed.


This proposal is primarily concerned with the first part, which will
make it possible for users to leverage fs-verity for RPM if they so
desire. Specifically, installing and enabling the fs-verity rpm plugin
by default is explicitly considered out of scope here.

=== Caveats ===

 Merkle tree cost 

The Merkle tree used by fsverity needs to be generated (once at build
time, once when the package is installed) and stored on disk. The
generation process involves reading all blocks and computing the hash,
which has a non-trivial cost; however, it does not appear to
meaningfully slow down package installs during empirical testing. Once
generated, the Merkle tree will use up some disk space for its storage
(about 1/127th of the original file size). Note that the Merkle tree
is ''not'' shipped with the RPM itself (only its signature is) and is
only generated and stored at install time if the fsverity rpm plugin
is enabled. Hence, there is no cost (neither in generation time nor in
disk space usage) if the plugin is disabled.

 Signature overhead cost 

To leverage fsverity every rpm needs to include the hash signature as
part of its metadata, which will increase its size. The signature size
is roughly proportional to the number of files in the package. From
empirical testing, in the vast majority of cases we expect to see
minimal to no size increase thanks to RPM header packing.

=== Relationship with IMA ===

[https://sourceforge.net/p/linux-ima/wiki/Home/ IMA] is another
technology meant to provide detection of file alterations. IMA and
fsverity operate very differently, and are somewhat complementary.

fs-verity works by using a Merkle tree to generate a checksum for
every data block in the system, and reads will fail if a single data
block read fails it’s checksum. The signature of the the file is
validated against a public key loaded into the kernel keyring. Because
fsverity operates on block reads, its runtime cost is small (as it
only needs to verify the block that is being accessed), and it can
protect from alterations at any point in time.

IMA works by measuring a file as a whole and comparing its signature
whenever it’s read of executed. It has a higher runtime cost than
fsverity (as it needs to verify the whole file at once) and it cannot
detect subsequent alternations. IMA provides a much more rich and
complex policy system, allowing one to define system-wide policies
around trusted files that tie into LSMs such as SELinux.

IMA and fsverity could potentially be integrated (meaning, an fsverity
backend for IMA could be implemented to leverage its policy controls),
but this is not currently planned or being worked on.

=== Relationship with native checksums ===

By default, btrfs already checksums each file extent, which could
potentially be leveraged to implement a HMAC solution. This currently
exists as a 
[https://lore.kernel.org/linux-btrfs/sn4pr0401mb3598ddef9bf9baca71a1041d9b...@sn4pr0401mb3598.namprd04.prod.outlook.com/T/
patch series] but it hasn’t been merged yet. Similarly to IMA, we see
this approach as complementary to fs-verity. The
[https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-btrfs/
blog post] goes into more details of the tradeoffs involved.

== Feedback ==

Pending the devel discussion

== Benefit to Fedora ==

The main benefit is the ability to do block-level verification of