Re: Malicious fs images was Re: ext4 regression in v5.9-rc2 from e7bfb5c9bb3d on ro fs with overlapped bitmaps
On Tue, Jan 12, 2021 at 11:28:40PM +0100, Pavel Machek wrote: > > This thread suggested that kernel is _not_ supposed to be robust > against corrupt filesystems (because fsck is not integrated in > kernel). Which was news to me (and I'm not the person that needs > warning in execve documentation). Define "supposed to be". In the ideal world, the kernel should be robust against corrupt file systems. In the ideal world, hard drives would never die, and memory bits would never get flipped due to cosmic rays, and so Intel would be correct that consumers don't need ECC memory. In the ideal world, drivers would never make mistakes, and so seat belts would be completely unnecessasry. Unfortunately, we live in the real world. And so there is risk associated with using various technologies, and that risk is not a binary 0% vs 100%. In my mind, assuming that file systems are robust against maliciously created images is much like driving around without a seat belt. There are plenty of people who drive without seat belts for years, and they haven't been killed yet. But it's not something *I* would do. But hey, I also spent extra money to buy a personal desktop computer with ECC memory, and I don't go bicycling without a helment, either. You're free to decide what *you* want to do. But I wouldn't take a random file system image from the Net and mount it without running fsck on the darned thing first. As far as secure boot is concerned, do you know how many zero days are out there with Windows machines? I'm pretty sure there are vast numbers. There are even more systems where the system adminsitrators haven't bothered to install latest updates, as well. > And if we have filesystems where corrupt image is known to allow > arbitrary code execution, we need to Define *known*. If you look at the syzbot dashboard, there are hundreds of reproducers where root is able to crash a Linux server. Some number of them will almost certainly be exploitable. The problem is it takes a huge amount of time to analyze them, and Syzbot's file system fuzzers are positively developer-hostile to root cause. So usually I find and fix ext4 file system fuzzing reports via reports from other fuzzers, and then eventually syzbot realizes that the reproducer no longer works, and it gets closed out. I'd certainly be willing to bet a beer or two that there are vulnerabilities in VFAT, but do I *know* that to be the case? No, because I don't have the time to take a syzbot report relating to a file system and root cause it. The time to extract out the image, and then figure out how to get a simple reproducer (as opposed to the mess that a syzbot reproducer that might be randomly toggling a network bridge interface on one thread while messing with a file system image on another) is easily 10-20 times the effort to actually *fix* the bug once we're able to come up with a trivial reproducer with a file system image which is a separate file so we can analyze it using file system debugging tools. I'm also *quite* confident that the NSA, KGB, and other state intelligence agencies have dozens of zero days for Windows, Linux, etc. We just don't know in what subsystem they are in, so we can't just "disable them when secure boot is enabled". - Ted
Re: Malicious fs images was Re: ext4 regression in v5.9-rc2 from e7bfb5c9bb3d on ro fs with overlapped bitmaps
Hi!
> > People want to use USB sticks from time to time. And while I
> > understand XFS is so complex it is unsuitable for such use, I'd still
> > expect bugs to be fixed there.
> >
> > I hope VFAT to be safe to mount, because that is very common on USB.
> >
> > I also hope ext2/3/4 is safe in that regard.
>
> Ext4 will fix file system fuzzing attack bugs on a best efforts basis.
> That is, when I have time, I've been known to stay up late to bugs
> reported by fuzzers. I hope ext4 is safe, but I'm not going to make
> any guarantees that it is Bug-Free(tm). If you want to trust it in
> that way, you do so at your risk.
Good.
> > Anyway it would be nice to have documentation explaining this. If I'm
> > wrong about VFAT being safe, it would be good to know, and I guess
> > many will be surprised that XFS is using different rules.
>
> Using USB sticks is fine, so long as you trust the provenance of the
> drive. If you take a random USB stick that is handed to you by
Well... That makes passing data between Windows and Linux machines
using USB stick "interesting", right?
> someone whom you don't trust implicitly, or worse, that you picked up
> abandoned on the sidewalk, there have been plenty of articles which
> describe why this is a REALLY BAD IDEA, and even if you ignore
> OS-level vuleranbilities, there are also firwmare and hardware based
> vulerabilities that would put your computer at risk. See [2] and
> [3]
I know, but bear with me.
> As far as documentation is concerned, how far should we go? Should
> there be a warning in the execve(2) system call man page that you
> shouldn't download random binaries from the network and execute them? :-)
No need to pull straw men for me.
This thread suggested that kernel is _not_ supposed to be robust
against corrupt filesystems (because fsck is not integrated in
kernel). Which was news to me (and I'm not the person that needs
warning in execve documentation).
I'd certainly like to hear that VFAT and EXT4 _is_ supposed to be
robust in that way.
And if we have filesystems where corrupt image is known to allow
arbitrary code execution, we need to
a) document that.
b) disable them when secure boot is enabled.
Because with secure boot, we are supposed to be secure against attacks
from root, and root can prepare malicious filesystems. ("The problem,
simply put, is this: the objective of secure boot is to prevent the
system from running any unsigned code in a privileged mode. So, if one
boots a Linux system that, in turn, gives access to the machine to
untrusted code, the entire purpose has been defeated. The consequences
could hurt both locally (bad code could take control of the machine)
and globally (the signing key used to boot Linux could be revoked), so
it is an outcome that is worth avoiding. Doing so, however, requires
placing limitations in the kernel so that not even root can circumvent
the secure boot chain of trust." -- https://lwn.net/Articles/514985/
).
Best regards,
Pavel
--
http://www.livejournal.com/~pavelmachek
signature.asc
Description: PGP signature
Re: Malicious fs images was Re: ext4 regression in v5.9-rc2 from e7bfb5c9bb3d on ro fs with overlapped bitmaps
On Sun, Jan 10, 2021 at 07:41:02PM +0100, Pavel Machek wrote: > > >From our perspective (and Darrick and I discussed this on this week's > > ext4 video conference, so it represents the ext4 and xfs maintainer's > > position) is that the file system format is different. First, the > > on-disk format is not an ABI, and it is several orders more complex > > than a system call interface. Second, we make no guarantees about > > what the file system created by malicious tools will do. For example, > > XFS developers reject bug reports from file system fuzzers, because > > the v5 format has CRC checks, so randomly corrupted file systems won't > > crash the kernel. Yes, this doesn't protect against maliciously > > created file systems where the attacker makes sure the checksums are > > valid, but only crazy people who think containers are just as secure > > Well, it is not just containers. It is also USB sticks. And people who > believe secure boot is good idea and try to protect kernel against > root. And crazy people who encrypt pointers in dmesg. And... > > People want to use USB sticks from time to time. And while I > understand XFS is so complex it is unsuitable for such use, I'd still > expect bugs to be fixed there. > > I hope VFAT to be safe to mount, because that is very common on USB. > > I also hope ext2/3/4 is safe in that regard. Ext4 will fix file system fuzzing attack bugs on a best efforts basis. That is, when I have time, I've been known to stay up late to bugs reported by fuzzers. I hope ext4 is safe, but I'm not going to make any guarantees that it is Bug-Free(tm). If you want to trust it in that way, you do so at your risk. As far as VFS is concerned, I'm not aware of anyone who has been working on fuzz-proofing VFAT, and looking at the Vault 2016 for "American Fuzzy Lop"[1] while VFAT wasn't specifically tested, for the vast majority of file systems, the "time to first bug" typically ranged from seconds to minutes, with the exception of XFS and ext4 (where it was roughly 2 hours). The specific bugs which triggered in the 2016 AFL presentation have been fixed, at least for the file systems which get regular maintainer attention, but this is why I try to caution people not to count on file systems being proof against maliciously formatted images. [1] https://events.static.linuxfound.org/sites/events/files/slides/AFL%20filesystem%20fuzzing,%20Vault%202016_0.pdf > Anyway it would be nice to have documentation explaining this. If I'm > wrong about VFAT being safe, it would be good to know, and I guess > many will be surprised that XFS is using different rules. Using USB sticks is fine, so long as you trust the provenance of the drive. If you take a random USB stick that is handed to you by someone whom you don't trust implicitly, or worse, that you picked up abandoned on the sidewalk, there have been plenty of articles which describe why this is a REALLY BAD IDEA, and even if you ignore OS-level vuleranbilities, there are also firwmare and hardware based vulerabilities that would put your computer at risk. See [2] and [3] for more details; there's a reason why I've visited at least one financial institution where they put epoxy in USB ports to prevent clueless workers from potentially compromising the bank's computers. [2] https://www.redteamsecure.com/blog/usb-drop-attacks-the-danger-of-lost-and-found-thumb-drives/ [3] https://www.bleepingcomputer.com/news/security/heres-a-list-of-29-different-types-of-usb-attacks/ As far as documentation is concerned, how far should we go? Should there be a warning in the execve(2) system call man page that you shouldn't download random binaries from the network and execute them? :-) Cheers, - Ted
Re: Malicious fs images was Re: ext4 regression in v5.9-rc2 from e7bfb5c9bb3d on ro fs with overlapped bitmaps
On Mon, Jan 11, 2021 at 10:51:20AM -0800, Darrick J. Wong wrote: > On Sun, Jan 10, 2021 at 07:41:02PM +0100, Pavel Machek wrote: > > Hi! > > > > On Fri 2020-10-09 10:37:32, Theodore Y. Ts'o wrote: > > > On Thu, Oct 08, 2020 at 03:22:59PM -0700, Josh Triplett wrote: > > > > > > > > I wasn't trying to make a *new* general principle or policy. I was under > > > > the impression that this *was* the policy, because it never occurred to > > > > me that it could be otherwise. It seemed like a natural aspect of the > > > > kernel/userspace boundary, to the point that the idea of it *not* being > > > > part of the kernel's stability guarantees didn't cross my mind. > > > > > > >From our perspective (and Darrick and I discussed this on this week's > > > ext4 video conference, so it represents the ext4 and xfs maintainer's > > > position) is that the file system format is different. First, the > > > on-disk format is not an ABI, and it is several orders more complex > > > than a system call interface. Second, we make no guarantees about > > > what the file system created by malicious tools will do. For example, > > > XFS developers reject bug reports from file system fuzzers, because > > My recollection of this is quite different -- sybot was sending multiple > zeroday exploits per week to the public xfs list, and nobody at Google > was helping us to /fix/ those bugs.Each report took hours of developer > time to extract the malicious image (because Dmitry couldn't figure out > how to add that ability to syzbot) and syscall sequence from the > reproducer program, plus whatever time it took to craft a patch, test > it, and push it through review. > > Dave and I complained to Dmitry about how the community had zero input > into the rate at which syzbot was allowed to look for xfs bugs. Nobody > at Google would commit to helping fix any of the XFS bugs, and Dmitry > would not give us better choices than (a) Google AI continuing to create > zerodays and leaving the community to clean up the mess, or (b) shutting > off syzbot entirely. At the time I said I would accept letting syzbot > run against xfs until it finds something, and turning it off until we > resolve the issue. That wasn't acceptable, because (I guess) nobody at > Google wants to put /any/ staff time into XFS at all. > > TLDR: XFS /does/ accept bug reports about fuzzed and broken images. > What we don't want is make-work Google AIs spraying zeroday code in > public places and the community developers having to clean up the mess. syzkaller is an open source project that implements a coverage-guided fuzzer for multiple operating system kernels; it's not "Google AI". Anyone can run syzkaller (either by itself, or as part of a syzbot instance) and find the same bugs. - Eric
Re: Malicious fs images was Re: ext4 regression in v5.9-rc2 from e7bfb5c9bb3d on ro fs with overlapped bitmaps
On Sun, Jan 10, 2021 at 07:41:02PM +0100, Pavel Machek wrote: > Hi! > > On Fri 2020-10-09 10:37:32, Theodore Y. Ts'o wrote: > > On Thu, Oct 08, 2020 at 03:22:59PM -0700, Josh Triplett wrote: > > > > > > I wasn't trying to make a *new* general principle or policy. I was under > > > the impression that this *was* the policy, because it never occurred to > > > me that it could be otherwise. It seemed like a natural aspect of the > > > kernel/userspace boundary, to the point that the idea of it *not* being > > > part of the kernel's stability guarantees didn't cross my mind. > > > > >From our perspective (and Darrick and I discussed this on this week's > > ext4 video conference, so it represents the ext4 and xfs maintainer's > > position) is that the file system format is different. First, the > > on-disk format is not an ABI, and it is several orders more complex > > than a system call interface. Second, we make no guarantees about > > what the file system created by malicious tools will do. For example, > > XFS developers reject bug reports from file system fuzzers, because My recollection of this is quite different -- sybot was sending multiple zeroday exploits per week to the public xfs list, and nobody at Google was helping us to /fix/ those bugs. Each report took hours of developer time to extract the malicious image (because Dmitry couldn't figure out how to add that ability to syzbot) and syscall sequence from the reproducer program, plus whatever time it took to craft a patch, test it, and push it through review. Dave and I complained to Dmitry about how the community had zero input into the rate at which syzbot was allowed to look for xfs bugs. Nobody at Google would commit to helping fix any of the XFS bugs, and Dmitry would not give us better choices than (a) Google AI continuing to create zerodays and leaving the community to clean up the mess, or (b) shutting off syzbot entirely. At the time I said I would accept letting syzbot run against xfs until it finds something, and turning it off until we resolve the issue. That wasn't acceptable, because (I guess) nobody at Google wants to put /any/ staff time into XFS at all. TLDR: XFS /does/ accept bug reports about fuzzed and broken images. What we don't want is make-work Google AIs spraying zeroday code in public places and the community developers having to clean up the mess. > > the v5 format has CRC checks, so randomly corrupted file systems > > won't > > crash the kernel. Yes, this doesn't protect against maliciously > > created file systems where the attacker makes sure the checksums are > > valid, but only crazy people who think containers are just as secure > > Well, it is not just containers. It is also USB sticks. And people who > believe secure boot is good idea and try to protect kernel against > root. And crazy people who encrypt pointers in dmesg. And... > > People want to use USB sticks from time to time. And while I > understand XFS is so complex it is unsuitable for such use, I'd still > expect bugs to be fixed there. > > I hope VFAT to be safe to mount, because that is very common on USB. > > I also hope ext2/3/4 is safe in that regard. None of them are, that's why you can't mount local filesystems as an unprivileged user by default. At a bare minimum you'd have to audit the fs driver code to make sure it's robust against unexpected metadata; and since filesystems store complex relational data across multiple indices, you end up needing to hoist a full fsck into the mount process. Or retain the current behavior, which is to delegate the trust decision to the sysadmin. It's not just an XFS thing. --D > Anyway it would be nice to have documentation explaining this. If I'm > wrong about VFAT being safe, it would be good to know, and I guess > many will be surprised that XFS is using different rules. > > Best regards, > Pavel > -- > http://www.livejournal.com/~pavelmachek
Malicious fs images was Re: ext4 regression in v5.9-rc2 from e7bfb5c9bb3d on ro fs with overlapped bitmaps
Hi! On Fri 2020-10-09 10:37:32, Theodore Y. Ts'o wrote: > On Thu, Oct 08, 2020 at 03:22:59PM -0700, Josh Triplett wrote: > > > > I wasn't trying to make a *new* general principle or policy. I was under > > the impression that this *was* the policy, because it never occurred to > > me that it could be otherwise. It seemed like a natural aspect of the > > kernel/userspace boundary, to the point that the idea of it *not* being > > part of the kernel's stability guarantees didn't cross my mind. > > >From our perspective (and Darrick and I discussed this on this week's > ext4 video conference, so it represents the ext4 and xfs maintainer's > position) is that the file system format is different. First, the > on-disk format is not an ABI, and it is several orders more complex > than a system call interface. Second, we make no guarantees about > what the file system created by malicious tools will do. For example, > XFS developers reject bug reports from file system fuzzers, because > the v5 format has CRC checks, so randomly corrupted file systems won't > crash the kernel. Yes, this doesn't protect against maliciously > created file systems where the attacker makes sure the checksums are > valid, but only crazy people who think containers are just as secure Well, it is not just containers. It is also USB sticks. And people who believe secure boot is good idea and try to protect kernel against root. And crazy people who encrypt pointers in dmesg. And... People want to use USB sticks from time to time. And while I understand XFS is so complex it is unsuitable for such use, I'd still expect bugs to be fixed there. I hope VFAT to be safe to mount, because that is very common on USB. I also hope ext2/3/4 is safe in that regard. Anyway it would be nice to have documentation explaining this. If I'm wrong about VFAT being safe, it would be good to know, and I guess many will be surprised that XFS is using different rules. Best regards, Pavel -- http://www.livejournal.com/~pavelmachek signature.asc Description: Digital signature
