Re: Unaligned access in kernel on ARMv6+ (Re: CVS commit: src/sys/dev/usb)
On 2019/01/08 17:33, Nick Hudson wrote:
On 07/01/2019 10:22, Rin Okuyama wrote:
[snip]
This kind of problems cannot be handled in software unless we
(1) use cached memory (for which unaligned access is allowed), or
(2) forbid compiler to generate unaligned access.
This patch
http://www.netbsd.org/~rin/armv6_cached_dma_memory_20180107.patch
adds ARMV6_CACHED_DMA_MEMORY option. If enabled, DMA memory is forcibly
mapped cacheable on ARMv6+ [option (1) above]. This allows us unaligned
access to DMA buffer, however, as Nick and others pointed out, breaks
drivers that do not invalidate cache appropriately. If this option is
disabled, -mno-unaligned-access is added to CFLAGS [option (2) above].
I've tested both on my RPI3B+ (earmv7hf). Kernel of (1) works more than
12 hours without panic. However, vchiq(4) fails to initialize, and mue(4)
receives strange packets of zero-length (two times in 12 hours). Both
smell like driver bugs. Kernel of (2) works without problems as far as
I can see.
Does it work on rpi1? I forget if ARM11_COMPAT_MMU behaves differently here.
I bought my RPI0W yesterday to test it :-).
If ARMV6_CACHED_DMA_MEMORY option is specified, the kernel hangs
indefinitely when attaching audio0 at vcaudio0; unlike RPI3B+, vchiq0
and vcaudio0 are attached, however the system freezes before attaching
message of audio0. If vcaudio0 is disabled in the kernel config file,
the system worked without any troubles more than 12 hours.
The kernel built with -mno-unaligned-access works fine. Also, even if
neither ARMV6_CACHED_DMA_MEMORY nor -mno-unaligned-access options are
specified, axe(4) works without my hack in the current revision. I
confirmed that an instruction is generated which does unaligned access
to USB buffer.
This indicates that unaligned access is permitted even for non-cached
memory on ARMv6 in backward compatible MMU mode. Is it right? (I could
not find any documents about it...)
If it is true, we can possibly restrict ARMV6_CACHED_DMA_MEMORY option
to !ARM_MMU_V6C case. For arch/arm/arm32/bus_dma.c:
#if defined(ARMV6_CACHED_DMA_MEMORY) && !ARM_MMU_V6C
/*
* Force cached mapping for ARMv6+, which allows us
* unaligned access to DMA buffer. For ARMv6 MMU in
* backward compatible mode, unaligned access is
* permitted for non-cached memory.
*/
if (!CPU_IS_ARMV6_P() && !CPU_IS_ARMV7_P())
#endif
{
bool uncached = (flags & BUS_DMA_COHERENT);
...
Also, we can omit -mno-unaligned-access option in
arch/arm/conf/std/Makefile.arm:
.if !empty(MACHINE_ARCH:Mearmv6*) || !empty(MACHINE_ARCH:Mearmv7*)
. if ARMV6_CACHED_DMA_MEMORY
# Use cached memory for DMA on ARMv6+, which allows us unaligned
# access to DMA buffer.
CPPFLAGS+= -DARMV6_CACHED_DMA_MEMORY
. elif ARM11_COMPAT_MMU
# For ARMv6 MMU in backward compatible mode, unaligned access is
# permitted for non-cached memory.
. else
# Otherwise, we need strictly-aligned access to DMA buffer.
CFLAGS+=-mno-unaligned-access
. endif
.endif
I wrote a patch including both changes:
http://www.netbsd.org/~rin/armv6_cached_dma_memory_20180109.patch
How do you think?
I also carried out simple benchmarks of building lang/perl5 of pkgsrc.
The working directory is USB SSD, TMPDIR is tmpfs, and terminal is ssh.
The difference is negligible: (1) 25:36.53 and (2) 25:34.81.
Using cached memory for DMA is slower?
Yes, at least for this single tries. However, it is only about 0.1%
difference, and I think it is within error margin. We may obtain
opposite results in another tries. Anyway, the difference would be
negligibly small.
We should use cached memory for DMA in the future. However, it may break
more drivers than I observed on RPI. Therefore, I would like to propose
a compromise plan:
(a) Before branching netbsd-9, disable ARMV6_CACHED_MEMORY, and use
-mno-unaligned-access.
(b) After branching netbsd-9, enable ARMV6_CACHED_MEMORY, and stop using
-mno-unaligned-access.
(c) After debugging drivers, use cached memory for DMA unconditionally
on ARMv6+ and remove ARMV6_CACHED_DMA_MEMORY option.
Thoughts? Nick, does this look reasonable for you?
OK.
Let's backport all related fixes and plan to remove ARMV6_CACHED_DMA_MEMORY
everywhere. (pending the rpi result)
Yeah, I would like to commit
the original patch
http://www.netbsd.org/~rin/armv6_cached_dma_memory_20180107.patch
or the revised one (explained above)
http://www.netbsd.org/~rin/armv6_cached_dma_memory_20180109.patch
if there's no objection until this weekend. Also, I will send a PR on
vchiq(4) problems. Which patch do you prefer? Or other ideas?
Thanks,
rin
Thanks,
Nick
Re: CVS commit: src/sys/kern
On Tue, Jan 08, 2019 at 04:03:14PM +0100, Manuel Bouyer wrote: > On Tue, Jan 08, 2019 at 03:40:23PM +0100, Christoph Badura wrote: > > On Mon, Jan 07, 2019 at 03:08:38PM +0100, Manuel Bouyer wrote: > > > it looks like this broke Xen domU: > > > http://www-soc.lip6.fr/~bouyer/NetBSD-tests/xen/HEAD/ > > > > > > [ 1.0800662] boot device: xbd0 > > > [ 1.0800662] xenbus: can't get state for device/suspend/event-channel > > > (2) > > > [ 1.0800662] unknown device major 0x > > > [ 1.0800662] root device (default xbd0a): > > > > > > and it waits for the user to press enter > > > do you have an idea ? > > > > Really? Are you sure that change makes a difference? > > No, but it happended at about the same time. > But a run with a 2019-01-06 13:20 UTC build installs and completes the tests > successfully, so I guess something after this commit fixed it. Are you going to test with kern_subr.c reverted to r1.220? That would be interesting. > > I spent Wednesday/Thursday instrumenting that code and figuring out the > > same symptom (unknown device major) for dom0. And the cause there is > > not changed by Michael's change. > > I'm not sure that the "unknown device major" message is a problem, it > may have been this way before. You get that message because rootdev isn't set. setroot_root() return, root_device is still NULL and RB_ASKNAME gets set in boothowto. After that it should sit at a prompt and wait for someone to enter a root device name. The system shouldn't boot automatically.
Re: CVS commit: src/doc
Christos Zoulas wrote in <20190108184053.e22c1f...@cvs.netbsd.org>: |Module Name: src |Committed By: christos |Date: Tue Jan 8 18:40:53 UTC 2019 | |Modified Files: | src/doc: 3RDPARTY | |Log Message: |- put all the GPLv3 software together |- put all the Last GPLv2 version software at the end. |Perhaps it is better to use separate files (per license) at this point? The current version of groff is 1.22.4. --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)
Re: CVS commit: src/sys/kern
On Tue, Jan 08, 2019 at 03:40:23PM +0100, Christoph Badura wrote: > On Mon, Jan 07, 2019 at 03:08:38PM +0100, Manuel Bouyer wrote: > > it looks like this broke Xen domU: > > http://www-soc.lip6.fr/~bouyer/NetBSD-tests/xen/HEAD/ > > > > [ 1.0800662] boot device: xbd0 > > [ 1.0800662] xenbus: can't get state for device/suspend/event-channel (2) > > [ 1.0800662] unknown device major 0x > > [ 1.0800662] root device (default xbd0a): > > > > and it waits for the user to press enter > > do you have an idea ? > > Really? Are you sure that change makes a difference? No, but it happended at about the same time. But a run with a 2019-01-06 13:20 UTC build installs and completes the tests successfully, so I guess something after this commit fixed it. > > I spent Wednesday/Thursday instrumenting that code and figuring out the > same symptom (unknown device major) for dom0. And the cause there is > not changed by Michael's change. I'm not sure that the "unknown device major" message is a problem, it may have been this way before. -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference --
Re: CVS commit: src/sys/kern
On Mon, Jan 07, 2019 at 03:08:38PM +0100, Manuel Bouyer wrote: > it looks like this broke Xen domU: > http://www-soc.lip6.fr/~bouyer/NetBSD-tests/xen/HEAD/ > > [ 1.0800662] boot device: xbd0 > [ 1.0800662] xenbus: can't get state for device/suspend/event-channel (2) > [ 1.0800662] unknown device major 0x > [ 1.0800662] root device (default xbd0a): > > and it waits for the user to press enter > do you have an idea ? Really? Are you sure that change makes a difference? I spent Wednesday/Thursday instrumenting that code and figuring out the same symptom (unknown device major) for dom0. And the cause there is not changed by Michael's change. --chris
Re: Unaligned access in kernel on ARMv6+ (Re: CVS commit: src/sys/dev/usb)
On 07/01/2019 10:22, Rin Okuyama wrote: [snip] This kind of problems cannot be handled in software unless we (1) use cached memory (for which unaligned access is allowed), or (2) forbid compiler to generate unaligned access. This patch http://www.netbsd.org/~rin/armv6_cached_dma_memory_20180107.patch adds ARMV6_CACHED_DMA_MEMORY option. If enabled, DMA memory is forcibly mapped cacheable on ARMv6+ [option (1) above]. This allows us unaligned access to DMA buffer, however, as Nick and others pointed out, breaks drivers that do not invalidate cache appropriately. If this option is disabled, -mno-unaligned-access is added to CFLAGS [option (2) above]. I've tested both on my RPI3B+ (earmv7hf). Kernel of (1) works more than 12 hours without panic. However, vchiq(4) fails to initialize, and mue(4) receives strange packets of zero-length (two times in 12 hours). Both smell like driver bugs. Kernel of (2) works without problems as far as I can see. Does it work on rpi1? I forget if ARM11_COMPAT_MMU behaves differently here. I also carried out simple benchmarks of building lang/perl5 of pkgsrc. The working directory is USB SSD, TMPDIR is tmpfs, and terminal is ssh. The difference is negligible: (1) 25:36.53 and (2) 25:34.81. Using cached memory for DMA is slower? We should use cached memory for DMA in the future. However, it may break more drivers than I observed on RPI. Therefore, I would like to propose a compromise plan: (a) Before branching netbsd-9, disable ARMV6_CACHED_MEMORY, and use -mno-unaligned-access. (b) After branching netbsd-9, enable ARMV6_CACHED_MEMORY, and stop using -mno-unaligned-access. (c) After debugging drivers, use cached memory for DMA unconditionally on ARMv6+ and remove ARMV6_CACHED_DMA_MEMORY option. Thoughts? Nick, does this look reasonable for you? OK. Let's backport all related fixes and plan to remove ARMV6_CACHED_DMA_MEMORY everywhere. (pending the rpi result) Thanks, Nick
