On Tue, Sep 05, 2023 at 11:40:42PM +0200, Peter J. Philipp wrote:
> Hi,
>
> I'm porting OpenBSD to the Mango Pi D1. Most of the work is done but now
> is the time to try to get a RAMDISK kernel to boot.
>
> I'm having an issue with atomic_store_64() as shown here (line 1192):
>
> 1184 for (; va < DMAP_MAX_ADDRESS && pa < max_pa;
> 1185 pa += L1_SIZE, va += L1_SIZE, l1_slot++) {
> 1186 KASSERT(l1_slot < Ln_ENTRIES);
> 1187
> 1188 /* gigapages */
> 1189 pn = (pa / PAGE_SIZE);
> 1190 entry = PTE_KERN;
> 1191 entry |= (pn << PTE_PPN0_S);
> 1192 atomic_store_64(&l1[l1_slot], entry);
> 1193 }
>
> The D1 seems to hang on this call. To me that seems like the store hangs that
> I did on powerpc64 when memory was not aligned. When I replaced the
> AMO instruction with a simple C replacement it worked. So I wonder if the
> following is correct or not:
>
> entry = _ALIGN(PTE_KERN);
>
> I looked at riscv-privileged-202111203.pdf (page 79) which is for Sv32 and
> a few pages down is the Sv39 (page 84). But I can't make sense of it. And
> my Mona Lisa book that I have from Waterman mentions on page 101:
>
> ''Misaligned address exceptions occur when the effective address isn't
> divisible
> by the access size - for example, amoadd.w with an address of 0x12.''
>
> I don't think in the bootstrapping of pmap that traps are turned on yet,
> but I could be mistaken, hence the hang.
There is something weird which I can't figure out, this document is golden
to find the truth:
https://github.com/riscv/riscv-isa-manual/releases/download/Priv-v1.12/riscv-privileged-20211203.pdf
OK on page 79 is the Sv32 (32-bit specification), it is kinda irrelevant
for us but there is something bothering me, and we must see into this so that
we can talk about Sv39 (64-bit) which is on page 84, and Sv48 which is on
page 86.
Sv32:
on page 80, figure 4.17 is the 32-bit physical address translation (based on
satp register's MODE) and PPN0 is 10 bits here. PPN1 is 12 bits. PPN is
the physical page number. And the 0 and 1 is the levels of what sort of
pages these are. 4096 byte size is level 0, and level 1 is 4MB (megapages)
size. PTESIZE=4
Sv39:
on page 85, figure 4.20, is the 64-bit (Sv39) physical address translation
based on the satp register which is 64 bits. PPN0 is 9 bits here, PPN1 is
9 bits and PPN2 is 26 bits. These three levels represent 4096 byte physical
page numbers (level 0), 2 MB megapages PPN's (level 1) and 1 GB gigapages
PPN's. "each of which must be virtually and physically aligned to a
boundary equal to its size. A page-fault exception is raised if the physical
address is insufficently aligned". PTESIZE=8
Sv48:
on page 86, figure 4.23, is a 64-bit (Sv48) physical address translation
based on the satp register again which is 64 bits. They have 4 levels of
PPN numbers; level 0 pages (9 bits), level 1 megapages (9 bits),
level 2 gigapages (9 bits) and also introduce 512 GB terapages on level 3
(17 bits). PTESIZE=8
OK now that I have repeated the document I have 3 questions:
1. AFAIK, we ONLY support Sv39 right now, is this true?
2. From /usr/src/sys/arch/riscv64/include/pte.h:
#define PTE_PPN0_S 10
This is the PPN shift value...why is it 10 and not 9? We're using Sv39.
If this is wrong then PTE_PPN[123] also have to be adjusted.
3. Why are we talking about 4 levels of PPN's when there is only 3 levels in
the Sv39 definition?
I put some debugging into my kernel this morning and added a printf in
pmap's pmap_bootstrap_dmap() and had it just before the atomic function:
>> OpenBSD/riscv64 BOOTRISCV64 1.5
boot> boot
cannot open sd0a:/etc/random.seed: No such file or directory
booting sd0a:/bsd: 2152904+693716+8907000+538152
[181604+122+283464+186377]=0xfa1420
bootargs:
aligned_entry=10001000,pn=40000,ae%pn=1000,&l1[l1s]=ffffffc00100ba00
This is what it showed me. In my tests instead of _ALIGN()'ing PTE_KERN I
round_page() the entry value to make it 4096 page aligned (10001000),
I also show the physmem in page numbers 0x40000 which is 262144 pages for
1 GB RAM. aligned_entry % pn == 0x1000, which didn't fit exactly into this.
and finally &l1[l1_slot] which gave this odd looking number:
0xffffffc00100ba00
This is not a page aligned offset, on any physical page number. L1_slot I
think was 320 from earlier debugs and I left it alone in my tests. Let's see,
here it is:
before atomic store l1=ffffffc00100a000 l1_slot=320,[ffffffc00100aa00], entry =
100000c7
So I'm wondering about these misalignments and the questions above.
> If you have another idea why I get these let me know, I'm going to bed now
> and won't be up until 6AM or later tomorrow. I'm letting my kernel compile
> on the slow QEMU host overnight.
Sorry about this, tb@, I'll try to refrain from mentioning my bed times.
Best Regards,
-peter
--
Over thirty years experience on Unix-like Operating Systems starting with QNX.
