RISCV - physmem is an address not pages in locore.S
Hi OpenBSD/riscv64'ers! After a week of debugging a different issue I noticed this issue with the L2 cache in locore.S: The physical address of the base boot memory is held in register s9, and this is shifted by the L2 cache code by 21 to the right. In order to make 2 MiB offsets. However, I have found in my research that the algorithm is flawed a little. It expects pages not an address on s9. I wrote this program to understand the algorithm better. And I wrote it in C and it should be an exact duplication of the asm code. Please point out if it isn't. Here is the output. I'm attaching the program after this it's colour coded so you can see it better. As you can see with the first output there is bits in the PTE beyond PPN[1] in PPN[2], in the L2 cache. In the second output which ends at the same address the bits are perfectly aligned in PPN[1]. pjp@polarstern$ ./l2shit | tail sd 1FB80003(0110111011) to 1014FB0 sd 1FC3(011111) to 1014FB8 sd 1FC80003(0111001011) to 1014FC0 sd 1FD3(0111010011) to 1014FC8 sd 1FD80003(0111011011) to 1014FD0 sd 1FE3(000011) to 1014FD8 sd 1FE80003(001011) to 1014FE0 sd 1FF3(010011) to 1014FE8 sd 1FF80003(011011) to 1014FF0 sd 2003(100011) to 1014FF8 pjp@polarstern$ ./l2shit pages | tail sd 0FB3(0010110011) to 1014FB0 sd 0FB80003(0010111011) to 1014FB8 sd 0FC3(001111) to 1014FC0 sd 0FC80003(0011001011) to 1014FC8 sd 0FD3(0011010011) to 1014FD0 sd 0FD80003(0011011011) to 1014FD8 sd 0FE3(0011100011) to 1014FE0 sd 0FE80003(0011101011) to 1014FE8 sd 0FF3(000011) to 1014FF0 sd 0FF80003(001011) to 1014FF8 /* 94 lla s1, pagetable_l2 95 srlit4, s9, L2_SHIFT 96 li t2, 512 97 add t3, t4, t2 98 li t0, (PTE_KERN | PTE_X) 99 1: 100 sllit2, t4, PTE_PPN1_S 101 or t5, t0, t2 102 sd t5, (s1) 103 addis1, s1, PTE_SIZE 104 105 addit4, t4, 1 106 bltut4, t3, 1b 107 */ #include #include #include #define P_KERN 0x1 /* not real */ #define P_X 0x2 /* not real */ char * binary(ulong t5) { static char ret[1280]; int i = 0; ret[0] = '\0'; for (i = 53; i >= 0; i--) { switch (i) { case (53 - 26): strlcat(ret,"[32m", sizeof(ret)); break; case (53 - 26 - 9): strlcat(ret,"[34m", sizeof(ret)); break; case (53 - 26 - 9 - 9): strlcat(ret,"[35m", sizeof(ret)); break; default: //strlcat(ret,"[0m", sizeof(ret)); break; } if (t5 & (1UL << i)) { strlcat(ret, "1", sizeof(ret)); } else { strlcat(ret, "0", sizeof(ret)); } } return (&ret[0]); } int main(int argc, char *argv[]) { u_long s1 = 0x1014000; /* pagetable l2 */ u_long s9 = 0x4020 >> ((argc > 1) ? 12 : 0);/* physmem s9 (pages?) */ u_long t4 = s9 >> 21; u_long t2 = 512; u_long t3 = t4 + t2; u_long t0 = (P_KERN | P_X); u_long t5; repeat: t2 = t4 << 19; t5 = t0 | t2; printf("sd %08lX(%s[0m) to %lX\n", t5, binary(t5), s1); s1 += 8; t4 += 1; if (t4 < t3) goto repeat; return 0; } Please look at this document section 4.4.1 figure 4.21 to see the structure of the PTE. https://mainrechner.de/riscv-privileged-20211203.pdf Best Regards, -peter -- Over thirty years experience on Unix-like Operating Systems starting with QNX.
Re: RISCV - physmem is an address not pages in locore.S
> Date: Sun, 17 Sep 2023 12:40:29 +0200 > From: "Peter J. Philipp" Sorry Peter, But this doesn't make any sense to me. Your C code is just as unreadable as the assembly code ;) And your explanation doesn't make sense. The code works fine on existing hardware supported by OpenBSD. Your previous mails were also high on speculation and low on facts. Cheers, Mark > Hi OpenBSD/riscv64'ers! > > After a week of debugging a different issue I noticed this issue with the > L2 cache in locore.S: > > The physical address of the base boot memory is held in register s9, > and this is shifted by the L2 cache code by 21 to the right. In order to > make 2 MiB offsets. However, I have found in my research that the algorithm > is flawed a little. It expects pages not an address on s9. I wrote this > program to understand the algorithm better. And I wrote it in C and it should > be an exact duplication of the asm code. Please point out if it isn't. > > Here is the output. I'm attaching the program after this it's colour coded > so you can see it better. As you can see with the first output there is > bits in the PTE beyond PPN[1] in PPN[2], in the L2 cache. In the second > output which ends at the same address the bits are perfectly aligned in > PPN[1]. > > pjp@polarstern$ ./l2shit | tail > sd 1FB80003(0110111011) to 1014FB0 > sd 1FC3(011111) to 1014FB8 > sd 1FC80003(0111001011) to 1014FC0 > sd 1FD3(0111010011) to 1014FC8 > sd 1FD80003(0111011011) to 1014FD0 > sd 1FE3(000011) to 1014FD8 > sd 1FE80003(001011) to 1014FE0 > sd 1FF3(010011) to 1014FE8 > sd 1FF80003(011011) to 1014FF0 > sd 2003(100011) to 1014FF8 > pjp@polarstern$ ./l2shit pages | tail > sd 0FB3(0010110011) to 1014FB0 > sd 0FB80003(0010111011) to 1014FB8 > sd 0FC3(001111) to 1014FC0 > sd 0FC80003(0011001011) to 1014FC8 > sd 0FD3(0011010011) to 1014FD0 > sd 0FD80003(0011011011) to 1014FD8 > sd 0FE3(0011100011) to 1014FE0 > sd 0FE80003(0011101011) to 1014FE8 > sd 0FF3(000011) to 1014FF0 > sd 0FF80003(001011) to 1014FF8 > > > /* > > 94 lla s1, pagetable_l2 > 95 srlit4, s9, L2_SHIFT > 96 li t2, 512 > 97 add t3, t4, t2 > 98 li t0, (PTE_KERN | PTE_X) > 99 1: > 100 sllit2, t4, PTE_PPN1_S > 101 or t5, t0, t2 > 102 sd t5, (s1) > 103 addis1, s1, PTE_SIZE > 104 > 105 addit4, t4, 1 > 106 bltut4, t3, 1b > 107 > > */ > > #include > #include > #include > > #define P_KERN0x1 /* not real */ > #define P_X 0x2 /* not real */ > > char * > binary(ulong t5) > { > static char ret[1280]; > int i = 0; > > ret[0] = '\0'; > > for (i = 53; i >= 0; i--) { > switch (i) { > case (53 - 26): > strlcat(ret,"[32m", sizeof(ret)); > break; > case (53 - 26 - 9): > strlcat(ret,"[34m", sizeof(ret)); > break; > case (53 - 26 - 9 - 9): > strlcat(ret,"[35m", sizeof(ret)); > break; > default: > //strlcat(ret,"[0m", sizeof(ret)); > break; > } > > if (t5 & (1UL << i)) { > strlcat(ret, "1", sizeof(ret)); > } else { > strlcat(ret, "0", sizeof(ret)); > } > } > > return (&ret[0]); > } > > > > int > main(int argc, char *argv[]) > { > u_long s1 = 0x1014000; /* pagetable l2 */ > u_long s9 = 0x4020 >> ((argc > 1) ? 12 : 0);/* physmem s9 > (pages?) */ > u_long t4 = s9 >> 21; > u_long t2 = 512; > u_long t3 = t4 + t2; > u_long t0 = (P_KERN | P_X); > u_long t5; > > repeat: > t2 = t4 << 19; > t5 = t0 |
Re: RISCV - physmem is an address not pages in locore.S
On Sun, Sep 17, 2023 at 04:22:14PM +0200, Mark Kettenis wrote: > > Date: Sun, 17 Sep 2023 12:40:29 +0200 > > From: "Peter J. Philipp" > > Sorry Peter, > > But this doesn't make any sense to me. Your C code is just as > unreadable as the assembly code ;) Yeah it should be torn out and replaced. I have some replacemnent code if you're willing to touch it with a 10 foot stick. Only.. it doesn't work (yet) and I haven't gotten the idea yet as to why. > And your explanation doesn't make sense. The code works fine on > existing hardware supported by OpenBSD. Your previous mails were also > high on speculation and low on facts. Yes it works fine. Well it boots. I hope the L2 cache does get remapped in pmap.c, I haven't looked at this. Also I found that the L1 cache is on a 4096 byte alignment, I have read some of the docs on the Mango Pi and they expect a 16K alignment. Why? I don't know. That's a simple fix I guess and it shouldn't make much diff other than perhaps making the kernel a bit bigger. If you do find that there is some truth to my translation from asm to C, then the last 200 MiB is weird. Is that where the stack resides in the boot btw? I dunno. The rest was probably speculation. For that, sorry for wasting time. Best Regards, -peter > Cheers, > > Mark > > > Hi OpenBSD/riscv64'ers! > > > > After a week of debugging a different issue I noticed this issue with the > > L2 cache in locore.S: > > > > The physical address of the base boot memory is held in register s9, > > and this is shifted by the L2 cache code by 21 to the right. In order to > > make 2 MiB offsets. However, I have found in my research that the algorithm > > is flawed a little. It expects pages not an address on s9. I wrote this > > program to understand the algorithm better. And I wrote it in C and it > > should > > be an exact duplication of the asm code. Please point out if it isn't. > > > > Here is the output. I'm attaching the program after this it's colour coded > > so you can see it better. As you can see with the first output there is > > bits in the PTE beyond PPN[1] in PPN[2], in the L2 cache. In the second > > output which ends at the same address the bits are perfectly aligned in > > PPN[1]. > > > > pjp@polarstern$ ./l2shit | tail > > sd 1FB80003(0110111011) to > > 1014FB0 > > sd 1FC3(011111) to > > 1014FB8 > > sd 1FC80003(0111001011) to > > 1014FC0 > > sd 1FD3(0111010011) to > > 1014FC8 > > sd 1FD80003(0111011011) to > > 1014FD0 > > sd 1FE3(000011) to > > 1014FD8 > > sd 1FE80003(001011) to > > 1014FE0 > > sd 1FF3(010011) to > > 1014FE8 > > sd 1FF80003(011011) to > > 1014FF0 > > sd 2003(100011) to > > 1014FF8 > > pjp@polarstern$ ./l2shit pages | tail > > sd 0FB3(0010110011) to > > 1014FB0 > > sd 0FB80003(0010111011) to > > 1014FB8 > > sd 0FC3(001111) to > > 1014FC0 > > sd 0FC80003(0011001011) to > > 1014FC8 > > sd 0FD3(0011010011) to > > 1014FD0 > > sd 0FD80003(0011011011) to > > 1014FD8 > > sd 0FE3(0011100011) to > > 1014FE0 > > sd 0FE80003(0011101011) to > > 1014FE8 > > sd 0FF3(000011) to > > 1014FF0 > > sd 0FF80003(001011) to > > 1014FF8 > > > > > > /* > > > > 94 lla s1, pagetable_l2 > > 95 srlit4, s9, L2_SHIFT > > 96 li t2, 512 > > 97 add t3, t4, t2 > > 98 li t0, (PTE_KERN | PTE_X) > > 99 1: > > 100 sllit2, t4, PTE_PPN1_S > > 101 or t5, t0, t2 > > 102 sd t5, (s1) > > 103 addis1, s1, PTE_SIZE > > 104 > > 105 addit4, t4, 1 > > 106 bltut4, t3, 1b > > 107 > > > > */ > > > > #include > > #include > > #include > > > > #define P_KERN 0x1 /* not real */ > > #define P_X 0x2 /* not real */ > > > > char * > > binary(ulong t5) > > { > > static char ret[1280]; > > int i = 0; > > > > ret[0] = '\0'; > > > > for (i = 53; i >= 0; i--) { > > switch (
Re: RISCV - physmem is an address not pages in locore.S
On Sun, Sep 17, 2023 at 04:51:11PM +0200, Peter J. Philipp wrote: > If you do find that there is some truth to my translation from asm to C, > then the last 200 MiB is weird. Is that where the stack resides in the > boot btw? I dunno. Sorry this should say 2MiB, I have too many 0x20 in my head. Best Regards, -peter
Re: RISCV - physmem is an address not pages in locore.S
On Sun, Sep 17, 2023 at 04:51:11PM +0200, Peter J. Philipp wrote: > On Sun, Sep 17, 2023 at 04:22:14PM +0200, Mark Kettenis wrote: > > > Date: Sun, 17 Sep 2023 12:40:29 +0200 > > > From: "Peter J. Philipp" > > > > Sorry Peter, > > > > But this doesn't make any sense to me. Your C code is just as > > unreadable as the assembly code ;) > > Yeah it should be torn out and replaced. I have some replacemnent code > if you're willing to touch it with a 10 foot stick. Only.. it doesn't > work (yet) and I haven't gotten the idea yet as to why. > > > And your explanation doesn't make sense. The code works fine on > > existing hardware supported by OpenBSD. Your previous mails were also > > high on speculation and low on facts. > > Yes it works fine. Well it boots. I hope the L2 cache does get remapped > in pmap.c, I haven't looked at this. > > Also I found that the L1 cache is on a 4096 byte alignment, I have read > some of the docs on the Mango Pi and they expect a 16K alignment. Why? I > don't know. That's a simple fix I guess and it shouldn't make much diff > other than perhaps making the kernel a bit bigger. Here is another one that I may have misinterpreted. Regarding unalignment ''Any level of PTE may be a leaf PTE, so in addition to 4 KiB pages, Sv39 supports 2 MiB megapages and 1 GiB gigapages, each of which must be virtually and physically aligned to a boundary equal to its size. A page-fault exception is raised if the physical address is insufficiently aligned.'' In OpenBSD locore.S, the L1 cache is not a leaf it is a branch (aka pointer) to the L2 Leaf. But say someone misinterprets this the same way I did, there may be something wrong in the way we map a: physmem (0x20 alignment and not 1 GiB alignment) to KERNBASE which is a 1 GiB alignment. I'm sorry this spooked me a little before I knew the distinction of a leaf and a branch. Difference is a Leaf has to be any of PTE_[RWX] set to true, where a branch does not. That's why PTE_V on the branch pointer is done. Best Regards, -peter > If you do find that there is some truth to my translation from asm to C, > then the last 200 MiB is weird. Is that where the stack resides in the > boot btw? I dunno. > > The rest was probably speculation. For that, sorry for wasting time. > > Best Regards, > -peter > > > > Cheers, > > > > Mark > > > > > Hi OpenBSD/riscv64'ers! > > > > > > After a week of debugging a different issue I noticed this issue with the > > > L2 cache in locore.S: > > > > > > The physical address of the base boot memory is held in register s9, > > > and this is shifted by the L2 cache code by 21 to the right. In order to > > > make 2 MiB offsets. However, I have found in my research that the > > > algorithm > > > is flawed a little. It expects pages not an address on s9. I wrote this > > > program to understand the algorithm better. And I wrote it in C and it > > > should > > > be an exact duplication of the asm code. Please point out if it isn't. > > > > > > Here is the output. I'm attaching the program after this it's colour > > > coded > > > so you can see it better. As you can see with the first output there is > > > bits in the PTE beyond PPN[1] in PPN[2], in the L2 cache. In the second > > > output which ends at the same address the bits are perfectly aligned in > > > PPN[1]. > > > > > > pjp@polarstern$ ./l2shit | tail > > > sd 1FB80003(0110111011) to > > > 1014FB0 > > > sd 1FC3(011111) to > > > 1014FB8 > > > sd 1FC80003(0111001011) to > > > 1014FC0 > > > sd 1FD3(0111010011) to > > > 1014FC8 > > > sd 1FD80003(0111011011) to > > > 1014FD0 > > > sd 1FE3(000011) to > > > 1014FD8 > > > sd 1FE80003(001011) to > > > 1014FE0 > > > sd 1FF3(010011) to > > > 1014FE8 > > > sd 1FF80003(011011) to > > > 1014FF0 > > > sd 2003(100011) to > > > 1014FF8 > > > pjp@polarstern$ ./l2shit pages | tail > > > sd 0FB3(0010110011) to > > > 1014FB0 > > > sd 0FB80003(0010111011) to > > > 1014FB8 > > > sd 0FC3(001111) to > > > 1014FC0 > > > sd 0FC80003(0011001011) to > > > 1014FC8 > > > sd 0FD3(0011010011) to > > > 1014FD0 > > > sd 0FD80003(0011011011) to > > > 1014FD8 > > > sd 0FE3(00111000
Re: RISCV - physmem is an address not pages in locore.S
On Sun, Sep 17, 2023 at 04:22:14PM +0200, Mark Kettenis wrote: > > Date: Sun, 17 Sep 2023 12:40:29 +0200 > > From: "Peter J. Philipp" > > Sorry Peter, > > But this doesn't make any sense to me. Your C code is just as > unreadable as the assembly code ;) > > And your explanation doesn't make sense. The code works fine on > existing hardware supported by OpenBSD. Your previous mails were also > high on speculation and low on facts. > > Cheers, > > Mark I took a break thought about what you said and bettered the C code. I hope it makes better sense? Code after .signature. Best Regards, -peter /* 94 lla s1, pagetable_l2 95 srlit4, s9, L2_SHIFT 96 li t2, 512 97 add t3, t4, t2 98 li t0, (PTE_KERN | PTE_X) 99 1: 100 sllit2, t4, PTE_PPN1_S 101 or t5, t0, t2 102 sd t5, (s1) 103 addis1, s1, PTE_SIZE 104 105 addit4, t4, 1 106 bltut4, t3, 1b 107 */ #include #include #include #define P_KERN 0x1 /* not real, for demonstration purposes only */ #define P_X 0x2 /* not real, for demonstration purposes only */ char * binary(ulong t5) { static char ret[1280]; int i = 0; ret[0] = '\0'; for (i = 53; i >= 0; i--) { switch (i) { case (53 - 26): strlcat(ret,"[32m", sizeof(ret)); break; case (53 - 26 - 9): strlcat(ret,"[34m", sizeof(ret)); break; case (53 - 26 - 9 - 9): strlcat(ret,"[35m", sizeof(ret)); break; default: //strlcat(ret,"[0m", sizeof(ret)); break; } if (t5 & (1UL << i)) { strlcat(ret, "1", sizeof(ret)); } else { strlcat(ret, "0", sizeof(ret)); } } return (&ret[0]); } /* * from /usr/src/sys/arch/riscv64/include/pte.h - * PTE magic numbers sed'ed s/PTE_/P_/g */ #define P_SIZE 8 /* 8 bytes PTE size */ #define P_PPN1_S19 /* explanation below */ /* * P_PPN1_S is a shift of bits from the LSb of the PTE leftwards to the * respective level (given 0, 1, 2, 3), this is becuase the PTE looks like * this: * * Sv39 Page Table Entry * 63 0 * * |N| rsvd| PPN[2] | PPN[1] | PPN[0] | ptebits | * * |26 | 9 | 9 | 10 * | || | * | || | * | || +--->P_PPN0_S 10 * | || * | |+---> P_PPN1_S 19 * | | * | +> P_PPN2_S 28 * | * +--> P_PPN3_S 37* * * * In Sv39 this is really 26? which is 26 bits pages, or 67,108,864 pages * or 64 Mega Pages (not to be confused with the Megapage ie, PPN[1] * or 274877906944 bytes (big number) * */ #define L2_SHIFT21 /* 2 MiB == 21 bits == 2 ^ 21 */ #define PAGE_SHIFT 12 /* 4096 bytes */ int main(int argc, char *argv[]) { u_long pagetable_l2 = 0x100c000UL;/* VA from readelf -a bsd |\ grep pagetable_l2 */ u_long physmem = 0x4020UL >> ((argc > 1) ? PAGE_SHIFT : 0); /* PA in s9 */ u_long megapages = physmem >> L2_SHIFT; /* physmem base / 2 MiB */ u_long slot = 512; /* slot # */ u_long slot_limit = megapages + slot; u_long pte_bits = (P_KERN | P_X); /* reg t0, spans 10b from LSb */ u_long pte; /* register t5 */ do { /* 100 sllit2, t4, PTE_PPN1_S */ slot = megapages << P_PPN1_S; /* 101 or t5, t0, t2 */ pte = pte_bits | slot; /* 102 sd t5, (s1) */ printf("sd %08lX(%s[0m) to %lX\n", pte
Re: RISCV - physmem is an address not pages in locore.S
On Sun, Sep 17, 2023 at 04:51:11PM +0200, Peter J. Philipp wrote: > On Sun, Sep 17, 2023 at 04:22:14PM +0200, Mark Kettenis wrote: > > > Date: Sun, 17 Sep 2023 12:40:29 +0200 > > > From: "Peter J. Philipp" > > > > Sorry Peter, > > > > But this doesn't make any sense to me. Your C code is just as > > unreadable as the assembly code ;) > > Yeah it should be torn out and replaced. I have some replacemnent code > if you're willing to touch it with a 10 foot stick. Only.. it doesn't > work (yet) and I haven't gotten the idea yet as to why. FYI, I have finished my work on making a generic asm function for page tables in locore.S, it may be overkill though but could be handy for the future. I've committed it to my MANGOPI branch and if you like it you can steal it. I don't expect this to be in OpenBSD if you don't like it ;). https://github.com/pbug44/openbsd-src/commit/b3a1c1206ddc7769233f00f0f5c1780666b78682 BTW I'm not an asm pro. I'm a newbie having only started with asm much since 2019 on powerpc. So I'm willing to share that this took me 10 days to get to where it is today, though not all days I was working on it, but close. I also refactored it completely about 3 or 4 times. Best Regards, -peter