Re: Simple contribution please?
On Fri, Aug 12, 2016 at 09:21:11PM +0200, Samuel Thibault wrote: > Richard Braun, on Fri 12 Aug 2016 21:16:02 +0200, wrote: > > On Fri, Aug 12, 2016 at 07:57:48PM +0200, Samuel Thibault wrote: > > > It becomes more and more clear that we shouldn't steal functions from > > > glibc into gnumach, it poses cross-building issues from Linux. > > > > > > Could somebody contribute, or steal from a BSD the following functions, > > > to be included in gnumach/kern/strings.c? > > > > > > - memcmp > > > - memcpy > > > - memmove > > > - strchr > > > - strsep > > > - strstr > > > > Note that it's not that "simple" since we'd like implementations that > > aren't naive, i.e. assembly with rep instructions. In particular, it > > makes a huge difference in virtualized guests compared to C-based ones > > because of vmenter/vmexit. > > > > When such functions are used on device mapped memory. > > I guess perhaps even only memcpy is used on device mapped memory? That's likely. I can take care of this when I have time. -- Richard Braun
Re: Simple contribution please?
Richard Braun, on Fri 12 Aug 2016 21:16:02 +0200, wrote: > On Fri, Aug 12, 2016 at 07:57:48PM +0200, Samuel Thibault wrote: > > It becomes more and more clear that we shouldn't steal functions from > > glibc into gnumach, it poses cross-building issues from Linux. > > > > Could somebody contribute, or steal from a BSD the following functions, > > to be included in gnumach/kern/strings.c? > > > > - memcmp > > - memcpy > > - memmove > > - strchr > > - strsep > > - strstr > > Note that it's not that "simple" since we'd like implementations that > aren't naive, i.e. assembly with rep instructions. In particular, it > makes a huge difference in virtualized guests compared to C-based ones > because of vmenter/vmexit. > > When such functions are used on device mapped memory. I guess perhaps even only memcpy is used on device mapped memory? Samuel
Re: Simple contribution please?
On Fri, Aug 12, 2016 at 09:16:02PM +0200, Richard Braun wrote: > Note that it's not that "simple" since we'd like implementations that > aren't naive, i.e. assembly with rep instructions. In particular, it > makes a huge difference in virtualized guests compared to C-based ones > because of vmenter/vmexit. When such functions are used on device mapped memory. -- Richard Braun
Re: Simple contribution please?
On Fri, Aug 12, 2016 at 07:57:48PM +0200, Samuel Thibault wrote: > It becomes more and more clear that we shouldn't steal functions from > glibc into gnumach, it poses cross-building issues from Linux. > > Could somebody contribute, or steal from a BSD the following functions, > to be included in gnumach/kern/strings.c? > > - memcmp > - memcpy > - memmove > - strchr > - strsep > - strstr Note that it's not that "simple" since we'd like implementations that aren't naive, i.e. assembly with rep instructions. In particular, it makes a huge difference in virtualized guests compared to C-based ones because of vmenter/vmexit. -- Richard Braun
Re: Fwd: Hurd shutdown problems
On Fri, Aug 12, 2016 at 09:07:48PM +0200, Samuel Thibault wrote: > > I'm curious: what makes it definitely wrong on a PC ? > > A PC has BIOS stuff between A and 10. Right, misread a 0 again. -- Richard Braun
Re: Fwd: Hurd shutdown problems
Richard Braun, on Fri 12 Aug 2016 21:06:48 +0200, wrote: > On Fri, Aug 12, 2016 at 07:53:08PM +0200, Samuel Thibault wrote: > > That's what I'm talking about, and that's the second part of the printfs > > above, and they are wrong: 1-100 is definitely wrong on a PC, > > and it includes the debugging symbols. > > I'm curious: what makes it definitely wrong on a PC ? A PC has BIOS stuff between A and 10. Samuel
Re: Fwd: Hurd shutdown problems
On Fri, Aug 12, 2016 at 07:53:08PM +0200, Samuel Thibault wrote: > That's what I'm talking about, and that's the second part of the printfs > above, and they are wrong: 1-100 is definitely wrong on a PC, > and it includes the debugging symbols. I'm curious: what makes it definitely wrong on a PC ? > Yes, but only the heap. The load of segments not containing the heap is > full: > > vm_page_load 1-100 1-100 > vm_page_load 7a00-7ffe 7a00-7ffe That's indeed a problem, and one that I don't see in X15... I guess I didn't have the case where a segment didn't clip with the heap until now, in which case, instead of being completely loaded as available, it should be loaded as reserved, and only later made available. -- Richard Braun
Re: Fwd: Hurd shutdown problems
Samuel Thibault, on Fri 12 Aug 2016 19:53:08 +0200, wrote: > Yes, but only the heap. The load of segments not containing the heap is > full: > > vm_page_load 1-100 1-100 > vm_page_load 7a00-7ffe 7a00-7ffe (and only loading the available part of the heap would not provide DMA-able memory for the linux layer) Samuel
Re: Fwd: Hurd shutdown problems
Richard Braun, on Fri 12 Aug 2016 19:57:10 +0200, wrote: > On Fri, Aug 12, 2016 at 05:17:26PM +0200, Samuel Thibault wrote: > > biosmem: heap: 114f000-7a00 > > > > and objdump shows: > > > > LOAD off0x1000 vaddr 0x8100 paddr 0x0100 align 2**12 > > filesz 0x00114700 memsz 0x00114700 flags r-x > > LOAD off0x00116000 vaddr 0x81115000 paddr 0x01115000 align 2**12 > > filesz 0xd151 memsz 0x00039384 flags rw- > > > > It seems biosmem's heap doesn't even exclude the kernel data?! > > Could be a mistake with regard to the linker script. No, I was just misreading the figures. 0x0100 + 0x00114700 is 0x01114700, and 0x01115000 plus 0x00039384 is 0x0114e384, and I confused 0x00114700 with 0x0114e384. Samuel
Re: Fwd: Hurd shutdown problems
On Fri, Aug 12, 2016 at 07:57:10PM +0200, Richard Braun wrote: > On Fri, Aug 12, 2016 at 05:17:26PM +0200, Samuel Thibault wrote: > > biosmem: heap: 114f000-7a00 > > > > and objdump shows: > > > > LOAD off0x1000 vaddr 0x8100 paddr 0x0100 align 2**12 > > filesz 0x00114700 memsz 0x00114700 flags r-x > > LOAD off0x00116000 vaddr 0x81115000 paddr 0x01115000 align 2**12 > > filesz 0xd151 memsz 0x00039384 flags rw- > > > > It seems biosmem's heap doesn't even exclude the kernel data?! > > Could be a mistake with regard to the linker script. Misread it too :). -- Richard Braun
Simple contribution please?
Hello, It becomes more and more clear that we shouldn't steal functions from glibc into gnumach, it poses cross-building issues from Linux. Could somebody contribute, or steal from a BSD the following functions, to be included in gnumach/kern/strings.c? - memcmp - memcpy - memmove - strchr - strsep - strstr Thanks, Samuel
Re: Fwd: Hurd shutdown problems
On Fri, Aug 12, 2016 at 05:17:26PM +0200, Samuel Thibault wrote: > biosmem: heap: 114f000-7a00 > > and objdump shows: > > LOAD off0x1000 vaddr 0x8100 paddr 0x0100 align 2**12 > filesz 0x00114700 memsz 0x00114700 flags r-x > LOAD off0x00116000 vaddr 0x81115000 paddr 0x01115000 align 2**12 > filesz 0xd151 memsz 0x00039384 flags rw- > > It seems biosmem's heap doesn't even exclude the kernel data?! Could be a mistake with regard to the linker script. -- Richard Braun
Re: Fwd: Hurd shutdown problems
Richard Braun, on Fri 12 Aug 2016 19:50:51 +0200, wrote: > On Fri, Aug 12, 2016 at 07:08:07PM +0200, Samuel Thibault wrote: > > More precisely though, adding debugging to vm_page_load: > > > > vm_page_load 1-100 1-100 > > vm_page_load 100-7a00 114f000-79c41000 > > vm_page_load 7a00-7ffe 7a00-7ffe > > > > I.e. it properly skips the kernel (100-114f000), but nothing else. > > > > What is supposed to exclude everything else? (modules, VGA BIOS, etc.) > > Look at the vm_page_load calls and you'll see there is a range of > available pages inside each loaded region. That's what I'm talking about, and that's the second part of the printfs above, and they are wrong: 1-100 is definitely wrong on a PC, and it includes the debugging symbols. > 3/ When enabling the virtual memory system, biosmem_setup is called. > It loads each segment into the vm_page module, but is careful to clip > the biosmem heap from them. But nothing else. > When loading a segment, the biosmem heap > part is passed as [avail_start, avail_end] to vm_page_load. > To properly answer your question, step 1/ is what looks at the boot > data (biosmem_find_boot_data) so that the resulting heap boundaries > completely exclude any. Yes, but only the heap. The load of segments not containing the heap is full: vm_page_load 1-100 1-100 vm_page_load 7a00-7ffe 7a00-7ffe Samuel
Re: Fwd: Hurd shutdown problems
On Fri, Aug 12, 2016 at 07:08:07PM +0200, Samuel Thibault wrote: > More precisely though, adding debugging to vm_page_load: > > vm_page_load 1-100 1-100 > vm_page_load 100-7a00 114f000-79c41000 > vm_page_load 7a00-7ffe 7a00-7ffe > > I.e. it properly skips the kernel (100-114f000), but nothing else. > > What is supposed to exclude everything else? (modules, VGA BIOS, etc.) Look at the vm_page_load calls and you'll see there is a range of available pages inside each loaded region. Here is how biosmem operates : 1/ biosmem_bootstrap is called. It sets the early allocator heap by looking at each segment (dma, directmap and highmem if present). Each segment and their associated heap is stored in biosmem_segments. The biosmem heap is the heap with most available pages that can be directly mapped by the kernel once paging is enabled. Other heaps are of no interest after this. 2/ When using the early allocator (biosmem_bootalloc), a chunk of contiguous pages is removed from the heap and given to the caller. 3/ When enabling the virtual memory system, biosmem_setup is called. It loads each segment into the vm_page module, but is careful to clip the biosmem heap from them. When loading a segment, the biosmem heap part is passed as [avail_start, avail_end] to vm_page_load. 4/ Once the VM system is enabled, memory that wasn't part of the heap is normally reserved, and can be made available by calling vm_page_manage. To properly answer your question, step 1/ is what looks at the boot data (biosmem_find_boot_data) so that the resulting heap boundaries completely exclude any. -- Richard Braun
Re: Fwd: Hurd shutdown problems
Samuel Thibault, on Fri 12 Aug 2016 19:08:07 +0200, wrote:
> What is supposed to exclude everything else? (modules, VGA BIOS, etc.)
I'm tempted to apply the attached patch at least to the Debian package
to brown-tape-fix the issue.
What it does is:
- Make biosmem_load_segment look for the biggest area available inside
the segment, instead of assuming that either the segment contains the
biosmem heap and only the available part of the heap should be used,
or it doesn't contain the heap, and thus the whole segment should be
used.
- Make biosmem_find_boot_data skip the biosmem heap, as well as the
wholes in the biosmem map.
Samuel
diff --git a/i386/i386at/biosmem.c b/i386/i386at/biosmem.c
index a7a440e..2ca1f61 100644
--- a/i386/i386at/biosmem.c
+++ b/i386/i386at/biosmem.c
@@ -440,9 +440,34 @@ biosmem_find_boot_data(const struct multiboot_raw_info
*mbi, uint32_t min,
struct elf_shdr *shdr;
uint32_t i, start, end = end;
unsigned long tmp;
+const struct biosmem_map_entry *entry;
start = max;
+/* Exclude unmapped areas */
+i = 0;
+entry = biosmem_map;
+while (entry < biosmem_map + biosmem_map_size)
+{
+/* Exclude memory before this entry */
+if (i < entry->base_addr)
+biosmem_find_boot_data_update(min, &start, &end, i,
entry->base_addr);
+if (entry->type == BIOSMEM_TYPE_AVAILABLE)
+/* Do not exclude this area */
+i = entry->base_addr + entry->length;
+else
+/* Exclude this area too */
+i = entry->base_addr;
+entry++;
+}
+/* Exclude last entry and anything else beyond */
+if (i < max)
+biosmem_find_boot_data_update(min, &start, &end, i, max);
+
+if (biosmem_heap_cur)
+/* Heap is in use */
+biosmem_find_boot_data_update(min, &start, &end, biosmem_heap_cur,
biosmem_heap_end);
+
biosmem_find_boot_data_update(min, &start, &end, _kvtophys(&_start),
_kvtophys(&_end));
@@ -738,6 +763,8 @@ biosmem_load_segment(struct biosmem_segment *seg, uint64_t
max_phys_end,
phys_addr_t avail_start, phys_addr_t avail_end)
{
unsigned int seg_index;
+phys_addr_t start, end, max_start, max_end;
+uint32_t next;
seg_index = seg - biosmem_segments;
@@ -753,6 +780,34 @@ biosmem_load_segment(struct biosmem_segment *seg, uint64_t
max_phys_end,
phys_end = max_phys_end;
}
+#ifndef MACH_HYP
+max_start = phys_start;
+max_end = phys_start;
+next = phys_start;
+
+do {
+ extern struct multiboot_info boot_info;
+
+start = next;
+end = biosmem_find_boot_data((struct multiboot_raw_info *)&boot_info,
start, phys_end, &next);
+
+if (end == 0) {
+end = phys_end;
+next = 0;
+}
+
+if ((end - start) > (max_end - max_start)) {
+max_start = start;
+max_end = end;
+}
+} while (next != 0);
+
+max_start = round_page(max_start);
+max_end = trunc_page(max_end);
+
+seg->avail_start = max_start;
+seg->avail_end = max_end;
+#else
if ((avail_start < phys_start) || (avail_start >= phys_end))
avail_start = phys_start;
@@ -761,7 +816,9 @@ biosmem_load_segment(struct biosmem_segment *seg, uint64_t
max_phys_end,
seg->avail_start = avail_start;
seg->avail_end = avail_end;
-vm_page_load(seg_index, phys_start, phys_end, avail_start, avail_end);
+#endif
+
+vm_page_load(seg_index, phys_start, phys_end, seg->avail_start,
seg->avail_end);
}
void __init
Re: Fwd: Hurd shutdown problems
Hello, Samuel Thibault, on Fri 12 Aug 2016 17:27:42 +0200, wrote: > Samuel Thibault, on Fri 12 Aug 2016 17:17:26 +0200, wrote: > > biosmem: heap: 114f000-7a00 > > > > and objdump shows: > > > > LOAD off0x1000 vaddr 0x8100 paddr 0x0100 align 2**12 > > filesz 0x00114700 memsz 0x00114700 flags r-x > > LOAD off0x00116000 vaddr 0x81115000 paddr 0x01115000 align 2**12 > > filesz 0xd151 memsz 0x00039384 flags rw- > > > > It seems biosmem's heap doesn't even exclude the kernel data?! > > Ah, sorry, it does, I misread it. But it doesn't exclude anything else, > apparently. Actually it does since it's only 114f000-7a00. I guess the biosmem heap was only meant for biosmem_bootalloc allocations. More precisely though, adding debugging to vm_page_load: vm_page_load 1-100 1-100 vm_page_load 100-7a00 114f000-79c41000 vm_page_load 7a00-7ffe 7a00-7ffe I.e. it properly skips the kernel (100-114f000), but nothing else. What is supposed to exclude everything else? (modules, VGA BIOS, etc.) Samuel
Re: Fwd: Hurd shutdown problems
Samuel Thibault, on Fri 12 Aug 2016 17:17:26 +0200, wrote: > biosmem: heap: 114f000-7a00 > > and objdump shows: > > LOAD off0x1000 vaddr 0x8100 paddr 0x0100 align 2**12 > filesz 0x00114700 memsz 0x00114700 flags r-x > LOAD off0x00116000 vaddr 0x81115000 paddr 0x01115000 align 2**12 > filesz 0xd151 memsz 0x00039384 flags rw- > > It seems biosmem's heap doesn't even exclude the kernel data?! Ah, sorry, it does, I misread it. But it doesn't exclude anything else, apparently. Samuel
Re: Fwd: Hurd shutdown problems
Brent W. Baccala, on Thu 11 Aug 2016 15:29:27 -1000, wrote: > On Wed, Aug 10, 2016 at 4:33 AM, Richard Braun <[1]rbr...@sceen.net> wrote: > > On Wed, Aug 10, 2016 at 04:26:35PM +0200, Richard Braun wrote: > > the boot loader (see MULTIBOOT_FLAGS in boothdr.S), and at > > some point, late during the boot process, module data are freed > > using (see free_bootstrap_pages in bootstrap.c). This might > > Using vm_page_manage(). > > The symbol table is far enough away from the module data that I don't think > it's getting freed at that point. Ok, but: > But it does seem to be freed. Please check my calculations. > > Here's the location of the symbol table in virtual memory. > > (gdb) print self->start > $15 = (Elf32_Sym *) 0x804fb5ec That's very far. i386/intel/pmap.c's pmap_bootstrap uses the etext symbol so as to know what to make read-only, notably. We'd probably want to make the symbol table read-only too. Looking at the output of the biosmem code on my box: biosmem: physical memory map: biosmem: 00:09fc00, available biosmem: 09fc00:0a, reserved biosmem: 0f:10, reserved biosmem: 10:007ffe, available biosmem: 007ffe:008000, reserved biosmem: 00feffc000:00ff00, reserved biosmem: 00fffc:01, reserved biosmem: heap: 114f000-7a00 and objdump shows: LOAD off0x1000 vaddr 0x8100 paddr 0x0100 align 2**12 filesz 0x00114700 memsz 0x00114700 flags r-x LOAD off0x00116000 vaddr 0x81115000 paddr 0x01115000 align 2**12 filesz 0xd151 memsz 0x00039384 flags rw- It seems biosmem's heap doesn't even exclude the kernel data?! Samuel
