Hello Chris, On Monday 17 of October 2016 03:00:56 Chris Johns wrote: > I have built and tested these patches with no issues and 0 spurious > interrupts. > > The changes look fine to me.
Thanks much for testing. I have pushed changes to the master. I have spent considerable time over weekend by debugging i386 SMP build but without success. I have found and fixed some problems, I get to the state when two CPUs runs under QEMU and even IPI is exchanged but then execution finish by different kinds of fatal exceptions. On the other hand, SMP build when only one CPU is enabled runs stable under QEMU. Some found issues and corrections tested with CONFIGURE_SMP_MAXIMUM_PROCESSORS 2 According to the more sources, INIT and SIPI duplication is not required on modern CPUs. INIT is ignored so no problem, but if the second SIPI is delayed too much then the first one causes complete start of the second core reaching RTEMS secondary_cpu_initialize() and then next SIPI resets the core and it starts again so you count CPU twice. The specification is strict about 200usec between two SIPI. But according to some discussion that second SIPI is not required on modern systems. Next modification tries bring up the second core by single SIPI and if the core does not reach RTEMS code in really long time it expect old system and restries the second SIPI. Probably LAPIC version can be used there to switch this workaround for newer cores at all. diff --combined c/src/lib/libbsp/i386/shared/smp/smp-imps.c --- a/c/src/lib/libbsp/i386/shared/smp/smp-imps.c +++ b/c/src/lib/libbsp/i386/shared/smp/smp-imps.c @@@ -301,12 -301,12 +334,21 @@@ */ if (proc->apic_ver >= APIC_VER_NEW) { -- int i; -- for (i = 1; i <= 2; i++) { ++ int retry = 2; ++ while (retry) { ++ int wait_for_ap = 100; ++ ++ printk("boot_cpu sending SIPI\n"); ++ send_ipi(apicid, LAPIC_ICR_DM_SIPI | ((bootaddr >> 12) & 0xFF)); -- UDELAY(1000); ++ do { ++ UDELAY(10); ++ if (acked_ap_cpus != _Atomic_Load_uint(&imps_ap_cpus_acked, ATOMIC_ORDER_SEQ_CST)) ++ retry = 0; ++ } while(wait_for_ap--); } } ++ printk("boot_cpu secondary acked\n"); /* * Generic CPU startup sequence ends here, the rest is cleanup. Another strange thing for SMP is, that secondary CPUs continues to run with temporary GDT set by CPU SIPI trampoline code at 0x70000. So I have added function to setup the secondary CPU the same GDT as is used on the primary diff --combined c/src/lib/libbsp/i386/shared/smp/smp-imps.c --- a/c/src/lib/libbsp/i386/shared/smp/smp-imps.c +++ b/c/src/lib/libbsp/i386/shared/smp/smp-imps.c @@@ -785,7 -785,7 +829,7 @@@ static void secondary_cpu_initialize(vo { int apicid; -- asm volatile( "lidt IDT_Descriptor" ); ++ _load_segments_secondary(); apicid = IMPS_LAPIC_READ(LAPIC_SPIV); IMPS_LAPIC_WRITE(LAPIC_SPIV, apicid|LAPIC_SPIV_ENABLE_APIC); @@@ -794,6 -794,6 +838,8 @@@ enable_sse(); #endif ++ _Atomic_Fetch_add_uint(&imps_ap_cpus_acked, 1, ATOMIC_ORDER_SEQ_CST); ++ _SMP_Start_multitasking_on_secondary_processor(); } +++ b/c/src/lib/libbsp/i386/pc386/startup/ldsegs.S ++ ++ .p2align 4 ++ ++ PUBLIC ( _load_segments_secondary) ++SYM (_load_segments_secondary): ++ ++ lgdt SYM(gdtdesc) ++ lidt SYM(IDT_Descriptor) ++ ++ /* Load CS, flush prefetched queue */ ++ ljmp $0x8, $next_step_secondary ++ ++next_step_secondary: ++ /* Load segment registers */ ++ movw $0x10, ax ++ movw ax, ss ++ movw ax, ds ++ movw ax, es ++ movw ax, fs ++ movw ax, gs ++ ret As for GDT, I am not sure if one GDT for all CPUs can work. Problem is TSS. RTEMS runs in ring 0 only, so TSS is not required to switch between ring 3 and ring 0 but I am not sure if there is not some situation when TSS is accessed anyway. The additional TSS is required for sure if we want to catch double fault exception. To be on safe side, it would worth to move user descriptors to LDT, left global GDT to point to LDT and area for primary CPU TSS and allocate new GDT and TSS for each CPU during bringup. Generally, I am not expert for x86 and do not like this area much so if some student or somebody else wants to exercise in this remains of dark age, I would be happy and help with some hints. On the other hand, RTEMS should provide reasonable x6 SMP support for testing and may it be Jailhouse or other interresting hypervisor base divisions to RT and generic OS domains. Other option is to left i386 SMP support unsolved and add x86_64 support which skips many of these oddities like TSS and GDT. (OK GDT is required but only that simple short lived in the trampoline is enough). Best wishes, Pavel _______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel