Re: tsc timer related problems/questions
Also thanks to all the others for their reply! Am Montag, den 10.09.2007, 19:19 -0600 schrieb Robert Hancock: > Dennis Lubert wrote: > > Hello list, > > [105771.581838] [] start_secondary+0x474/0x483 > > > > Question: Is this a known bug already or should further investigation > > take place? > > It's unclear what that could be. As Arjan mentioned this can be caused > by the BIOS going off into SMI mode for a long time. If you don't have > ACPI turned on, doing this may prevent this from happening. ACPI is turned on. As I have no experience with what SMI mode is, or how it could be entered, is there a way to figure that out? When that BUG happens the system is running one of our tests which involves lots of calls to gettimeofday and/or clock_gettime on all cores, all being constantly under 100% cpu usage. There does not seem to be anything notable different in that situation. > What time source is getting used? The best alternative is HPET, most > newer systems are providing that now. After that, there's ACPI PM timer > (make sure you have ACPI enabled). The worst possible fallback is the > PIT, which from this poor resolution sounds like what it is using. The systems all do not have HPET unfortunately. Using the timesource "jiffies" is the worst fallback (giving HZ resolution), I assume thats driven by PIT. Using pmtimer is also a suboptimal choice, as its takes (depending on the system) between 8 and 12 times more time than the tsc based calls, which sometimes gets into 2µs per call, much more than the optimal resolution of <1µs, and that also gets some apps to use significant more CPU for the timer calls, as an example one of our proxy apps needs to check timeouts of internal data very very often and on the bad pmtimer machines it spends up to 40% in the gettimeofday calls. > AMD CPUs don't seem to have synchronized TSCs across multiple CPUs. It > seems this is the case even with different cores in the same CPU > package. Therefore the TSC is not considered a suitable time source on > multi-CPU AMD systems. Ok, I hope you won't get angry if I try to get an official statement from AMD about this, it seems a bit strange to me ;) Anyways, I got some time-warp-test.c program written by Ingo Molnar which should check TSC synchronity on the CPUs. Running that program for a while (30 minutes) did not lead to any negative results. Does anyone have experiences with this program, and under what conditions it should be run? and how long? > This is expected behavior if you force TSC usage on with CPU frequency > scaling enabled, there's a reason we turn that off normally. (Also in > the case where some CPUs stop the TSC in certain power-saving halt > states.) Theoretically one could track the TSC between different CPUs > running at different clock speeds, etc. and across halts, but it doesn't > really seem worth the trouble, especially in cases like AMD multi-CPU > where the TSC can't be trusted across CPUs anyway. I found some patch attempts in various places, e.g. one by Suleiman Souhlal posted on this list that try to this and similar things. Wouldn't those patches (although seemingly suboptimal) not be better than nothing? As I described earlier, it wouldn't be that much of a problem to have small differences (one could try syncing the tsc every now and then) for us, one of the main concerns (beside an "ok" accuracy) is the performance, a variante thats 8-12 times slower is (hopefully understandable) very suboptimal greets Dennis - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: tsc timer related problems/questions
Also thanks to all the others for their reply! Am Montag, den 10.09.2007, 19:19 -0600 schrieb Robert Hancock: Dennis Lubert wrote: Hello list, [105771.581838] [806bc5f7] start_secondary+0x474/0x483 Question: Is this a known bug already or should further investigation take place? It's unclear what that could be. As Arjan mentioned this can be caused by the BIOS going off into SMI mode for a long time. If you don't have ACPI turned on, doing this may prevent this from happening. ACPI is turned on. As I have no experience with what SMI mode is, or how it could be entered, is there a way to figure that out? When that BUG happens the system is running one of our tests which involves lots of calls to gettimeofday and/or clock_gettime on all cores, all being constantly under 100% cpu usage. There does not seem to be anything notable different in that situation. What time source is getting used? The best alternative is HPET, most newer systems are providing that now. After that, there's ACPI PM timer (make sure you have ACPI enabled). The worst possible fallback is the PIT, which from this poor resolution sounds like what it is using. The systems all do not have HPET unfortunately. Using the timesource jiffies is the worst fallback (giving HZ resolution), I assume thats driven by PIT. Using pmtimer is also a suboptimal choice, as its takes (depending on the system) between 8 and 12 times more time than the tsc based calls, which sometimes gets into 2µs per call, much more than the optimal resolution of 1µs, and that also gets some apps to use significant more CPU for the timer calls, as an example one of our proxy apps needs to check timeouts of internal data very very often and on the bad pmtimer machines it spends up to 40% in the gettimeofday calls. AMD CPUs don't seem to have synchronized TSCs across multiple CPUs. It seems this is the case even with different cores in the same CPU package. Therefore the TSC is not considered a suitable time source on multi-CPU AMD systems. Ok, I hope you won't get angry if I try to get an official statement from AMD about this, it seems a bit strange to me ;) Anyways, I got some time-warp-test.c program written by Ingo Molnar which should check TSC synchronity on the CPUs. Running that program for a while (30 minutes) did not lead to any negative results. Does anyone have experiences with this program, and under what conditions it should be run? and how long? This is expected behavior if you force TSC usage on with CPU frequency scaling enabled, there's a reason we turn that off normally. (Also in the case where some CPUs stop the TSC in certain power-saving halt states.) Theoretically one could track the TSC between different CPUs running at different clock speeds, etc. and across halts, but it doesn't really seem worth the trouble, especially in cases like AMD multi-CPU where the TSC can't be trusted across CPUs anyway. I found some patch attempts in various places, e.g. one by Suleiman Souhlal posted on this list that try to this and similar things. Wouldn't those patches (although seemingly suboptimal) not be better than nothing? As I described earlier, it wouldn't be that much of a problem to have small differences (one could try syncing the tsc every now and then) for us, one of the main concerns (beside an ok accuracy) is the performance, a variante thats 8-12 times slower is (hopefully understandable) very suboptimal greets Dennis - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: tsc timer related problems/questions
Dennis Lubert wrote: Hello list, we are encountering a few behaviours regarding the ways to get accurate timer values under Linux that we would call bugs, and where we are currently stuck in further diagnosing and/or fixing. Background: We are developing for SMP servers with up to 8 CPUs (mostly AMD64) and for various reasons would like to have time measurements with a resolution of maybe a few microseconds. - Using Kernel 2.6.20.7 and surroundings per default the TSC Timer is used. We are very happy with that (accuracy ~400nanoseconds) but after a while the system goes wild with the following message for each CPU: [105771.523771] BUG: soft lockup detected on CPU#1! [105771.527869] [105771.527871] Call Trace: [105771.536079][] _spin_lock+0x9/0xb [105771.540294] [] softlockup_tick+0xd2/0xe7 [105771.544359] [] run_local_timers+0x13/0x15 [105771.548541] [] update_process_times+0x4c/0x79 [105771.552737] [] smp_local_timer_interrupt +0x34/0x54 [105771.556934] [] smp_apic_timer_interrupt+0x51/0x68 [105771.561022] [] default_idle+0x0/0x42 [105771.565199] [] apic_timer_interrupt+0x66/0x70 [105771.569386][] default_idle+0x2d/0x42 [105771.573597] [] enter_idle+0x22/0x24 [105771.577665] [] cpu_idle+0x5a/0x79 [105771.581838] [] start_secondary+0x474/0x483 Question: Is this a known bug already or should further investigation take place? It's unclear what that could be. As Arjan mentioned this can be caused by the BIOS going off into SMI mode for a long time. If you don't have ACPI turned on, doing this may prevent this from happening. - Using Kernels from 2.6.21 on (random sampled) we experience that the TSC isn't used per default anymore (we usually set the nopmtimer option at boot for a while now). Looking briefly at the 2.6.23-rc5 code shows that in the function where the check is done whether the tsc is stable the only code path where a "is stable" result could be returned is one where the vendor of the CPU is detected as Intel. Instead a much slower timesource (10ms instead of a few us resolution, same for getting the time at all) is used which is totally unusable for us (Within 10ms so much things happen). What time source is getting used? The best alternative is HPET, most newer systems are providing that now. After that, there's ACPI PM timer (make sure you have ACPI enabled). The worst possible fallback is the PIT, which from this poor resolution sounds like what it is using. Question: Why are only Intel CPUs considered as stable? Could there be implemented a more sophisticated heuristic, that actually does some tests for tsc stability? AMD CPUs don't seem to have synchronized TSCs across multiple CPUs. It seems this is the case even with different cores in the same CPU package. Therefore the TSC is not considered a suitable time source on multi-CPU AMD systems. - Enabling tsc explicitly as a time source via sysfs we had good results so far, with quit good resolution, and also various tests about synchronization between the CPUs didn't show any measurable changes in the deviation over time. However, once accidentally someone enabled cpufrequency scaling and scaled down two of four CPUs. From then on the time on the slower CPU was totally wrong, and all time displaying programs (simple date program) showed different (hours in difference) results, depending on which CPU they where run, so results were randomly. Programs doing a simple usleep() could hang (likely because the time to wakeup was gathered from another CPU whith time in the future). The system was essentially unusable and also after setting the CPUs back to the correct speed, things were still wrong. Question: Is this a known problem? It looks like there is a huge problem in synchronizing the way the time is calculated from the TSC and the cpu frequency scaling, also something else seems to be buggy since also after setting things back even after a few seconds only, times are off by hours. This is expected behavior if you force TSC usage on with CPU frequency scaling enabled, there's a reason we turn that off normally. (Also in the case where some CPUs stop the TSC in certain power-saving halt states.) Theoretically one could track the TSC between different CPUs running at different clock speeds, etc. and across halts, but it doesn't really seem worth the trouble, especially in cases like AMD multi-CPU where the TSC can't be trusted across CPUs anyway. -- Robert Hancock Saskatoon, SK, Canada To email, remove "nospam" from [EMAIL PROTECTED] Home Page: http://www.roberthancock.com/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: tsc timer related problems/questions
Dennis Lubert wrote: Hello list, we are encountering a few behaviours regarding the ways to get accurate timer values under Linux that we would call bugs, and where we are currently stuck in further diagnosing and/or fixing. Background: We are developing for SMP servers with up to 8 CPUs (mostly AMD64) and for various reasons would like to have time measurements with a resolution of maybe a few microseconds. - Using Kernel 2.6.20.7 and surroundings per default the TSC Timer is used. We are very happy with that (accuracy ~400nanoseconds) but after a while the system goes wild with the following message for each CPU: [105771.523771] BUG: soft lockup detected on CPU#1! [105771.527869] [105771.527871] Call Trace: [105771.536079] IRQ [802619cc] _spin_lock+0x9/0xb [105771.540294] [802a6f9d] softlockup_tick+0xd2/0xe7 [105771.544359] [8024bcbb] run_local_timers+0x13/0x15 [105771.548541] [80289fc1] update_process_times+0x4c/0x79 [105771.552737] [80270327] smp_local_timer_interrupt +0x34/0x54 [105771.556934] [80270834] smp_apic_timer_interrupt+0x51/0x68 [105771.561022] [80268121] default_idle+0x0/0x42 [105771.565199] [8025cce6] apic_timer_interrupt+0x66/0x70 [105771.569386] EOI [8026814e] default_idle+0x2d/0x42 [105771.573597] [80247929] enter_idle+0x22/0x24 [105771.577665] [80247a92] cpu_idle+0x5a/0x79 [105771.581838] [806bc5f7] start_secondary+0x474/0x483 Question: Is this a known bug already or should further investigation take place? It's unclear what that could be. As Arjan mentioned this can be caused by the BIOS going off into SMI mode for a long time. If you don't have ACPI turned on, doing this may prevent this from happening. - Using Kernels from 2.6.21 on (random sampled) we experience that the TSC isn't used per default anymore (we usually set the nopmtimer option at boot for a while now). Looking briefly at the 2.6.23-rc5 code shows that in the function where the check is done whether the tsc is stable the only code path where a is stable result could be returned is one where the vendor of the CPU is detected as Intel. Instead a much slower timesource (10ms instead of a few us resolution, same for getting the time at all) is used which is totally unusable for us (Within 10ms so much things happen). What time source is getting used? The best alternative is HPET, most newer systems are providing that now. After that, there's ACPI PM timer (make sure you have ACPI enabled). The worst possible fallback is the PIT, which from this poor resolution sounds like what it is using. Question: Why are only Intel CPUs considered as stable? Could there be implemented a more sophisticated heuristic, that actually does some tests for tsc stability? AMD CPUs don't seem to have synchronized TSCs across multiple CPUs. It seems this is the case even with different cores in the same CPU package. Therefore the TSC is not considered a suitable time source on multi-CPU AMD systems. - Enabling tsc explicitly as a time source via sysfs we had good results so far, with quit good resolution, and also various tests about synchronization between the CPUs didn't show any measurable changes in the deviation over time. However, once accidentally someone enabled cpufrequency scaling and scaled down two of four CPUs. From then on the time on the slower CPU was totally wrong, and all time displaying programs (simple date program) showed different (hours in difference) results, depending on which CPU they where run, so results were randomly. Programs doing a simple usleep() could hang (likely because the time to wakeup was gathered from another CPU whith time in the future). The system was essentially unusable and also after setting the CPUs back to the correct speed, things were still wrong. Question: Is this a known problem? It looks like there is a huge problem in synchronizing the way the time is calculated from the TSC and the cpu frequency scaling, also something else seems to be buggy since also after setting things back even after a few seconds only, times are off by hours. This is expected behavior if you force TSC usage on with CPU frequency scaling enabled, there's a reason we turn that off normally. (Also in the case where some CPUs stop the TSC in certain power-saving halt states.) Theoretically one could track the TSC between different CPUs running at different clock speeds, etc. and across halts, but it doesn't really seem worth the trouble, especially in cases like AMD multi-CPU where the TSC can't be trusted across CPUs anyway. -- Robert Hancock Saskatoon, SK, Canada To email, remove nospam from [EMAIL PROTECTED] Home Page: http://www.roberthancock.com/ - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at
Re: tsc timer related problems/questions
On Sun, 9 Sep 2007 20:17:28 +0200 (CEST) Jan Engelhardt <[EMAIL PROTECTED]> wrote: > > On Sep 9 2007 17:49, Arjan van de Ven wrote: > >> > >> Question: Why are only Intel CPUs considered as stable? Could > >> there be implemented a more sophisticated heuristic, that actually > >> does some tests for tsc stability? > > > >on AMD multi-socket systems, afaik the tsc is not synchronized > >between packages. On Intel the tsc is derived from the FSB which is > >shared between the packages. > > Also, the TSC is not necessarily constant wrt. CPU clock speed. > If your program stalls, the core may reduce frequency and hence > TSC values are not linear to time anymore. > that depends on the CPU generation; most current cpus have constant-rate tscs (at least constant wrt their own concept of time, but at least independent of cpu frequency)... it gets more iffy on idle; on deep idle the tsc tends to stop ;( - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: tsc timer related problems/questions
On Sep 9 2007 17:49, Arjan van de Ven wrote: >> >> Question: Why are only Intel CPUs considered as stable? Could there be >> implemented a more sophisticated heuristic, that actually does some >> tests for tsc stability? > >on AMD multi-socket systems, afaik the tsc is not synchronized between >packages. On Intel the tsc is derived from the FSB which is shared >between the packages. Also, the TSC is not necessarily constant wrt. CPU clock speed. If your program stalls, the core may reduce frequency and hence TSC values are not linear to time anymore. Jan -- - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: tsc timer related problems/questions
On Sun, 09 Sep 2007 18:31:45 +0200 Dennis Lubert <[EMAIL PROTECTED]> wrote: Hi, > > [105771.523771] BUG: soft lockup detected on CPU#1! > [105771.527869] > [105771.527871] Call Trace: > [105771.536079][] _spin_lock+0x9/0xb > [105771.540294] [] softlockup_tick+0xd2/0xe7 > [105771.544359] [] run_local_timers+0x13/0x15 > [105771.548541] [] update_process_times+0x4c/0x79 > [105771.552737] [] smp_local_timer_interrupt > +0x34/0x54 > [105771.556934] [] > smp_apic_timer_interrupt+0x51/0x68 [105771.561022] > [] default_idle+0x0/0x42 [105771.565199] > [] apic_timer_interrupt+0x66/0x70 [105771.569386] > [] default_idle+0x2d/0x42 [105771.573597] > [] enter_idle+0x22/0x24 [105771.577665] > [] cpu_idle+0x5a/0x79 [105771.581838] > [] start_secondary+0x474/0x483 > > Question: Is this a known bug already or should further investigation > take place? this needs further investigation but note that this one can have many different causes; this kind of oops trace often is also found for certain types of BIOS bugs. (which makes it really painful to debug) > > - Using Kernels from 2.6.21 on (random sampled) we experience that the > TSC isn't used per default anymore (we usually set the nopmtimer > option at boot for a while now). Looking briefly at the 2.6.23-rc5 > code shows that in the function where the check is done whether the > tsc is stable the only code path where a "is stable" result could be > returned is one where the vendor of the CPU is detected as Intel. > Instead a much slower timesource (10ms instead of a few us > resolution, same for getting the time at all) is used which is > totally unusable for us (Within 10ms so much things happen). Please consider enabling the HPET in your bios; while it's not as fast as the tsc, it does provide the accuracy you want. > > Question: Why are only Intel CPUs considered as stable? Could there be > implemented a more sophisticated heuristic, that actually does some > tests for tsc stability? on AMD multi-socket systems, afaik the tsc is not synchronized between packages. On Intel the tsc is derived from the FSB which is shared between the packages. HPET is central to the system and solves all the issues, but is a tad slower to access than the tsc. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
tsc timer related problems/questions
Hello list, we are encountering a few behaviours regarding the ways to get accurate timer values under Linux that we would call bugs, and where we are currently stuck in further diagnosing and/or fixing. Background: We are developing for SMP servers with up to 8 CPUs (mostly AMD64) and for various reasons would like to have time measurements with a resolution of maybe a few microseconds. - Using Kernel 2.6.20.7 and surroundings per default the TSC Timer is used. We are very happy with that (accuracy ~400nanoseconds) but after a while the system goes wild with the following message for each CPU: [105771.523771] BUG: soft lockup detected on CPU#1! [105771.527869] [105771.527871] Call Trace: [105771.536079][] _spin_lock+0x9/0xb [105771.540294] [] softlockup_tick+0xd2/0xe7 [105771.544359] [] run_local_timers+0x13/0x15 [105771.548541] [] update_process_times+0x4c/0x79 [105771.552737] [] smp_local_timer_interrupt +0x34/0x54 [105771.556934] [] smp_apic_timer_interrupt+0x51/0x68 [105771.561022] [] default_idle+0x0/0x42 [105771.565199] [] apic_timer_interrupt+0x66/0x70 [105771.569386][] default_idle+0x2d/0x42 [105771.573597] [] enter_idle+0x22/0x24 [105771.577665] [] cpu_idle+0x5a/0x79 [105771.581838] [] start_secondary+0x474/0x483 Question: Is this a known bug already or should further investigation take place? - Using Kernels from 2.6.21 on (random sampled) we experience that the TSC isn't used per default anymore (we usually set the nopmtimer option at boot for a while now). Looking briefly at the 2.6.23-rc5 code shows that in the function where the check is done whether the tsc is stable the only code path where a "is stable" result could be returned is one where the vendor of the CPU is detected as Intel. Instead a much slower timesource (10ms instead of a few us resolution, same for getting the time at all) is used which is totally unusable for us (Within 10ms so much things happen). Question: Why are only Intel CPUs considered as stable? Could there be implemented a more sophisticated heuristic, that actually does some tests for tsc stability? - Enabling tsc explicitly as a time source via sysfs we had good results so far, with quit good resolution, and also various tests about synchronization between the CPUs didn't show any measurable changes in the deviation over time. However, once accidentally someone enabled cpufrequency scaling and scaled down two of four CPUs. From then on the time on the slower CPU was totally wrong, and all time displaying programs (simple date program) showed different (hours in difference) results, depending on which CPU they where run, so results were randomly. Programs doing a simple usleep() could hang (likely because the time to wakeup was gathered from another CPU whith time in the future). The system was essentially unusable and also after setting the CPUs back to the correct speed, things were still wrong. Question: Is this a known problem? It looks like there is a huge problem in synchronizing the way the time is calculated from the TSC and the cpu frequency scaling, also something else seems to be buggy since also after setting things back even after a few seconds only, times are off by hours. Is there maybe a mechanism (or could it be implemented) that synchronizes the TSCs on demand? It usually isn't a huge problem if they are off a few nanoseconds, maybe even a few microseconds. For quite some programs they could even be off a few hundred microseconds, so a synchronization every now and then could still be useful. greets Dennis - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
tsc timer related problems/questions
Hello list, we are encountering a few behaviours regarding the ways to get accurate timer values under Linux that we would call bugs, and where we are currently stuck in further diagnosing and/or fixing. Background: We are developing for SMP servers with up to 8 CPUs (mostly AMD64) and for various reasons would like to have time measurements with a resolution of maybe a few microseconds. - Using Kernel 2.6.20.7 and surroundings per default the TSC Timer is used. We are very happy with that (accuracy ~400nanoseconds) but after a while the system goes wild with the following message for each CPU: [105771.523771] BUG: soft lockup detected on CPU#1! [105771.527869] [105771.527871] Call Trace: [105771.536079] IRQ [802619cc] _spin_lock+0x9/0xb [105771.540294] [802a6f9d] softlockup_tick+0xd2/0xe7 [105771.544359] [8024bcbb] run_local_timers+0x13/0x15 [105771.548541] [80289fc1] update_process_times+0x4c/0x79 [105771.552737] [80270327] smp_local_timer_interrupt +0x34/0x54 [105771.556934] [80270834] smp_apic_timer_interrupt+0x51/0x68 [105771.561022] [80268121] default_idle+0x0/0x42 [105771.565199] [8025cce6] apic_timer_interrupt+0x66/0x70 [105771.569386] EOI [8026814e] default_idle+0x2d/0x42 [105771.573597] [80247929] enter_idle+0x22/0x24 [105771.577665] [80247a92] cpu_idle+0x5a/0x79 [105771.581838] [806bc5f7] start_secondary+0x474/0x483 Question: Is this a known bug already or should further investigation take place? - Using Kernels from 2.6.21 on (random sampled) we experience that the TSC isn't used per default anymore (we usually set the nopmtimer option at boot for a while now). Looking briefly at the 2.6.23-rc5 code shows that in the function where the check is done whether the tsc is stable the only code path where a is stable result could be returned is one where the vendor of the CPU is detected as Intel. Instead a much slower timesource (10ms instead of a few us resolution, same for getting the time at all) is used which is totally unusable for us (Within 10ms so much things happen). Question: Why are only Intel CPUs considered as stable? Could there be implemented a more sophisticated heuristic, that actually does some tests for tsc stability? - Enabling tsc explicitly as a time source via sysfs we had good results so far, with quit good resolution, and also various tests about synchronization between the CPUs didn't show any measurable changes in the deviation over time. However, once accidentally someone enabled cpufrequency scaling and scaled down two of four CPUs. From then on the time on the slower CPU was totally wrong, and all time displaying programs (simple date program) showed different (hours in difference) results, depending on which CPU they where run, so results were randomly. Programs doing a simple usleep() could hang (likely because the time to wakeup was gathered from another CPU whith time in the future). The system was essentially unusable and also after setting the CPUs back to the correct speed, things were still wrong. Question: Is this a known problem? It looks like there is a huge problem in synchronizing the way the time is calculated from the TSC and the cpu frequency scaling, also something else seems to be buggy since also after setting things back even after a few seconds only, times are off by hours. Is there maybe a mechanism (or could it be implemented) that synchronizes the TSCs on demand? It usually isn't a huge problem if they are off a few nanoseconds, maybe even a few microseconds. For quite some programs they could even be off a few hundred microseconds, so a synchronization every now and then could still be useful. greets Dennis - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: tsc timer related problems/questions
On Sun, 09 Sep 2007 18:31:45 +0200 Dennis Lubert [EMAIL PROTECTED] wrote: Hi, [105771.523771] BUG: soft lockup detected on CPU#1! [105771.527869] [105771.527871] Call Trace: [105771.536079] IRQ [802619cc] _spin_lock+0x9/0xb [105771.540294] [802a6f9d] softlockup_tick+0xd2/0xe7 [105771.544359] [8024bcbb] run_local_timers+0x13/0x15 [105771.548541] [80289fc1] update_process_times+0x4c/0x79 [105771.552737] [80270327] smp_local_timer_interrupt +0x34/0x54 [105771.556934] [80270834] smp_apic_timer_interrupt+0x51/0x68 [105771.561022] [80268121] default_idle+0x0/0x42 [105771.565199] [8025cce6] apic_timer_interrupt+0x66/0x70 [105771.569386] EOI [8026814e] default_idle+0x2d/0x42 [105771.573597] [80247929] enter_idle+0x22/0x24 [105771.577665] [80247a92] cpu_idle+0x5a/0x79 [105771.581838] [806bc5f7] start_secondary+0x474/0x483 Question: Is this a known bug already or should further investigation take place? this needs further investigation but note that this one can have many different causes; this kind of oops trace often is also found for certain types of BIOS bugs. (which makes it really painful to debug) - Using Kernels from 2.6.21 on (random sampled) we experience that the TSC isn't used per default anymore (we usually set the nopmtimer option at boot for a while now). Looking briefly at the 2.6.23-rc5 code shows that in the function where the check is done whether the tsc is stable the only code path where a is stable result could be returned is one where the vendor of the CPU is detected as Intel. Instead a much slower timesource (10ms instead of a few us resolution, same for getting the time at all) is used which is totally unusable for us (Within 10ms so much things happen). Please consider enabling the HPET in your bios; while it's not as fast as the tsc, it does provide the accuracy you want. Question: Why are only Intel CPUs considered as stable? Could there be implemented a more sophisticated heuristic, that actually does some tests for tsc stability? on AMD multi-socket systems, afaik the tsc is not synchronized between packages. On Intel the tsc is derived from the FSB which is shared between the packages. HPET is central to the system and solves all the issues, but is a tad slower to access than the tsc. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: tsc timer related problems/questions
On Sep 9 2007 17:49, Arjan van de Ven wrote: Question: Why are only Intel CPUs considered as stable? Could there be implemented a more sophisticated heuristic, that actually does some tests for tsc stability? on AMD multi-socket systems, afaik the tsc is not synchronized between packages. On Intel the tsc is derived from the FSB which is shared between the packages. Also, the TSC is not necessarily constant wrt. CPU clock speed. If your program stalls, the core may reduce frequency and hence TSC values are not linear to time anymore. Jan -- - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: tsc timer related problems/questions
On Sun, 9 Sep 2007 20:17:28 +0200 (CEST) Jan Engelhardt [EMAIL PROTECTED] wrote: On Sep 9 2007 17:49, Arjan van de Ven wrote: Question: Why are only Intel CPUs considered as stable? Could there be implemented a more sophisticated heuristic, that actually does some tests for tsc stability? on AMD multi-socket systems, afaik the tsc is not synchronized between packages. On Intel the tsc is derived from the FSB which is shared between the packages. Also, the TSC is not necessarily constant wrt. CPU clock speed. If your program stalls, the core may reduce frequency and hence TSC values are not linear to time anymore. that depends on the CPU generation; most current cpus have constant-rate tscs (at least constant wrt their own concept of time, but at least independent of cpu frequency)... it gets more iffy on idle; on deep idle the tsc tends to stop ;( - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
