I'll give your suggestions I try. I need to move to a server that I can
forcibly reboot remotely (to recover), so it will be a while.
david
Avi Kivity wrote:
> david ahern wrote:
>> As a quick test I added a printk to the loop, right after the while():
>>
>> while (atomic_read(&completed) != needed) {
>> printk("kvm_flush_remote_tlbs: completed = %d, needed = %d\n",
>> atomic_read(&completed), needed);
>> cpu_relax();
>> barrier();
>> }
>>
>>
>> This is the output right before a lockup:
>>
>> Oct 24 16:03:47 bldr-ccm20 kernel: kvm_flush_remote_tlbs: completed = 2,
>> needed = 2
>> Oct 24 16:03:47 bldr-ccm20 kernel: kvm_flush_remote_tlbs: completed = 2,
>> needed = 2
>> Oct 24 16:03:47 bldr-ccm20 kernel: kvm_flush_remote_tlbs: completed = 1,
>> needed = 2
>> Oct 24 16:03:57 bldr-ccm20 last message repeated 105738 times
>> Oct 24 16:03:57 bldr-ccm20 kernel: BUG: soft lockup detected on CPU#0!
>> Oct 24 16:03:57 bldr-ccm20 kernel: [<c044a0b7>] softlockup_tick+0x98/0xa6
>> Oct 24 16:03:57 bldr-ccm20 kernel: [<c042cc98>]
>> update_process_times+0x39/0x5c
>> Oct 24 16:03:57 bldr-ccm20 kernel: [<c04176ec>]
>> smp_apic_timer_interrupt+0x5c/0x64
>> Oct 24 16:03:57 bldr-ccm20 kernel: [<c04049bf>]
>> apic_timer_interrupt+0x1f/0x24
>> Oct 24 16:03:57 bldr-ccm20 kernel: [<c0424130>] vprintk+0x288/0x2bc
>> Oct 24 16:03:57 bldr-ccm20 kernel: [<c0459db7>] follow_page+0x168/0x1b6
>> Oct 24 16:03:57 bldr-ccm20 kernel: [<c04d8067>]
>> cfq_slice_async_store+0x5/0x38
>> Oct 24 16:03:57 bldr-ccm20 kernel: [<c0459db7>] follow_page+0x168/0x1b6
>> Oct 24 16:03:57 bldr-ccm20 kernel: [<c0406406>] do_IRQ+0xa5/0xae
>> Oct 24 16:03:57 bldr-ccm20 kernel: [<c040492e>] common_interrupt+0x1a/0x20
>> Oct 24 16:03:57 bldr-ccm20 kernel: [<c042417c>] printk+0x18/0x8e
>> Oct 24 16:03:57 bldr-ccm20 kernel: [<f89a9812>]
>> kvm_flush_remote_tlbs+0xe0/0xf2 [kvm]
>> ...
>>
>>
>> I'd like to get a solution for RHEL5, so I am attempting to backport
>> smp_call_function_mask(). I'm open to other suggestions if you think it is
>> corruption or the problem is somewhere else.
>>
>
> No, it looks like the problem is indeed in kvm_flush_remote_tlbs(), and
> not a corruption elsewhere.
>
> Things to check:
>
> - whether cpus_weight(mask) == needed
> - whether wrapping the whole thing in preempt_disable()/preempt_enable()
> helps
>
> hey! I see a bug!
>
>> continue;
>> cpu = vcpu->cpu;
>> if (cpu != -1 && cpu != raw_smp_processor_id())
>> if (!cpu_isset(cpu, cpus)) {
>> cpu_set(cpu, cpus);
>> ++needed;
>> }
>> }
>>
>
> vcpu->cpu can change during execution if this snippet due to a vcpu
> being migrated concurrently with this being executed. Since the
> compiler is free to reload 'cpu' from 'vcpu->cpu', the code can operate
> on corrupted data.
>
> A 'barrier();' after 'cpu = vcpu->cpu;' should fix it, if this is indeed
> the bug.
>
>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
kvm-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel
