Re: [Xen-devel] Help with: Xen BUG at timer.c:189

[Dietmar Hahn]

Am Mittwoch, 7. Juni 2017, 04:28:33 schrieb Jan Beulich:
> >>> On 07.06.17 at 11:28,  wrote:
> > Am Freitag, 19. Mai 2017, 06:41:36 schrieb Jan Beulich:
> >> >>> On 19.05.17 at 11:52,  wrote:
> >> > I'am struggling with a hypervisor panic. The hypervisor version is 4.4.3,
> >> > yes I know - very old ;-), but the affected code hasn't much changed.
> >> 
> >> Well, at the very least I'd expect you to base your code on
> >> 4.4.4 plus the about 199 backports we've added there over time
> >> (assuming yours is based on ours), some of which are scheduler
> >> ones.
> > 
> > Don't you have older versions in the field?
> 
> We certainly do, but when people run into problems first thing
> support is supposed to do is tell them to update.
> 
> >> > The panic was never seen before and occured while booting the system,
> >> > only dom0 was running.
> >> > 
> >> > (XEN) Xen BUG at timer.c:189
> >> > (XEN) [ Xen-4.4.3_02-26.2_x3  x86_64  debug=n  Tainted:C ]
> >> > (XEN) CPU:14
> >> > (XEN) RIP:e008:[] remove_entry+0x26/0x80
> >> > (XEN) RFLAGS: 00010086   CONTEXT: hypervisor
> >> > (XEN) rax: 00bc   rbx: 830839bcd5c8   rcx: 
> >> > 82d080343a80
> >> > (XEN) rdx: d5f0   rsi: 002887fa   rdi: 
> >> > 830839bcd5c8
> >> > (XEN) rbp: d5f0   rsp: 830839bbfe20   r8:  
> >> > 002887211398
> >> > (XEN) r9:  00288721132f   r10: bf8b2994   r11: 
> >> > 82d08011cfa0
> >> > (XEN) r12: 830839bcd5c8   r13: 82d080343a80   r14: 
> >> > d5f0
> >> > (XEN) r15: 82d080343a80   cr0: 8005003b   cr4: 
> >> > 001526f0
> >> > (XEN) cr3: 75679000   cr2: 0069b8ec
> >> > (XEN) ds: 002b   es: 002b   fs:    gs:    ss: e010   cs: e008
> >> > (XEN) Xen stack trace from rsp=830839bbfe20:
> >> > (XEN)830839bcb020 82d08012d3a8 002887fa 
> >> > 0286
> >> > (XEN)830839bcd5f0 830839bcd9c0 002887c1915d 
> >> > 000e
> >> > (XEN)82d080343a70 0002 830839bcda30 
> >> > 82d0801affd0
> >> > (XEN)0001000e 0028872113f4  
> >> > 82d08030fc00
> >> > (XEN)82d0802ffc00 82d080189a1c  
> >> > 
> >> > (XEN) 107800076e92 82d0801292aa 
> >> > 830839bb8000
> >> > (XEN)8300754fd000 82d08031abe0  
> >> > 830839bcb048
> >> > (XEN) 82d080161ad6 0001 
> >> > 
> >> > (XEN)   
> >> > 0009
> >> > (XEN)8803dda20010 0246  
> >> > fffe
> >> > (XEN)  800033aa 
> >> > 
> >> > (XEN) 0001 0100 
> >> > 800033aa
> >> > (XEN)e033 0246 8803dda21f08 
> >> > e02b
> >> > (XEN)   
> >> > 
> >> > (XEN)000e 8300754fd000 0037b9887700 
> >> > 
> >> > (XEN) Xen call trace:
> >> > (XEN)[] remove_entry+0x26/0x80
> >> > (XEN)[] set_timer+0x178/0x230
> >> > (XEN)[] mwait_idle+0x270/0x370
> >> > (XEN)[] local_time_calibration+0x19c/0x1a0
> >> > (XEN)[] __do_softirq+0x6a/0xa0
> >> > (XEN)[] idle_loop+0x16/0x50
> >> > (XEN) 
> >> > (XEN) 
> >> > (XEN) 
> >> > (XEN) Panic on CPU 14:
> >> > (XEN) Xen BUG at timer.c:189
> >> > (XEN) 
> >> > (XEN) 
> >> > (XEN) Reboot in five seconds...
> >> > (XEN) Executing kexec image on cpu14
> >> > (XEN) Shot down all CPUs
> >> > 
> >> > This is the call of BUG() in remove_entry() in case timer->status is not
> >> > TIMER_STATUS_in_heap or TIMER_STATUS_in_list.
> >> > 
> >> > With the address of the timer from the vmcore:
> >> > crash> struct timer 830839bcd5c8
> >> > struct timer {
> >> >   expires = 0,
> >> >   {
> >> > heap_offset = 968676784, 
> >> > list_next = 0x830839bcd5b0, 
> >> > inactive = {
> >> >   next = 0x830839bcd5b0, 
> >> >   prev = 0x20020020050
> >> > }
> >> >   }, 
> >> >   function = 0x830839bcd5e0, 
> >> >   data = 0x830839bcd5e0, 
> >> >   cpu = 54768, 
> >> >   status = 188 '\274'
> >> > }
> >> > 
> >> > This looks like the structure was overwritten.
> >> > 
> >> > The call path seems to be:
> >> >  mwait_idle()
> >> >sched_tick_resume()
> >> >  csched_tick_resume()
> >> >set_timer()
> >> >  remove_timer()
> >> > 
> >> > In sched_tick_resume():
> >> >  unsigned int cpu = smp_processor_id();
> >> > 
> >> > I extracted struct cpu_info from the stack and found:
> >> > crash> cpu_info 830839bbff18
> 

Re: [Xen-devel] Help with: Xen BUG at timer.c:189

[Jan Beulich]

>>> On 07.06.17 at 11:28,  wrote:
> Am Freitag, 19. Mai 2017, 06:41:36 schrieb Jan Beulich:
>> >>> On 19.05.17 at 11:52,  wrote:
>> > I'am struggling with a hypervisor panic. The hypervisor version is 4.4.3,
>> > yes I know - very old ;-), but the affected code hasn't much changed.
>> 
>> Well, at the very least I'd expect you to base your code on
>> 4.4.4 plus the about 199 backports we've added there over time
>> (assuming yours is based on ours), some of which are scheduler
>> ones.
> 
> Don't you have older versions in the field?

We certainly do, but when people run into problems first thing
support is supposed to do is tell them to update.

>> > The panic was never seen before and occured while booting the system,
>> > only dom0 was running.
>> > 
>> > (XEN) Xen BUG at timer.c:189
>> > (XEN) [ Xen-4.4.3_02-26.2_x3  x86_64  debug=n  Tainted:C ]
>> > (XEN) CPU:14
>> > (XEN) RIP:e008:[] remove_entry+0x26/0x80
>> > (XEN) RFLAGS: 00010086   CONTEXT: hypervisor
>> > (XEN) rax: 00bc   rbx: 830839bcd5c8   rcx: 82d080343a80
>> > (XEN) rdx: d5f0   rsi: 002887fa   rdi: 830839bcd5c8
>> > (XEN) rbp: d5f0   rsp: 830839bbfe20   r8:  002887211398
>> > (XEN) r9:  00288721132f   r10: bf8b2994   r11: 82d08011cfa0
>> > (XEN) r12: 830839bcd5c8   r13: 82d080343a80   r14: d5f0
>> > (XEN) r15: 82d080343a80   cr0: 8005003b   cr4: 001526f0
>> > (XEN) cr3: 75679000   cr2: 0069b8ec
>> > (XEN) ds: 002b   es: 002b   fs:    gs:    ss: e010   cs: e008
>> > (XEN) Xen stack trace from rsp=830839bbfe20:
>> > (XEN)830839bcb020 82d08012d3a8 002887fa 
>> > 0286
>> > (XEN)830839bcd5f0 830839bcd9c0 002887c1915d 
>> > 000e
>> > (XEN)82d080343a70 0002 830839bcda30 
>> > 82d0801affd0
>> > (XEN)0001000e 0028872113f4  
>> > 82d08030fc00
>> > (XEN)82d0802ffc00 82d080189a1c  
>> > 
>> > (XEN) 107800076e92 82d0801292aa 
>> > 830839bb8000
>> > (XEN)8300754fd000 82d08031abe0  
>> > 830839bcb048
>> > (XEN) 82d080161ad6 0001 
>> > 
>> > (XEN)   
>> > 0009
>> > (XEN)8803dda20010 0246  
>> > fffe
>> > (XEN)  800033aa 
>> > 
>> > (XEN) 0001 0100 
>> > 800033aa
>> > (XEN)e033 0246 8803dda21f08 
>> > e02b
>> > (XEN)   
>> > 
>> > (XEN)000e 8300754fd000 0037b9887700 
>> > 
>> > (XEN) Xen call trace:
>> > (XEN)[] remove_entry+0x26/0x80
>> > (XEN)[] set_timer+0x178/0x230
>> > (XEN)[] mwait_idle+0x270/0x370
>> > (XEN)[] local_time_calibration+0x19c/0x1a0
>> > (XEN)[] __do_softirq+0x6a/0xa0
>> > (XEN)[] idle_loop+0x16/0x50
>> > (XEN) 
>> > (XEN) 
>> > (XEN) 
>> > (XEN) Panic on CPU 14:
>> > (XEN) Xen BUG at timer.c:189
>> > (XEN) 
>> > (XEN) 
>> > (XEN) Reboot in five seconds...
>> > (XEN) Executing kexec image on cpu14
>> > (XEN) Shot down all CPUs
>> > 
>> > This is the call of BUG() in remove_entry() in case timer->status is not
>> > TIMER_STATUS_in_heap or TIMER_STATUS_in_list.
>> > 
>> > With the address of the timer from the vmcore:
>> > crash> struct timer 830839bcd5c8
>> > struct timer {
>> >   expires = 0,
>> >   {
>> > heap_offset = 968676784, 
>> > list_next = 0x830839bcd5b0, 
>> > inactive = {
>> >   next = 0x830839bcd5b0, 
>> >   prev = 0x20020020050
>> > }
>> >   }, 
>> >   function = 0x830839bcd5e0, 
>> >   data = 0x830839bcd5e0, 
>> >   cpu = 54768, 
>> >   status = 188 '\274'
>> > }
>> > 
>> > This looks like the structure was overwritten.
>> > 
>> > The call path seems to be:
>> >  mwait_idle()
>> >sched_tick_resume()
>> >  csched_tick_resume()
>> >set_timer()
>> >  remove_timer()
>> > 
>> > In sched_tick_resume():
>> >  unsigned int cpu = smp_processor_id();
>> > 
>> > I extracted struct cpu_info from the stack and found:
>> > crash> cpu_info 830839bbff18
>> > struct cpu_info {
>> >   ...
>> >   processor_id = 14, 
>> >   current_vcpu = 0x8300754fd000, 
>> >   per_cpu_offset = 0x37b9887700,
>> >   ...
>> > 
>> > And from the code in sched_credit.c:  spc = CSCHED_PCPU(cpu);
>> > I calculated spc:
>> > 
>> > crash> csched_pcpu 0x830839be2b40
>> > struct csched_pcpu {
>> >   runq = {

Re: [Xen-devel] Help with: Xen BUG at timer.c:189

[Dietmar Hahn]

Am Freitag, 19. Mai 2017, 06:41:36 schrieb Jan Beulich:
> >>> On 19.05.17 at 11:52,  wrote:
> > I'am struggling with a hypervisor panic. The hypervisor version is 4.4.3,
> > yes I know - very old ;-), but the affected code hasn't much changed.
> 
> Well, at the very least I'd expect you to base your code on
> 4.4.4 plus the about 199 backports we've added there over time
> (assuming yours is based on ours), some of which are scheduler
> ones.

Don't you have older versions in the field?

> 
> > The panic was never seen before and occured while booting the system,
> > only dom0 was running.
> > 
> > (XEN) Xen BUG at timer.c:189
> > (XEN) [ Xen-4.4.3_02-26.2_x3  x86_64  debug=n  Tainted:C ]
> > (XEN) CPU:14
> > (XEN) RIP:e008:[] remove_entry+0x26/0x80
> > (XEN) RFLAGS: 00010086   CONTEXT: hypervisor
> > (XEN) rax: 00bc   rbx: 830839bcd5c8   rcx: 82d080343a80
> > (XEN) rdx: d5f0   rsi: 002887fa   rdi: 830839bcd5c8
> > (XEN) rbp: d5f0   rsp: 830839bbfe20   r8:  002887211398
> > (XEN) r9:  00288721132f   r10: bf8b2994   r11: 82d08011cfa0
> > (XEN) r12: 830839bcd5c8   r13: 82d080343a80   r14: d5f0
> > (XEN) r15: 82d080343a80   cr0: 8005003b   cr4: 001526f0
> > (XEN) cr3: 75679000   cr2: 0069b8ec
> > (XEN) ds: 002b   es: 002b   fs:    gs:    ss: e010   cs: e008
> > (XEN) Xen stack trace from rsp=830839bbfe20:
> > (XEN)830839bcb020 82d08012d3a8 002887fa 0286
> > (XEN)830839bcd5f0 830839bcd9c0 002887c1915d 000e
> > (XEN)82d080343a70 0002 830839bcda30 82d0801affd0
> > (XEN)0001000e 0028872113f4  82d08030fc00
> > (XEN)82d0802ffc00 82d080189a1c  
> > (XEN) 107800076e92 82d0801292aa 830839bb8000
> > (XEN)8300754fd000 82d08031abe0  830839bcb048
> > (XEN) 82d080161ad6 0001 
> > (XEN)   0009
> > (XEN)8803dda20010 0246  fffe
> > (XEN)  800033aa 
> > (XEN) 0001 0100 800033aa
> > (XEN)e033 0246 8803dda21f08 e02b
> > (XEN)   
> > (XEN)000e 8300754fd000 0037b9887700 
> > (XEN) Xen call trace:
> > (XEN)[] remove_entry+0x26/0x80
> > (XEN)[] set_timer+0x178/0x230
> > (XEN)[] mwait_idle+0x270/0x370
> > (XEN)[] local_time_calibration+0x19c/0x1a0
> > (XEN)[] __do_softirq+0x6a/0xa0
> > (XEN)[] idle_loop+0x16/0x50
> > (XEN) 
> > (XEN) 
> > (XEN) 
> > (XEN) Panic on CPU 14:
> > (XEN) Xen BUG at timer.c:189
> > (XEN) 
> > (XEN) 
> > (XEN) Reboot in five seconds...
> > (XEN) Executing kexec image on cpu14
> > (XEN) Shot down all CPUs
> > 
> > This is the call of BUG() in remove_entry() in case timer->status is not
> > TIMER_STATUS_in_heap or TIMER_STATUS_in_list.
> > 
> > With the address of the timer from the vmcore:
> > crash> struct timer 830839bcd5c8
> > struct timer {
> >   expires = 0,
> >   {
> > heap_offset = 968676784, 
> > list_next = 0x830839bcd5b0, 
> > inactive = {
> >   next = 0x830839bcd5b0, 
> >   prev = 0x20020020050
> > }
> >   }, 
> >   function = 0x830839bcd5e0, 
> >   data = 0x830839bcd5e0, 
> >   cpu = 54768, 
> >   status = 188 '\274'
> > }
> > 
> > This looks like the structure was overwritten.
> > 
> > The call path seems to be:
> >  mwait_idle()
> >sched_tick_resume()
> >  csched_tick_resume()
> >set_timer()
> >  remove_timer()
> > 
> > In sched_tick_resume():
> >  unsigned int cpu = smp_processor_id();
> > 
> > I extracted struct cpu_info from the stack and found:
> > crash> cpu_info 830839bbff18
> > struct cpu_info {
> >   ...
> >   processor_id = 14, 
> >   current_vcpu = 0x8300754fd000, 
> >   per_cpu_offset = 0x37b9887700,
> >   ...
> > 
> > And from the code in sched_credit.c:  spc = CSCHED_PCPU(cpu);
> > I calculated spc:
> > 
> > crash> csched_pcpu 0x830839be2b40
> > struct csched_pcpu {
> >   runq = {
> > next = 0x830839be2b40, 
> > prev = 0x830839be2b40
> >   }, 
> >   runq_sort_last = 0, 
> >   ticker = {
> > expires = 17408000, 
> > {
> >   heap_offset = 1, 
> >   list_next = 0x10010010001, 
> >   inactive = {
> > next = 0x10010010001, 
> > prev = 0x200200200200200
> >   }
> > }, 
> > 

Re: [Xen-devel] Help with: Xen BUG at timer.c:189

[Jan Beulich]

>>> On 19.05.17 at 11:52,  wrote:
> I'am struggling with a hypervisor panic. The hypervisor version is 4.4.3,
> yes I know - very old ;-), but the affected code hasn't much changed.

Well, at the very least I'd expect you to base your code on
4.4.4 plus the about 199 backports we've added there over time
(assuming yours is based on ours), some of which are scheduler
ones.

> The panic was never seen before and occured while booting the system,
> only dom0 was running.
> 
> (XEN) Xen BUG at timer.c:189
> (XEN) [ Xen-4.4.3_02-26.2_x3  x86_64  debug=n  Tainted:C ]
> (XEN) CPU:14
> (XEN) RIP:e008:[] remove_entry+0x26/0x80
> (XEN) RFLAGS: 00010086   CONTEXT: hypervisor
> (XEN) rax: 00bc   rbx: 830839bcd5c8   rcx: 82d080343a80
> (XEN) rdx: d5f0   rsi: 002887fa   rdi: 830839bcd5c8
> (XEN) rbp: d5f0   rsp: 830839bbfe20   r8:  002887211398
> (XEN) r9:  00288721132f   r10: bf8b2994   r11: 82d08011cfa0
> (XEN) r12: 830839bcd5c8   r13: 82d080343a80   r14: d5f0
> (XEN) r15: 82d080343a80   cr0: 8005003b   cr4: 001526f0
> (XEN) cr3: 75679000   cr2: 0069b8ec
> (XEN) ds: 002b   es: 002b   fs:    gs:    ss: e010   cs: e008
> (XEN) Xen stack trace from rsp=830839bbfe20:
> (XEN)830839bcb020 82d08012d3a8 002887fa 0286
> (XEN)830839bcd5f0 830839bcd9c0 002887c1915d 000e
> (XEN)82d080343a70 0002 830839bcda30 82d0801affd0
> (XEN)0001000e 0028872113f4  82d08030fc00
> (XEN)82d0802ffc00 82d080189a1c  
> (XEN) 107800076e92 82d0801292aa 830839bb8000
> (XEN)8300754fd000 82d08031abe0  830839bcb048
> (XEN) 82d080161ad6 0001 
> (XEN)   0009
> (XEN)8803dda20010 0246  fffe
> (XEN)  800033aa 
> (XEN) 0001 0100 800033aa
> (XEN)e033 0246 8803dda21f08 e02b
> (XEN)   
> (XEN)000e 8300754fd000 0037b9887700 
> (XEN) Xen call trace:
> (XEN)[] remove_entry+0x26/0x80
> (XEN)[] set_timer+0x178/0x230
> (XEN)[] mwait_idle+0x270/0x370
> (XEN)[] local_time_calibration+0x19c/0x1a0
> (XEN)[] __do_softirq+0x6a/0xa0
> (XEN)[] idle_loop+0x16/0x50
> (XEN) 
> (XEN) 
> (XEN) 
> (XEN) Panic on CPU 14:
> (XEN) Xen BUG at timer.c:189
> (XEN) 
> (XEN) 
> (XEN) Reboot in five seconds...
> (XEN) Executing kexec image on cpu14
> (XEN) Shot down all CPUs
> 
> This is the call of BUG() in remove_entry() in case timer->status is not
> TIMER_STATUS_in_heap or TIMER_STATUS_in_list.
> 
> With the address of the timer from the vmcore:
> crash> struct timer 830839bcd5c8
> struct timer {
>   expires = 0,
>   {
> heap_offset = 968676784, 
> list_next = 0x830839bcd5b0, 
> inactive = {
>   next = 0x830839bcd5b0, 
>   prev = 0x20020020050
> }
>   }, 
>   function = 0x830839bcd5e0, 
>   data = 0x830839bcd5e0, 
>   cpu = 54768, 
>   status = 188 '\274'
> }
> 
> This looks like the structure was overwritten.
> 
> The call path seems to be:
>  mwait_idle()
>sched_tick_resume()
>  csched_tick_resume()
>set_timer()
>  remove_timer()
> 
> In sched_tick_resume():
>  unsigned int cpu = smp_processor_id();
> 
> I extracted struct cpu_info from the stack and found:
> crash> cpu_info 830839bbff18
> struct cpu_info {
>   ...
>   processor_id = 14, 
>   current_vcpu = 0x8300754fd000, 
>   per_cpu_offset = 0x37b9887700,
>   ...
> 
> And from the code in sched_credit.c:  spc = CSCHED_PCPU(cpu);
> I calculated spc:
> 
> crash> csched_pcpu 0x830839be2b40
> struct csched_pcpu {
>   runq = {
> next = 0x830839be2b40, 
> prev = 0x830839be2b40
>   }, 
>   runq_sort_last = 0, 
>   ticker = {
> expires = 17408000, 
> {
>   heap_offset = 1, 
>   list_next = 0x10010010001, 
>   inactive = {
> next = 0x10010010001, 
> prev = 0x200200200200200
>   }
> }, 
> function = 0x82d08011f200 , 
> data = 0xe, 
> cpu = 14, 
> status = 3 '\003'
>   }, 
>   tick = 0, 
>   idle_bias = 47, 
>   balance_mask = 0x830839be2bb0
> }
> 
> And this ticker looks ok!
> So the ticker address extracted from the vmcore in remove_entry()
> is not the same as the expected one read from the code.

Right, and 

Re: [Xen-devel] help please...

[Wei Liu]

On Mon, Mar 13, 2017 at 10:36:50AM +0800, 刘瑞丰 wrote:
> [image: 内嵌图片 1]
> 

Please don't send pictures to mailing list. Not everyone can see them
(some people use text based email clients) and they consume a lot of
bandwidth combined.

> it reports:
> 
> checking for PyArg_ParseTuple in -lpython2.7... no
> configure: error: Unable to find a suitable python development library
> configure: error: ./configure failed for tools
> 
> -
> My system is ubuntu16.04 LTS, with kernel linux-4.8.10
> I have installed python2.7 & python3.7 including python*-dev

You seem to have done the right thing.

The next thing to do is to create a program to link against those
library to figure out if it is an issue in Xen build system or your
Linux installation.

Wei.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] help please...

[Gémes Géza]

2017-03-13 12:15 keltezéssel, 刘瑞丰 írta:

own@Sylvanas ‹ master › : ~/xen/raisin
[127] % ./raise -y build
[raisin] calling seabios_check_package
[raisin] seabios_check_package done
[raisin] calling ovmf_check_package
[raisin] ovmf_check_package done
[raisin] calling xen_check_package
[raisin] xen_check_package done
[raisin] calling qemu_check_package
[raisin] qemu_check_package done
[raisin] calling qemu_traditional_check_package
[raisin] qemu_traditional_check_package done
[raisin] calling grub_check_package
[raisin] grub_check_package done
[raisin] calling libvirt_check_package
[raisin] libvirt_check_package done
[raisin] Missing packages: iasl iasl nasm iasl pciutils-dev bison flex 
autopoint xsltproc libxml-xpath-perl libxml2-dev libdevmapper-dev 
libnl-3-dev libnl-route-3-dev libpciaccess-dev

[raisin] Installing...
[sudo] password for own:
[raisin] calling seabios_build
[raisin] seabios_build done
[raisin] calling ovmf_build

own@Sylvanas ‹ master › : ~/xen/raisin
[1] % cd ..

==
own@Sylvanas : ~/xen/xen-4.5.1
[0] % ./configure
*
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
Will build the following subsystems:
  xen
  tools
  stubdom
  docs
configure: creating ./config.status
config.status: creating config/Toplevel.mk
config.status: creating config/Paths.mk
=== configuring in tools (/home/own/xen/xen-4.5.1/tools)
configure: running /bin/bash ./configure --disable-option-checking 
'--prefix=/usr/local' --cache-file=/dev/null --srcdir=.

checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether make sets $(MAKE)... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking for bison... /usr/bin/bison
checking for flex... /usr/bin/flex
checking for perl... /usr/bin/perl
checking for ocamlc... no
checking for ocaml... no
checking for ocamldep... no
checking for ocamlmktop... no
checking for ocamlmklib... no
checking for ocamldoc... no
checking for ocamlbuild... no
checking for ocamlfind... no
checking for gawk... gawk
checking for checkpolicy... no
checking for bash... /bin/bash
checking for python... /usr/local/bin/python
checking for python version >= 2.3 ... yes
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin//grep
checking for egrep... /bin//grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for python-config... /usr/local/bin/python-config
checking Python.h usability... yes
checking Python.h presence... yes
checking for Python.h... yes
checking for PyArg_ParseTuple in -lpython2.7... no
configure: error: Unable to find a suitable python development library
configure: error: ./configure failed for tools

own@Sylvanas : ~/xen/xen-4.5.1
[1] %



Hi,

Two questions:
1. Any specific reason for building Xen 4.5.1? That is a quite old release.
2. Did your raisin build succeed at the end? If you want you can set it 
to build 4.5 as well.


Geza


2017-03-13 18:52 GMT+08:00 刘瑞丰 >:




I've tried what it said, but still blocking at that point.
Did I miss something?

2017-03-13 14:47 GMT+08:00 Gémes Géza >:

2017-03-13 03:36 keltezéssel, 刘瑞丰 írta:


it reports:

checking for PyArg_ParseTuple in -lpython2.7... no
configure: error: Unable to find a suitable python
development library
configure: error: ./configure failed for tools

-
My system is ubuntu16.04 LTS, with kernel linux-4.8.10
I have installed python2.7 & python3.7 including python*-dev
I have did everything i can image...
what's going wrong..? I'm desiring for help...

Re: [Xen-devel] help regarding intel board

[Dario Faggioli]

On Thu, 2016-12-22 at 10:52 +0530, George John wrote:
> hi, 
> 
> Can anybody give a list of intel based boards  for automotive
> embedded use case in which xen could be installed?.
> 
I personally can't.

I tend to think that you may have more probability of getting an useful
answer if you turn this the other way round.

Can you provide a list of Intel board you'd be interested in using
(specifying, for each of them, main characteristics, and links to as
complete as possible specs)?

Then someone may be able/willing to have a look and tell you which ones
are suitable and/or recommended.

Just my 2 cents.

Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)

signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Help regarding bringing up dom0 for lager board

[Julien Grall]

Hello George,

On 09/11/16 14:47, George John wrote:

Thanks it was really the problem of dts. I just followed the proceedings
of Ferger in Mailing lists who tried to bring up Dom0 in lager, and I
got it booted up. But now the problem is with rootfs. I am checking on
that..


May I ask you to update the wikipage? So future users will not get 
caught with the same problem :).


Regards,



Thanks and regards,
George

On Tue, Nov 8, 2016 at 5:20 PM, Julien Grall > wrote:



On 07/11/2016 16:39, George John wrote:

Hi,


Hello,

(XEN) Freed 276kB init memory.
(XEN) traps.c:2505:d0v0 HSR=0x93820007 pc=0xc001d084 gva=0xe7804060
gpa=0x00e6160060


Looking at the log, DOM0 is trying to access an region that is not
mapped (0x00e6160060).

When booting Xen is going through the device tree and mapping to
dom0 all the regions described. So it seems that this region is not
present in the device tree.

Which Linux kernel are you using? I would recommend you to try the
latest as possible and use the device-tree provided in
arch/arm/boot/dts/ to see if it solves the problem.

Cheers,

--
Julien Grall




--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Help regarding bringing up dom0 for lager board

[George John]

Thanks it was really the problem of dts. I just followed the proceedings of
Ferger in Mailing lists who tried to bring up Dom0 in lager, and I got it
booted up. But now the problem is with rootfs. I am checking on that..

Thanks and regards,
George

On Tue, Nov 8, 2016 at 5:20 PM, Julien Grall  wrote:

>
>
> On 07/11/2016 16:39, George John wrote:
>
>> Hi,
>>
>
> Hello,
>
> (XEN) Freed 276kB init memory.
>> (XEN) traps.c:2505:d0v0 HSR=0x93820007 pc=0xc001d084 gva=0xe7804060
>> gpa=0x00e6160060
>>
>
> Looking at the log, DOM0 is trying to access an region that is not mapped
> (0x00e6160060).
>
> When booting Xen is going through the device tree and mapping to dom0 all
> the regions described. So it seems that this region is not present in the
> device tree.
>
> Which Linux kernel are you using? I would recommend you to try the latest
> as possible and use the device-tree provided in arch/arm/boot/dts/ to see
> if it solves the problem.
>
> Cheers,
>
> --
> Julien Grall
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Help regarding bringing up dom0 for lager board

[Julien Grall]

On 07/11/2016 16:39, George John wrote:

Hi,


Hello,


(XEN) Freed 276kB init memory.
(XEN) traps.c:2505:d0v0 HSR=0x93820007 pc=0xc001d084 gva=0xe7804060
gpa=0x00e6160060


Looking at the log, DOM0 is trying to access an region that is not 
mapped (0x00e6160060).


When booting Xen is going through the device tree and mapping to dom0 
all the regions described. So it seems that this region is not present 
in the device tree.


Which Linux kernel are you using? I would recommend you to try the 
latest as possible and use the device-tree provided in 
arch/arm/boot/dts/ to see if it solves the problem.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Wei Yang]

On Mon, Sep 26, 2016 at 12:18:06PM +0200, Dario Faggioli wrote:
>On Sat, 2016-09-24 at 11:39 +0800, Wei Yang wrote:
>> On Thu, Sep 22, 2016 at 11:12:15AM +0200, Dario Faggioli wrote:
>> > _Almost_ correct. However, the problem is more that vcpu_wake() can
>> > happen in response to an IRQ, and when you grab a spinlock in IRQ
>> > context, you need to disable IRQs.
>> > 
>> 
>> Ok, now I have a question to this sentence.
>> 
>> I have checked some posts which says in the interrupt handling, irq
>> would be
>> disabled. Well, this really depends on the implementation.
>> 
>> This means in Xen, irq is still enabled during interrupt handling?
>> Because of
>> this, we have to disable IRQ when we need to grab the spin lock?
>> 
>So, I don't think I'm getting all you're saying and asking.
>
>The situation is like this: currently, vcpu_wake() can be called both
>from inside or outside IRQ context, and both with IRQs enabled or
>disabled.
>
>Given this situation, to be safe, we need to disable IRQs when taking
>whatever spinlock vcpu_wake() (and the functions it calls, of course)
>calls. Since it takes the scheduler's runq lock, this means we need to
>take the lock with IRQ disabled.
>
>And because of that, _everyone_ that takes the schedulr's lock must do
>that with IRQs disabled. If we manage to *not* take the scheduler's
>lock with IRQ disabled in vcpu_wake(), we then can do the same
>everywhere else. But we can't do that with the current structure of the
>code, and that's why we're thinking to defer the actual wakeup --i.e.,
>the actual call to vcpu_wake()-- to a moment when:
> - IRQs are enabled,
> - we don't need to disable them.
>

Dario,

I appreciate your patience and almost get the background and your purpose.

First, let me give my conclusion. By taking the schedule lock with IRQ
enabled, it will improve the parallelism to some extend, while it would be
hard to say the net effect. Need more data from benchmark to verify.

Then, let me describe what I understand. (Maybe not correct :-))

This is all about the SMP, spinlock and IRQ/IRQ context.

Two basic assumptions:
a. all interrupt are the same priority, interrupt could not be preempted by 
others. 
b. in the whole interrupt handler, IRQ is disabled on local processor

Let's assume there is only one processor first,
1. If the spinlock just used in interrupt handler, looks it is save, no race
   condition.
2. While if the spinlock maybe used outside interrupt handler, ok, we need to
   at least disable IRQ when grab the lock outside interrupt handler.
   Otherwise there is deadlock.

Then if there are several CPUs in the system.
1. If the spinlock just used in interrupt handler, would this be still save to
   not disable the IRQ? It looks ok to me.
2. While if the spinlock maybe used outside interrupt handler, ok, we need to
   at least disable IRQ when grab the lock outside interrupt handler.

This leads to the spinlock usage convention: if a spinlock would be grabbed in
interrupt handler(IRQ context), we need to grab it with IRQ disabled _always_.

Then, to achieve your purpose -- grab the spinlock with IRQ enabled, you need
to move all the lock operation outside of the interrupt handler. What you are
doing in the code is to deffer the job to softirq when it is in_irq.

By now, I can understand your first part of your check. If vcpu_wake() is
not running in_irq() we can do the job directly now. But not the second part,
why we need to check the local irq is also enabled? In my mind, we need to
check it is not in_irq() is enough.

Thanks a lot for your time :-)

>This is it. About the following...
>
>> Looks I almost get every thing. Let me do a summary to see whether I
>> have got
>> your words.
>> 
>>    (spinlock usage convention in Xen)  &  (vcpu_wake() may happen in
>> an IRQ)
>> 
>This is ok.
>
>> =>
>> 
>>    (irq all disabled || irq all enabled) & (disable irq on grabbing
>> spinlock)
>> 
>This, I don't get it.
>
>> =>
>> 
>>    should grab schedule lock with IRQ disabled
>> 
>Yes, sounds right.
>
>Dario
>-- 
><> (Raistlin Majere)
>-
>Dario Faggioli, Ph.D, http://about.me/dario.faggioli
>Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)


-- 
Richard Yang\nHelp you, Help me

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Dario Faggioli]

On Sat, 2016-09-24 at 11:39 +0800, Wei Yang wrote:
> On Thu, Sep 22, 2016 at 11:12:15AM +0200, Dario Faggioli wrote:
> > _Almost_ correct. However, the problem is more that vcpu_wake() can
> > happen in response to an IRQ, and when you grab a spinlock in IRQ
> > context, you need to disable IRQs.
> > 
> 
> Ok, now I have a question to this sentence.
> 
> I have checked some posts which says in the interrupt handling, irq
> would be
> disabled. Well, this really depends on the implementation.
> 
> This means in Xen, irq is still enabled during interrupt handling?
> Because of
> this, we have to disable IRQ when we need to grab the spin lock?
> 
So, I don't think I'm getting all you're saying and asking.

The situation is like this: currently, vcpu_wake() can be called both
from inside or outside IRQ context, and both with IRQs enabled or
disabled.

Given this situation, to be safe, we need to disable IRQs when taking
whatever spinlock vcpu_wake() (and the functions it calls, of course)
calls. Since it takes the scheduler's runq lock, this means we need to
take the lock with IRQ disabled.

And because of that, _everyone_ that takes the schedulr's lock must do
that with IRQs disabled. If we manage to *not* take the scheduler's
lock with IRQ disabled in vcpu_wake(), we then can do the same
everywhere else. But we can't do that with the current structure of the
code, and that's why we're thinking to defer the actual wakeup --i.e.,
the actual call to vcpu_wake()-- to a moment when:
 - IRQs are enabled,
 - we don't need to disable them.

This is it. About the following...

> Looks I almost get every thing. Let me do a summary to see whether I
> have got
> your words.
> 
>    (spinlock usage convention in Xen)  &  (vcpu_wake() may happen in
> an IRQ)
> 
This is ok.

> =>
> 
>    (irq all disabled || irq all enabled) & (disable irq on grabbing
> spinlock)
> 
This, I don't get it.

> =>
> 
>    should grab schedule lock with IRQ disabled
> 
Yes, sounds right.

Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)

signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Wei Yang]

On Thu, Sep 22, 2016 at 11:55:16AM +0200, Dario Faggioli wrote:
>On Thu, 2016-09-22 at 11:12 +0200, Dario Faggioli wrote:
>> _Almost_ correct. However, the problem is more that vcpu_wake() can
>> happen in response to an IRQ, and when you grab a spinlock in IRQ
>> context, you need to disable IRQs.
>> 
>> There is a good explanation of why, here:
>> 
>Ah, sorry, link is missing! Here:
>
>https://www.kernel.org/doc/Documentation/locking/spinlocks.txt
>
>Read the "Lesson 3: spinlocks revisited." section.
>

Oh, you have a link here ... let me digest it.

>Dario
>-- 
><> (Raistlin Majere)
>-
>Dario Faggioli, Ph.D, http://about.me/dario.faggioli
>Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)


-- 
Richard Yang\nHelp you, Help me

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Wei Yang]

On Thu, Sep 22, 2016 at 11:12:15AM +0200, Dario Faggioli wrote:
>On Sat, 2016-09-17 at 03:30 +, Wei Yang wrote:
>> Dario,
>> 
>Hey, hi again, and sorry for the in getting back at this particular
>part of the conversation.
>

Sure, I was busy with other stuffs these days :-(

>> Just get chance to look into this. This is interesting and I am
>> trying to
>> understand the problem you want to solve first.
>> 
>:-)
>
>> vcpu_wake() is a _special_ case who has to turn off the irq, because
>> SCHED_OP(wake, v) would not only enqueue the vcpu but also tickle
>> pcpu to pick
>> up the queued vcpu. And the tickling part needd the irq to be turned
>> off.
>> 
>_Almost_ correct. However, the problem is more that vcpu_wake() can
>happen in response to an IRQ, and when you grab a spinlock in IRQ
>context, you need to disable IRQs.
>

_start:
jmp 1f
2:

Ok, now I have a question to this sentence.

I have checked some posts which says in the interrupt handling, irq would be
disabled. Well, this really depends on the implementation.

This means in Xen, irq is still enabled during interrupt handling? Because of
this, we have to disable IRQ when we need to grab the spin lock?

jmp $$

>There is a good explanation of why, here:
>
>
>> I don't get the this part. Why we have to turn off the irq for
>> tickling pcpu?
>> 
>We don't. The point here is that, in Xen, we wants spinlocks to either
>_always_ be taken with IRQs disabled or _always_ be taken with IRQs
>enabled. Well, since we now take the scheduler's runqueue lock in
>vcpu_wake() (and as said that must be done with IRQs disabled), this
>implies that the scheduler's runqueue lock needs to always be taken
>with IRQs disabled, even when leaving them enabled would not actually
>be an issue, for being consistent with the locking rule.
>
>So, if we manage to avoid taking the scheduler's runqueue lock in vcpu_wake(), 
>we will manage to put ourself in the opposite situation, wrt the locking 
>consistency rule, i.e., we can _always_ take the scheduler's runqueue lock 
>with IRQs enabled.
>
>This is, broadly speaking, the problem we wanted to solve, and how we thought 
>about solving it.

1:

Looks I almost get every thing. Let me do a summary to see whether I have got
your words.

   (spinlock usage convention in Xen)  &  (vcpu_wake() may happen in an IRQ)

=>

   (irq all disabled || irq all enabled) & (disable irq on grabbing spinlock)

=>

   should grab schedule lock with IRQ disabled

jmp 2b

>
>> And by enabling irq in schedule handlers benefits the performance? or
>> ? The
>> motivation behind this is?
>> 
>As I said to you about your modification, here too it is not super-easy 
>to tell in advance whether, and if yes, by how much, we'll see a boost
>in performance.
>
>In this case, the idea is that, in general, keeping IRQs disabled is
>bad. It is bad for concurrency, it is bad for latency in both
>scheduling and when it comes to reacting to external event. And it has
>been profiled that the scheduler is one of the component that keeps the
>IRQs disabled for long chunks of time.
>
>I honestly did expect things to improve a bit, especially on certain
>workloads, but of course, as usual, benchmarks will tell. :-)
>
>> Didn't look into your code yet. 
>>
>Ok. 
>Even if/when you look at it, bear in mind it's still only a stub.
>
>> From the description from Andrew, I comes with
>> several questions:
>> 1. which per-cpu list you want to queue the vcpu? v->processor?
>>
>ISTR having thought, and even given a try, to both v->processor (where
>v is the waking vcpu) and the current cpu (where for 'current cpu' I
>meant the cpu where the wake-up is occurring).
>
>I think I decided to use the current cpu (mostly for the same reasons
>why I don't think there is much advantage in what you are trying to do
>:-P)
>
>> 2. SCHEDULE_WAKE_SOFTIRQ would do runqueue insert and tickle?
>>
>It would call what is now vcpu_wake(). So, yes, for most scheduler that
>means inserting and tickling, but it's really schedulere specific.
>
>> 3. purpose is so that schedule softirq could run with irq enabled
>>
>Purpose is that scheduling functions can be executed with IRQs enabled,
>yes.
>
>> 4. the schedule softirq would do the jobs currently in vcpu_wake()?
>> 
>Err... not sure what you mean (but maybe I've already answered at point
>2.?).
>
>> Took the per-cpu branch as an example, I see several commits. The top
>> 6 are
>> related ones, right?
>> 
>The top 4 (I told you it's a stub :-P).
>
>Regards,
>Dario
>-- 
><> (Raistlin Majere)
>-
>Dario Faggioli, Ph.D, http://about.me/dario.faggioli
>Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)


-- 
Richard Yang\nHelp you, Help me

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Dario Faggioli]

On Thu, 2016-09-22 at 11:12 +0200, Dario Faggioli wrote:
> _Almost_ correct. However, the problem is more that vcpu_wake() can
> happen in response to an IRQ, and when you grab a spinlock in IRQ
> context, you need to disable IRQs.
> 
> There is a good explanation of why, here:
> 
Ah, sorry, link is missing! Here:

https://www.kernel.org/doc/Documentation/locking/spinlocks.txt

Read the "Lesson 3: spinlocks revisited." section.

Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)

signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Dario Faggioli]

On Sun, 2016-09-18 at 12:03 +0800, Wei Yang wrote:
> Dario,
> 
> Took a look into your code, some questions as below:
>
Haha, here you are! :-D

> 1. vcpu_wake is split into two cases, the first case is "not in irq"
> and "irq
> enabled". Some reason for this classification? Maybe some background
> knowledge
> I need to have.
> 
Mmm... yes. Well, probably that distinction should not be there, as it
is a (potential, benchmark will tell if that's the case) optimization.
And optimizations should be added later, with their own patch and patch
description, etc... But as I said in the other email, the code is not
even RFC. :-/

So, point is, the whole point of the patch series is to leave
interrupts enabled inside scheduling functions, including vcpu_wake()
(which is becoming _vcpu_wake()). In order to do that, we taking the
scheduler lock, we'll start using spin_lock() and vcpu_spin_lock(),
instead of vcpu_spinlock_irq[save](). But this also means that IRQs
can't be disabled when we do that, or:
 - we violate Xen's lock consistency rule (there's ASSERT()s checking
   that in debug builds)
 - we risk re-enabling interrupts at the wrong time.

So, if IRQs are disabled when vcpu_wake() is called, or we are in IRQ
context and not disabling them expose us to races, we want to go the
deferred way the patch series is implementing.

However, if vcpu_wake() is called with IRQ enabled and not from IRQ
context, none of the above risks stand, and we can avoid doing all the
dance required for deferred wakeup, and just call _vcpu_wake().

I'm saying this is a (potential) optimization because deferring the
wakeup has a cost, and this 'if' is trying to avoid paying such when
not strictly necessary.

But really, in a proper version of this series I should just always
defer the wakeup in this patch, and have another patch, at the end of
the series, where this attempt to avoid deferral is implemented, and
run benchmarks with and without said patch applied.

> 2. in vcpu_wake(), I see you raise def_vcpu_wake softirq when the
> list is
> empty. and then add the vcpu to the list. any reason for this
> sequence? BTW,
> at this moment vcpu_wake() holds the spin lock and
> vcpu_wake_deferred() should
> take the spin lock before continuing.
> 
If the list is empty it means there is no one going through it and
waking up vcpus from there. This means I need to raise the softirq that
will cause someone to go check the list, so it will find the vcpu that
I'm inserting there right now.

OTOH, if the list is not empty, it means there is already someone going
through the list itself. This means I don't need to raise anything, as
the vcpu that I'm inserting in the list right now will be found by such
waker, and woken up.

This is for avoiding raising too many softirq and poking too many cpus
to run their deferred wakers, which would introduce contention issue on
the deferred list lock.

The reason why the check is before the insertion is pure cosmetics. I
can well insert first and check for list_is_singleton(). In the code
you see there, that does not matter, because the deferred list has it's
spinlock. But this will probably change if I get rid of that, making
the list lockless.

> In the last patch, I saw you introduce a flag wakeup_triggered, and
> then
> change the sequence a little. Why at this place, you want to change
> the
> sequence here?
> 
It was another attempt of optimization. But again, this is all changing
with the list becoming lockless.

> 3. The main purpose of this series is to enable irq for "scheduler
> handlers".
> I am not sure the original initiative for this, while I guess this
> could
> increase the parallelism, right? And I think we have to make sure
> those
> functions are not re-entrant, otherwise they will face dead lock when
> locking
> on the same spin-lock.
> 
Hopefully, I explained the purpose in my previous email.

I'm not sure what you mean here with 're-entrant'. The only reason why
we want IRQs to be disabled when taking a lock, is because we take that
lock from both inside and outside of IRQ context.

So, if we're sure that a lock is never taken from IRQ context, we can
avoid disabling IRQs. And this is what this is all about: the scheduler
lock is taken once from IRQ context, and this series is trying to get
rid of that one case.

> 4. This looks similar with what we try to achieve. While we try to
> give the
> vcpu to someone else, who we think may pick it, to wake it up, and in
> your
> implementation, you at this vcpu to its own list and wake it up some
> time
> later. 
>
It does looks similar (this is the reason why I mentioned it to you :-
)), although:
 - both the goal and the expectations are different,
 - I'm not really 'giving the vcpu to someone else', I'm _deferring_ 
   it, i.e., it is still going to happen on the same cpu where the 
   event happens, only a little later in time (when no longer in IRQ
   context)

> I guess next step you want to do is to make the wakeup_defer.list a
> 

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Dario Faggioli]

On Sat, 2016-09-17 at 03:30 +, Wei Yang wrote:
> Dario,
> 
Hey, hi again, and sorry for the in getting back at this particular
part of the conversation.

> Just get chance to look into this. This is interesting and I am
> trying to
> understand the problem you want to solve first.
> 
:-)

> vcpu_wake() is a _special_ case who has to turn off the irq, because
> SCHED_OP(wake, v) would not only enqueue the vcpu but also tickle
> pcpu to pick
> up the queued vcpu. And the tickling part needd the irq to be turned
> off.
> 
_Almost_ correct. However, the problem is more that vcpu_wake() can
happen in response to an IRQ, and when you grab a spinlock in IRQ
context, you need to disable IRQs.

There is a good explanation of why, here:


> I don't get the this part. Why we have to turn off the irq for
> tickling pcpu?
> 
We don't. The point here is that, in Xen, we wants spinlocks to either
_always_ be taken with IRQs disabled or _always_ be taken with IRQs
enabled. Well, since we now take the scheduler's runqueue lock in
vcpu_wake() (and as said that must be done with IRQs disabled), this
implies that the scheduler's runqueue lock needs to always be taken
with IRQs disabled, even when leaving them enabled would not actually
be an issue, for being consistent with the locking rule.

So, if we manage to avoid taking the scheduler's runqueue lock in vcpu_wake(), 
we will manage to put ourself in the opposite situation, wrt the locking 
consistency rule, i.e., we can _always_ take the scheduler's runqueue lock with 
IRQs enabled.

This is, broadly speaking, the problem we wanted to solve, and how we thought 
about solving it.

> And by enabling irq in schedule handlers benefits the performance? or
> ? The
> motivation behind this is?
> 
As I said to you about your modification, here too it is not super-easy 
to tell in advance whether, and if yes, by how much, we'll see a boost
in performance.

In this case, the idea is that, in general, keeping IRQs disabled is
bad. It is bad for concurrency, it is bad for latency in both
scheduling and when it comes to reacting to external event. And it has
been profiled that the scheduler is one of the component that keeps the
IRQs disabled for long chunks of time.

I honestly did expect things to improve a bit, especially on certain
workloads, but of course, as usual, benchmarks will tell. :-)

> Didn't look into your code yet. 
>
Ok. 
Even if/when you look at it, bear in mind it's still only a stub.

> From the description from Andrew, I comes with
> several questions:
> 1. which per-cpu list you want to queue the vcpu? v->processor?
>
ISTR having thought, and even given a try, to both v->processor (where
v is the waking vcpu) and the current cpu (where for 'current cpu' I
meant the cpu where the wake-up is occurring).

I think I decided to use the current cpu (mostly for the same reasons
why I don't think there is much advantage in what you are trying to do
:-P)

> 2. SCHEDULE_WAKE_SOFTIRQ would do runqueue insert and tickle?
>
It would call what is now vcpu_wake(). So, yes, for most scheduler that
means inserting and tickling, but it's really schedulere specific.

> 3. purpose is so that schedule softirq could run with irq enabled
>
Purpose is that scheduling functions can be executed with IRQs enabled,
yes.

> 4. the schedule softirq would do the jobs currently in vcpu_wake()?
> 
Err... not sure what you mean (but maybe I've already answered at point
2.?).

> Took the per-cpu branch as an example, I see several commits. The top
> 6 are
> related ones, right?
> 
The top 4 (I told you it's a stub :-P).

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)

signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Wei Yang]

On Tue, Sep 20, 2016 at 01:24:17AM +0200, Dario Faggioli wrote:
>On Sat, 2016-09-17 at 00:31 +, Wei Yang wrote:
>> On Fri, Sep 16, 2016 at 06:07:08PM +0200, Dario Faggioli wrote:
>> > But then again, if the system is not oversubscribed, I'd tend to
>> > think
>> > it to be tolerable, and I'd expect the biggest problem to be the
>> > work-
>> > stealing logic (considering the high number of pcpus), rather than
>> > the
>> > duration of the critical sections within vcpu_wake().
>> > 
>> Yes, we are trying to improve the stealing part too.
>> 
>Great.
>
>> Sounds reasonable, vcpu_wake() is O(1) while "stealing" is O(N) in
>> terms of
>> #PCPUs.
>> 
>Exactly!
>
>> > I also think current upstream is a bit better, especially because
>> > it's
>> > mostly me that made it look the way it does. :-D
>> 
>> Ah, not intended to flattering you.
>> 
>EhEh! Sure, I was joking myself. :-)
>

Ah, really enjoy talking with you :-)
You got some special sense of humor, and I like it~

Have a good day!

>> > 
>> > 
>> > But I was actually describing how 4.1 works. In fact, in 4.1, if
>> > pcpu 6
>> > is idle (se the '//xxx xxx xxx' comments I'm adding to the code
>> > excerpts:
>> > 
>> > if ( new->pri > cur->pri )  //is true, so we put pcpu 6 in mask
>> > {
>> > cpu_set(cpu, mask);
>> > }
>> > if ( cur->pri > CSCHED_PRI_IDLE )  //is false!!
>> > {
>> >    
>> > }
>> > if ( !cpus_empty(mask) ) //the mask contains pcpu 6 only
>> > cpumask_raise_softirq(mask, SCHEDULE_SOFTIRQ);
>> > 
>> 
>> Hmm... don't have the code at hand, while looks you are right. I
>> misunderstood
>> the code.
>
>> So only one idler would be tickled even there would be several idlers
>> in the
>> system. I thought we would tickle several idlers, which may introduce
>> some
>> contention between them.
>> 
>You can ask Xen to tickle more than one idle, by means of that
>parameter. But again, tickling idlers --either one or many-- would only
>happen if there's actual need for it.
>
>> BTW, any idea behind the cycle_cpu()? If this is about the locality,
>> how about
>> cycle within the node? and cycle within the node where v->processor
>> is?
>> 
>Cycle is there as a form of load spreading, i.e., we don't want to risk
>tickling always the same set of pcpus, or always the ones with the
>lower cpu IDs.
>
>You're right that taking locality into account could be a good thing in
>big systems. No, that is not done, not in 4.1 nor in upstream, and it
>may be something actually worth trying (although, again, it's probably
>unrelated to your issue).
>
>> > And again, I'd personally spend some more time --even by
>> > temporarily
>> > and hackishly instrumenting the code-- to understand better where
>> > the
>> > bottleneck is.
>> > 
>> 
>> Hmm... usually we use xentrace and lock profile to identify the
>> bottleneck,
>> other method you would like to suggest? and instrumenting the code
>> means to
>> add some log in code?
>> 
>xentrace and lock profile is all I use too, and there's not much else,
>without touching the code. And in fact, yes, with "instrumenting the
>code" I means temporary changing the code to display the information
>you need.
>
>In this specific case, rather than adding printks here and there
>(which, e.g., is what you usually do for debugging crashes or alike),
>I'd see about modifying a little bit either xentrace or lock profiling
>code (or both!) to make them provide the info you need.
>
>Sorry, but I don't think I can be much more specific without going
>looking at the code and actually trying to do that...
>
>> > E.g., something that has proven effective for others (and for which
>> > I've got an unfinished and never submitted patch to upstream), is
>> > keeping track of what pcpus actually have at least 1 vcpu in their
>> > runqueues (i.e., at least 1 vcpus that is runnable and not running)
>> > and, inside balance_load(), only consider those when looking for
>> > work
>> > to steal.
>> 
>> Looks like we keep a cpumask with those who have 1 more vcpus and
>> during
>> stealing process, just scan those instead of the online cpumask.
>> 
>Yep, that sounds like a plan, and was along the lines of what I was
>doing. If you want to give it a try, patches (for upstream, of course
>:-D) are welcome. :-P
>
>Regards,
>Dario
>-- 
><> (Raistlin Majere)
>-
>Dario Faggioli, Ph.D, http://about.me/dario.faggioli
>Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)


-- 
Richard Yang\nHelp you, Help me

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Dario Faggioli]

On Sat, 2016-09-17 at 00:31 +, Wei Yang wrote:
> On Fri, Sep 16, 2016 at 06:07:08PM +0200, Dario Faggioli wrote:
> > But then again, if the system is not oversubscribed, I'd tend to
> > think
> > it to be tolerable, and I'd expect the biggest problem to be the
> > work-
> > stealing logic (considering the high number of pcpus), rather than
> > the
> > duration of the critical sections within vcpu_wake().
> > 
> Yes, we are trying to improve the stealing part too.
> 
Great.

> Sounds reasonable, vcpu_wake() is O(1) while "stealing" is O(N) in
> terms of
> #PCPUs.
> 
Exactly!

> > I also think current upstream is a bit better, especially because
> > it's
> > mostly me that made it look the way it does. :-D
> 
> Ah, not intended to flattering you.
> 
EhEh! Sure, I was joking myself. :-)

> > 
> > 
> > But I was actually describing how 4.1 works. In fact, in 4.1, if
> > pcpu 6
> > is idle (se the '//xxx xxx xxx' comments I'm adding to the code
> > excerpts:
> > 
> > if ( new->pri > cur->pri )  //is true, so we put pcpu 6 in mask
> > {
> > cpu_set(cpu, mask);
> > }
> > if ( cur->pri > CSCHED_PRI_IDLE )  //is false!!
> > {
> >    
> > }
> > if ( !cpus_empty(mask) ) //the mask contains pcpu 6 only
> > cpumask_raise_softirq(mask, SCHEDULE_SOFTIRQ);
> > 
> 
> Hmm... don't have the code at hand, while looks you are right. I
> misunderstood
> the code.

> So only one idler would be tickled even there would be several idlers
> in the
> system. I thought we would tickle several idlers, which may introduce
> some
> contention between them.
> 
You can ask Xen to tickle more than one idle, by means of that
parameter. But again, tickling idlers --either one or many-- would only
happen if there's actual need for it.

> BTW, any idea behind the cycle_cpu()? If this is about the locality,
> how about
> cycle within the node? and cycle within the node where v->processor
> is?
> 
Cycle is there as a form of load spreading, i.e., we don't want to risk
tickling always the same set of pcpus, or always the ones with the
lower cpu IDs.

You're right that taking locality into account could be a good thing in
big systems. No, that is not done, not in 4.1 nor in upstream, and it
may be something actually worth trying (although, again, it's probably
unrelated to your issue).

> > And again, I'd personally spend some more time --even by
> > temporarily
> > and hackishly instrumenting the code-- to understand better where
> > the
> > bottleneck is.
> > 
> 
> Hmm... usually we use xentrace and lock profile to identify the
> bottleneck,
> other method you would like to suggest? and instrumenting the code
> means to
> add some log in code?
> 
xentrace and lock profile is all I use too, and there's not much else,
without touching the code. And in fact, yes, with "instrumenting the
code" I means temporary changing the code to display the information
you need.

In this specific case, rather than adding printks here and there
(which, e.g., is what you usually do for debugging crashes or alike),
I'd see about modifying a little bit either xentrace or lock profiling
code (or both!) to make them provide the info you need.

Sorry, but I don't think I can be much more specific without going
looking at the code and actually trying to do that...

> > E.g., something that has proven effective for others (and for which
> > I've got an unfinished and never submitted patch to upstream), is
> > keeping track of what pcpus actually have at least 1 vcpu in their
> > runqueues (i.e., at least 1 vcpus that is runnable and not running)
> > and, inside balance_load(), only consider those when looking for
> > work
> > to steal.
> 
> Looks like we keep a cpumask with those who have 1 more vcpus and
> during
> stealing process, just scan those instead of the online cpumask.
> 
Yep, that sounds like a plan, and was along the lines of what I was
doing. If you want to give it a try, patches (for upstream, of course
:-D) are welcome. :-P

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)

signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Wei Yang]

On Tue, Sep 13, 2016 at 01:30:17PM +0200, Dario Faggioli wrote:
>[using xendevel correct address]
>
>On Tue, 2016-09-13 at 16:54 +0800, Wei Yang wrote:
>> On Fri, 2016-09-09 at 17:41 +0800, Wei Yang wrote:
>> > 
>> > I'm not surprised by that. Yet, I'd be interested in hearing more
>> > about this profiling you have done (things like, how you captured
>> > the data, what workloads you are exactly considering, how you
>> > determined what is the bottleneck, etc).
>> Let me try to explain this.
>> 
>> Workload: a. Blue Screen in Windows Guests
>>    b. some client using Windows to do some video
>> processing
>>   which need precise timestamp (we are not sure the
>> core
>>   reason but we see the system is very slow)
>>
>Do you mind sharing just a bit more, such as:
> - number of pcpus
> - number of vcpus of the various VMs
>
>I also am not sure what "a. Blue screen in Windows guests" above
>means... is there a workload called "Blue Screen"? Or is it that you
>are hitting a BSOD in some of your guests (which ones, what were they
>doing)? Or is it that you _want_ to provoke a BSOD on some of your
>guests? Or is that something else? :-P
>
>> Capture the data: lock profile
>> Bottleneck Check: The schedule lock wait time is really high  
>> 
>Ok, cool. Interesting that lock profiling works on 4.1! :-O
>
>> > The scheduler tries to see whether the v->processor of the waking
>> > vcpu can be re-used, but that's not at all guaranteed, and again,
>> > on a very loaded system, not even that likely!
>> > 
>> Hmm... I may missed something.
>> 
>> Took your assumption below for example. 
>> In my mind, the process looks like this:
>> 
>> csched_vcpu_wake()
>> __runq_insert(),  insert the vcpu in pcpu's 6 runq
>> __runq_tickle(),  raise SCHEDULE_SOFTIRQ on pcpu 6 or other
>> (1)
>> 
>Well, yes. More precisely, at least in current staging,
>SCHEDULE_SOFTIRQ is raised for pcpu 6:
> - if pcpu 6 is idle, or
> - if pcpu 6 is not idle but there actually isn't any idle vcpu, and
>   the waking vcpu is higher in priority than what is running on pcpu 6
>
>> __do_softirq()
>> schedule()
>> csched_schedule(),  for pcpu 6, it may wake up d2v2 based
>> on it's
>> priority
>>
>Yes, but it is pcpu 6 that will run csched_schedule() only if the
>conditions mentioned above are met. If not, it will be some other pcpu
>that will do so (or none!).
>
>But I just realized that the fact that you are actually working on 4.1
>is going to be an issue. In fact, the code that it is now in Xen has
>changed **a lot** since 4.1. In fact, you're missing all the soft-
>affinity work (which you may or may not find useful) but also
>improvements and bugfixing.
>
>I'll have a quick look at how __runq_tickle() looks like in Xen 4.1,
>but there's very few chances I'll be available to provide detailed
>review, advice, testing, etc., on such an old codebase. :-(
>
>> By looking at the code, what I missed may be in __runq_tickle(),
>> which in case
>> there are idle pcpu, schedule_softirq is also raised on them. By
>> doing so,
>> those idle pcpus would steal other busy tasks and may in chance get
>> d2v2?
>> 
>Yes, it looks like, in Xen 4.1, this is more or less what happens. The
>idea is to always tickle pcpu 6, if the priority of the waking vcpu is
>higher than what is running there. 
>
>If pcpu 6 was not idle, we also tickle one or more idle pcpus so that:
> - if the waking vcpu preempted what was running on pcpu 6, an idler
>   can pick-up ("steal") such preempted vcpu and run it;
> - if the waking vcpu ended up in the runqueue, an idler can steal it
>
>> BTW, if the system is in heavy load, how much chance would we have
>> idle pcpu?
>> 
>It's not very likely that there will be idle pcpus in a very busy
>system, I agree.
>
>> > > We can divide this in three steps:
>> > > a) Send IPI to the target CPU and tell it which vcpu we want it
>> > > to 
>> > > wake up.
>> > > b) The interrupt handler on target cpu insert vcpu to a percpu
>> > > queue 
>> > > and raise
>> > >    softirq.
>> > > c) softirq will dequeue the vcpu and wake it up.
>> > > 
>> > I'm not sure I see how you think this would improve the situation.
>> > 
>> > So, quickly, right now:
>> > 
>> > - let's say vcpu 2 of domain 2 (from now d2v2) is waking up
>> > - let's assume d2v2->processor = 6
>> > - let's assume the wakeup is happening on pcpu 4
>> > 
>> > Right now:
>> > 
>> > - on pcpu 4, vcpu_wake(d2v2) takes the scheduler lock of d2v2,
>> >   which is the runqueue lock of pcpu 6 (in Credit1, there is 1
>> >   runqueue per pcpu, and locks are per-cpu already)
>> > - in csched_vcpu_wake(d2v2), d2v2 is inserted in pcpu's 6 runqueue
>> > - still executing on pcpu 4, __runq_tickle() is called, and it
>> >   determines on which pcpu d2v2 should run
>> > - it raises the SCHEDULE_SOFTIRQ for such pcpu. Let's look at the
>> >   following two cases:
>> >    a) if 

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Wei Yang]

On Wed, Sep 14, 2016 at 06:44:17PM +0800, Wei Yang wrote:
>On Tue, Sep 13, 2016 at 01:30:17PM +0200, Dario Faggioli wrote:
>>[using xendevel correct address]
>>
>>On Tue, 2016-09-13 at 16:54 +0800, Wei Yang wrote:
>>> On Fri, 2016-09-09 at 17:41 +0800, Wei Yang wrote:
>>> > 
>>> > I'm not surprised by that. Yet, I'd be interested in hearing more
>>> > about this profiling you have done (things like, how you captured
>>> > the data, what workloads you are exactly considering, how you
>>> > determined what is the bottleneck, etc).
>>> Let me try to explain this.
>>> 
>>> Workload: a. Blue Screen in Windows Guests
>>>   b. some client using Windows to do some video
>>> processing
>>>  which need precise timestamp (we are not sure the
>>> core
>>>  reason but we see the system is very slow)
>>>
>>Do you mind sharing just a bit more, such as:
>> - number of pcpus
>> - number of vcpus of the various VMs
>
>160 pcpus
>16 vcpus in VM and 8 VMs
>
>>
>>I also am not sure what "a. Blue screen in Windows guests" above
>>means... is there a workload called "Blue Screen"? Or is it that you
>>are hitting a BSOD in some of your guests (which ones, what were they
>>doing)? Or is it that you _want_ to provoke a BSOD on some of your
>>guests? Or is that something else? :-P
>>
>
>Yes, the "Blue Screen" is what we want to mimic the behavior from client.
>
>The "Blue Screen" will force the hypervisor to do load balance in my mind.
>
>>> Capture the data: lock profile
>>> Bottleneck Check: The schedule lock wait time is really high  
>>> 
>>Ok, cool. Interesting that lock profiling works on 4.1! :-O
>>
>>> > The scheduler tries to see whether the v->processor of the waking
>>> > vcpu can be re-used, but that's not at all guaranteed, and again,
>>> > on a very loaded system, not even that likely!
>>> > 
>>> Hmm... I may missed something.
>>> 
>>> Took your assumption below for example. 
>>> In my mind, the process looks like this:
>>> 
>>> csched_vcpu_wake()
>>> __runq_insert(),  insert the vcpu in pcpu's 6 runq
>>> __runq_tickle(),  raise SCHEDULE_SOFTIRQ on pcpu 6 or other
>>> (1)
>>> 
>>Well, yes. More precisely, at least in current staging,
>>SCHEDULE_SOFTIRQ is raised for pcpu 6:
>> - if pcpu 6 is idle, or
>> - if pcpu 6 is not idle but there actually isn't any idle vcpu, and
>>   the waking vcpu is higher in priority than what is running on pcpu 6
>>
>>> __do_softirq()
>>> schedule()
>>> csched_schedule(),  for pcpu 6, it may wake up d2v2 based
>>> on it's
>>> priority
>>>
>>Yes, but it is pcpu 6 that will run csched_schedule() only if the
>>conditions mentioned above are met. If not, it will be some other pcpu
>>that will do so (or none!).
>>
>>But I just realized that the fact that you are actually working on 4.1
>>is going to be an issue. In fact, the code that it is now in Xen has
>>changed **a lot** since 4.1. In fact, you're missing all the soft-
>>affinity work (which you may or may not find useful) but also
>>improvements and bugfixing.
>>
>
>That's true...
>
>>I'll have a quick look at how __runq_tickle() looks like in Xen 4.1,
>>but there's very few chances I'll be available to provide detailed
>>review, advice, testing, etc., on such an old codebase. :-(
>>
>>> By looking at the code, what I missed may be in __runq_tickle(),
>>> which in case
>>> there are idle pcpu, schedule_softirq is also raised on them. By
>>> doing so,
>>> those idle pcpus would steal other busy tasks and may in chance get
>>> d2v2?
>>> 
>>Yes, it looks like, in Xen 4.1, this is more or less what happens. The
>>idea is to always tickle pcpu 6, if the priority of the waking vcpu is
>>higher than what is running there. 
>>
>
>Hmm... in case there are idle pcpus and the priority of the waking vcpu is
>higher than what is running on pcpu 6, would pcpu 6 have more chance to run it?
>or other idle pcup would steal it from pcpu6? or they have equal chance?
>
>>If pcpu 6 was not idle, we also tickle one or more idle pcpus so that:
>> - if the waking vcpu preempted what was running on pcpu 6, an idler
>>   can pick-up ("steal") such preempted vcpu and run it;
>> - if the waking vcpu ended up in the runqueue, an idler can steal it
>>
>
>Hmm... I don't get the difference between these two cases.
>
>Looks both are an idler steal the vcpu.
>
>>> BTW, if the system is in heavy load, how much chance would we have
>>> idle pcpu?
>>> 
>>It's not very likely that there will be idle pcpus in a very busy
>>system, I agree.
>>
>>> > > We can divide this in three steps:
>>> > > a) Send IPI to the target CPU and tell it which vcpu we want it
>>> > > to 
>>> > > wake up.
>>> > > b) The interrupt handler on target cpu insert vcpu to a percpu
>>> > > queue 
>>> > > and raise
>>> > >    softirq.
>>> > > c) softirq will dequeue the vcpu and wake it up.
>>> > > 
>>> > I'm not sure I see how you think this would improve the situation.

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Wei Yang]

On Fri, Sep 16, 2016 at 06:07:08PM +0200, Dario Faggioli wrote:
>On Fri, 2016-09-16 at 10:49 +0800, Wei Yang wrote:
>> On Wed, Sep 14, 2016 at 06:18:48PM +0200, Dario Faggioli wrote:
>> > On Wed, 2016-09-14 at 18:44 +0800, Wei Yang wrote:
>> > If the system is not overbooked, it's a bit strange that the
>> > scheduler
>> > is the bottleneck.
>> I looked at the original data again. I don't see detailed data to
>> describe the
>> dom0 configuration.
>> 
>I see. No collection of output of top and xentop in dom0 either then,
>I'm guessing? :-/
>

Probably, let me check with some one to see whether we have the luck.

>> The exact user model is not accessible from our client. We guess
>> their model
>> looks like this.
>> 
>> 
>>  ++ +---+ +-+
>>  |Timer   | --->|Coordinator| ---+--->|Worker   |
>>  ++ +---+|+-+
>>  |
>>  |+-+
>>  +--->|Worker   |
>>  |+-+
>>  |
>>  |+-+
>>  +--->|Worker   |
>>   +-+
>> 
>> One Coordinator would driver several workers based on a high
>> resolution timer.
>> Periodically, workers would be waked up by the coordinator. So at one
>> moment,
>> many workers would be waked up and would triggers the vcpu_wake() in
>> xen.
>> 
>It's not clear to me whether 'Coordinator' and 'Worker's are VMs, or if
>the graph describes the workload run inside the (and if yes, which
>ones) VMs... but that is not terribly important, after all.
>

Oh, yes, these are threads in a VM. Each VM may contain several
groups(instance) of these threads.

>> Not sure this would be a possible reason for the burst vcpu_wake()?
>> 
>Well, there would be, at least potentially, a sensible number of vcpus
>waking up, which indeed can make the runqueue locks of the various
>pcpus contended.
>
>But then again, if the system is not oversubscribed, I'd tend to think
>it to be tolerable, and I'd expect the biggest problem to be the work-
>stealing logic (considering the high number of pcpus), rather than the
>duration of the critical sections within vcpu_wake().
>

Yes, we are trying to improve the stealing part too.

Sounds reasonable, vcpu_wake() is O(1) while "stealing" is O(N) in terms of
#PCPUs.

>> >  - pcpu 6 is eithe idle or it is running someone already (say d3v4)
>> >    + if pcpu 6 is idle, we tickle (i.e., we raise SCHEDULE_SOFTIRQ)
>> >      pcpu 6 itself, and we're done
>> Ok, it looks the behavior differs from 4.1 and current upstream. The
>> upstream
>> just raise the softirq to pcpu6 when it is idle, while 4.1 would
>> raise softirq
>> to both pcpu6 and other idlers even pcpu6 is idle.
>> 
>> I think current upstream is more cleaver.
>> 
>I also think current upstream is a bit better, especially because it's
>mostly me that made it look the way it does. :-D

Ah, not intended to flattering you.

>
>But I was actually describing how 4.1 works. In fact, in 4.1, if pcpu 6
>is idle (se the '//xxx xxx xxx' comments I'm adding to the code
>excerpts:
>
>if ( new->pri > cur->pri )  //is true, so we put pcpu 6 in mask
>{
>cpu_set(cpu, mask);
>}
>if ( cur->pri > CSCHED_PRI_IDLE )  //is false!!
>{
>
>}
>if ( !cpus_empty(mask) ) //the mask contains pcpu 6 only
>cpumask_raise_softirq(mask, SCHEDULE_SOFTIRQ);
>

Hmm... don't have the code at hand, while looks you are right. I misunderstood
the code.

>On the other hand, if pcpu 6 is not idle (and, sticking to the example
>started in the last email, is running d3v4):
>
>if ( new->pri > cur->pri )  //depends from d2v2's prio and d3v4 prio's
>{
>cpu_set(cpu, mask);
>}
>if ( cur->pri > CSCHED_PRI_IDLE ) //is true, so let's see...
>{
>if ( cpus_empty(prv->idlers) )  //is true *only* if there are no idle 
>pcpu 6. Let's assume there are (i.e., let's assume this is false)
>{
>
>}
>else
>{
>cpumask_t idle_mask;
>
>cpus_and(idle_mask, prv->idlers, new->vcpu->cpu_affinity);
>if ( !cpus_empty(idle_mask) )  //is true if there are idlers 
>suitable for new (let's assume there are)
>{
>if ( opt_tickle_one_idle ) //chosen on boot, default is true
>{
>this_cpu(last_tickle_cpu) = 
>cycle_cpu(this_cpu(last_tickle_cpu), idle_mask);
>cpu_set(this_cpu(last_tickle_cpu), mask);

May misunderstand the code previously and like this part too.

So only one idler would be tickled even there would be several idlers in the
system. I thought we would tickle several idlers, which 

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Dario Faggioli]

On Fri, 2016-09-16 at 10:49 +0800, Wei Yang wrote:
> On Wed, Sep 14, 2016 at 06:18:48PM +0200, Dario Faggioli wrote:
> > On Wed, 2016-09-14 at 18:44 +0800, Wei Yang wrote:
> > If the system is not overbooked, it's a bit strange that the
> > scheduler
> > is the bottleneck.
> I looked at the original data again. I don't see detailed data to
> describe the
> dom0 configuration.
> 
I see. No collection of output of top and xentop in dom0 either then,
I'm guessing? :-/

> The exact user model is not accessible from our client. We guess
> their model
> looks like this.
> 
> 
>  ++ +---+ +-+
>  |Timer   | --->|Coordinator| ---+--->|Worker   |
>  ++ +---+|+-+
>  |
>  |+-+
>  +--->|Worker   |
>  |+-+
>  |
>  |+-+
>  +--->|Worker   |
>   +-+
> 
> One Coordinator would driver several workers based on a high
> resolution timer.
> Periodically, workers would be waked up by the coordinator. So at one
> moment,
> many workers would be waked up and would triggers the vcpu_wake() in
> xen.
> 
It's not clear to me whether 'Coordinator' and 'Worker's are VMs, or if
the graph describes the workload run inside the (and if yes, which
ones) VMs... but that is not terribly important, after all.

> Not sure this would be a possible reason for the burst vcpu_wake()?
> 
Well, there would be, at least potentially, a sensible number of vcpus
waking up, which indeed can make the runqueue locks of the various
pcpus contended.

But then again, if the system is not oversubscribed, I'd tend to think
it to be tolerable, and I'd expect the biggest problem to be the work-
stealing logic (considering the high number of pcpus), rather than the
duration of the critical sections within vcpu_wake().

> >  - pcpu 6 is eithe idle or it is running someone already (say d3v4)
> >    + if pcpu 6 is idle, we tickle (i.e., we raise SCHEDULE_SOFTIRQ)
> >      pcpu 6 itself, and we're done
> Ok, it looks the behavior differs from 4.1 and current upstream. The
> upstream
> just raise the softirq to pcpu6 when it is idle, while 4.1 would
> raise softirq
> to both pcpu6 and other idlers even pcpu6 is idle.
> 
> I think current upstream is more cleaver.
> 
I also think current upstream is a bit better, especially because it's
mostly me that made it look the way it does. :-D

But I was actually describing how 4.1 works. In fact, in 4.1, if pcpu 6
is idle (se the '//xxx xxx xxx' comments I'm adding to the code
excerpts:

if ( new->pri > cur->pri )  //is true, so we put pcpu 6 in mask
{
cpu_set(cpu, mask);
}
if ( cur->pri > CSCHED_PRI_IDLE )  //is false!!
{

}
if ( !cpus_empty(mask) ) //the mask contains pcpu 6 only
cpumask_raise_softirq(mask, SCHEDULE_SOFTIRQ);

On the other hand, if pcpu 6 is not idle (and, sticking to the example
started in the last email, is running d3v4):

if ( new->pri > cur->pri )  //depends from d2v2's prio and d3v4 prio's
{
cpu_set(cpu, mask);
}
if ( cur->pri > CSCHED_PRI_IDLE ) //is true, so let's see...
{
if ( cpus_empty(prv->idlers) )  //is true *only* if there are no idle 
pcpu 6. Let's assume there are (i.e., let's assume this is false)
{

}
else
{
cpumask_t idle_mask;

cpus_and(idle_mask, prv->idlers, new->vcpu->cpu_affinity);
if ( !cpus_empty(idle_mask) )  //is true if there are idlers 
suitable for new (let's assume there are)
{
if ( opt_tickle_one_idle ) //chosen on boot, default is true
{
this_cpu(last_tickle_cpu) = 
cycle_cpu(this_cpu(last_tickle_cpu), idle_mask);
cpu_set(this_cpu(last_tickle_cpu), mask);
}
else

}
cpus_and(mask, mask, new->vcpu->cpu_affinity);
}
}
if ( !cpus_empty(mask) ) //mask contains pcpu 6 if d2v2 prio's was higher, 
and also contains one idle pcpu
cpumask_raise_softirq(mask, SCHEDULE_SOFTIRQ);

So, as I was saying, if pcpu 6 is idle, only pcpu 6 is tickled. If it's
not, at least one idler (if it exists) is tickled, and pcpu 6 is
tickled or not, depending or priorities.

> > And in fact, in more recent version of Xen, I made the code do
> > something very close to what you're suggesting. Here's the comment
> > that
> > you can find where all this logic is implemented (it's way more
> > complicated than this, and than the code in 4.1, because it takes
> > soft-
> > affinity 

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Wei Yang]

On Wed, Sep 14, 2016 at 06:18:48PM +0200, Dario Faggioli wrote:
>On Wed, 2016-09-14 at 18:44 +0800, Wei Yang wrote:
>> On Tue, Sep 13, 2016 at 01:30:17PM +0200, Dario Faggioli wrote:
>> > 
>> > Do you mind sharing just a bit more, such as:
>> >  - number of pcpus
>> >  - number of vcpus of the various VMs
>> 160 pcpus
>> 16 vcpus in VM and 8 VMs
>> 
>So, 16x8=128, which means you're not even oversubscribing. Maybe some
>of the pcpus are hyperthreads and not full cores (are they?), but
>still, this is not something I'd describe "super intensive cpu load".
>
>Oh, well, there's dom0 of course. So, if dom0 has more than 160-128=32
>vcpus (is this the case?), you technically are oversubscribing. But
>then, what does dom0 do? Is it very busy doing some stuff on its own,
>or running the backends/etc for the VMs? Can you check that with top,
>xentop, and alike?
>
>If the system is not overbooked, it's a bit strange that the scheduler
>is the bottleneck.

I looked at the original data again. I don't see detailed data to describe the
dom0 configuration.

The exact user model is not accessible from our client. We guess their model
looks like this.


 ++ +---+ +-+
 |Timer   | --->|Coordinator| ---+--->|Worker   |
 ++ +---+|+-+
 |
 |+-+
 +--->|Worker   |
 |+-+
 |
 |+-+
 +--->|Worker   |
  +-+

One Coordinator would driver several workers based on a high resolution timer.
Periodically, workers would be waked up by the coordinator. So at one moment,
many workers would be waked up and would triggers the vcpu_wake() in xen.

Not sure this would be a possible reason for the burst vcpu_wake()?

>
>> Yes, the "Blue Screen" is what we want to mimic the behavior from
>> client.
>> 
>> The "Blue Screen" will force the hypervisor to do load balance in my
>> mind.
>> 
>I've no idea what this means (but I don't know much of Windows and of
>what happens when it crashes with a blue screen).
>
>> > I'll have a quick look at how __runq_tickle() looks like in Xen
>> > 4.1,
>> > but there's very few chances I'll be available to provide detailed
>> > review, advice, testing, etc., on such an old codebase. :-(
>> > 
>> > > 
>> > > By looking at the code, what I missed may be in __runq_tickle(),
>> > > which in case
>> > > there are idle pcpu, schedule_softirq is also raised on them. By
>> > > doing so,
>> > > those idle pcpus would steal other busy tasks and may in chance
>> > > get
>> > > d2v2?
>> > > 
>> > Yes, it looks like, in Xen 4.1, this is more or less what happens.
>> > The
>> > idea is to always tickle pcpu 6, if the priority of the waking vcpu
>> > is
>> > higher than what is running there. 
>> > 
>> Hmm... in case there are idle pcpus and the priority of the waking
>> vcpu is
>> higher than what is running on pcpu 6, would pcpu 6 have more chance
>> to run it?
>> or other idle pcup would steal it from pcpu6? or they have equal
>> chance?
>> 
>No, it works like this:
>
> - pcpu 6 is eithe idle or it is running someone already (say d3v4)
>   + if pcpu 6 is idle, we tickle (i.e., we raise SCHEDULE_SOFTIRQ)
>     pcpu 6 itself, and we're done

Ok, it looks the behavior differs from 4.1 and current upstream. The upstream
just raise the softirq to pcpu6 when it is idle, while 4.1 would raise softirq
to both pcpu6 and other idlers even pcpu6 is idle.

I think current upstream is more cleaver.

>   + if pcpu 6 is running d3v4 and there is no other idle pcpu:
>     * if prio(d2v2) > prio(d3v4) we tickle pcpu 6 and we're done
>     * if prio(d2v2) < prio(d3v4) we just don't tickle anyone
>   + if pcpu 6 is running d3v4 and there are some idle pcpus:
>     * if prio(d2v2) > prio(d3v4) we tickle pcpu 6 _AND_ one or  [1]
>       more of the idle pcpus
>     * if prio(d2v2) < prio(d3v4) we _ONLY_ tickle one or more   [2]
>       of the idle pcpus
>
>Idea behind [1] is that d2v2 should preempt d3v4 on pcpu 6, and that's
>why we tickle pcpu 6. However, that would mean that d3v4 would be put
>back in pcpu's 6 runqueue. So, if there are idle pcpus, we tickle them
>so that one can come and steam d3v4.
>
>Idea behind [2] is that d3v4 should continue to run on pcpu 6, while
>d2v2 will be put in pcpu's 6 runqueue. However, if there are idle
>pcpus, we tickle them so that one can come and steal d2v2.
>
>What really happens is not always what is expected, because it's
>possible that, even if prio(d2v2)>prio(d3v4), an idler is quicker in
>waking up and stealing d2v2 from pcpu's 6 runqueue where it was stashed
>while pcpu 6 itself reschedules and enacts the preemption... But that's
>unlikely and, all in 

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Dario Faggioli]

On Wed, 2016-09-14 at 18:44 +0800, Wei Yang wrote:
> On Tue, Sep 13, 2016 at 01:30:17PM +0200, Dario Faggioli wrote:
> > 
> > Do you mind sharing just a bit more, such as:
> >  - number of pcpus
> >  - number of vcpus of the various VMs
> 160 pcpus
> 16 vcpus in VM and 8 VMs
> 
So, 16x8=128, which means you're not even oversubscribing. Maybe some
of the pcpus are hyperthreads and not full cores (are they?), but
still, this is not something I'd describe "super intensive cpu load".

Oh, well, there's dom0 of course. So, if dom0 has more than 160-128=32
vcpus (is this the case?), you technically are oversubscribing. But
then, what does dom0 do? Is it very busy doing some stuff on its own,
or running the backends/etc for the VMs? Can you check that with top,
xentop, and alike?

If the system is not overbooked, it's a bit strange that the scheduler
is the bottleneck.

> Yes, the "Blue Screen" is what we want to mimic the behavior from
> client.
> 
> The "Blue Screen" will force the hypervisor to do load balance in my
> mind.
> 
I've no idea what this means (but I don't know much of Windows and of
what happens when it crashes with a blue screen).

> > I'll have a quick look at how __runq_tickle() looks like in Xen
> > 4.1,
> > but there's very few chances I'll be available to provide detailed
> > review, advice, testing, etc., on such an old codebase. :-(
> > 
> > > 
> > > By looking at the code, what I missed may be in __runq_tickle(),
> > > which in case
> > > there are idle pcpu, schedule_softirq is also raised on them. By
> > > doing so,
> > > those idle pcpus would steal other busy tasks and may in chance
> > > get
> > > d2v2?
> > > 
> > Yes, it looks like, in Xen 4.1, this is more or less what happens.
> > The
> > idea is to always tickle pcpu 6, if the priority of the waking vcpu
> > is
> > higher than what is running there. 
> > 
> Hmm... in case there are idle pcpus and the priority of the waking
> vcpu is
> higher than what is running on pcpu 6, would pcpu 6 have more chance
> to run it?
> or other idle pcup would steal it from pcpu6? or they have equal
> chance?
> 
No, it works like this:

 - pcpu 6 is eithe idle or it is running someone already (say d3v4)
   + if pcpu 6 is idle, we tickle (i.e., we raise SCHEDULE_SOFTIRQ)
     pcpu 6 itself, and we're done
   + if pcpu 6 is running d3v4 and there is no other idle pcpu:
     * if prio(d2v2) > prio(d3v4) we tickle pcpu 6 and we're done
     * if prio(d2v2) < prio(d3v4) we just don't tickle anyone
   + if pcpu 6 is running d3v4 and there are some idle pcpus:
     * if prio(d2v2) > prio(d3v4) we tickle pcpu 6 _AND_ one or  [1]
       more of the idle pcpus
     * if prio(d2v2) < prio(d3v4) we _ONLY_ tickle one or more   [2]
       of the idle pcpus

Idea behind [1] is that d2v2 should preempt d3v4 on pcpu 6, and that's
why we tickle pcpu 6. However, that would mean that d3v4 would be put
back in pcpu's 6 runqueue. So, if there are idle pcpus, we tickle them
so that one can come and steam d3v4.

Idea behind [2] is that d3v4 should continue to run on pcpu 6, while
d2v2 will be put in pcpu's 6 runqueue. However, if there are idle
pcpus, we tickle them so that one can come and steal d2v2.

What really happens is not always what is expected, because it's
possible that, even if prio(d2v2)>prio(d3v4), an idler is quicker in
waking up and stealing d2v2 from pcpu's 6 runqueue where it was stashed
while pcpu 6 itself reschedules and enacts the preemption... But that's
unlikely and, all in all, not catastrophic (although, of course, worse
in terms of overhead, locality, etc)

> > If pcpu 6 was not idle, we also tickle one or more idle pcpus so
> > that:
> >  - if the waking vcpu preempted what was running on pcpu 6, an
> > idler
> >    can pick-up ("steal") such preempted vcpu and run it;
> >  - if the waking vcpu ended up in the runqueue, an idler can steal
> > it
> > 
> Hmm... I don't get the difference between these two cases.
> 
> Looks both are an idler steal the vcpu.
> 
I hope it's more clear now. :-)

> > Yes. But both in 4.1 and in current staging, we only raise
> > SCHEDULE_SOFTIRQ on pcpu 6, if and only if the priority of the
> > waking
> > vcpu is higher and it should preempt the currently running vcpu.
> > 
> Oh, you are right. I didn't catch this either.
> 
> This means in case 
> a) the priority is lower than current one 
> b) no idler in system
> 
> The vcpu will sit quietly in the runq and waiting for the schedule
> next time.
> 
Well, yes. It will stay in pcpu's 6 runqueue until either:
 - what's running on pcpu 6 blocks, or finishes its credits, etc.,
   and pcpu 6 reschedules and picks it up
 - some other pcpu becomes idle and, when poking around the various
   pcpus for stealing work, picks it up

Of course, both depends on in what position in pcpu's 6 runqueue the
vcpu is when the specific event happens.

This is a limit of the Credit1 scheduler, especially of the version
that you find in 4.1. Newer code is a bit better wrt this (although not

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Wei Yang]

On Tue, Sep 13, 2016 at 01:30:17PM +0200, Dario Faggioli wrote:
>[using xendevel correct address]
>
>On Tue, 2016-09-13 at 16:54 +0800, Wei Yang wrote:
>> On Fri, 2016-09-09 at 17:41 +0800, Wei Yang wrote:
>> > 
>> > I'm not surprised by that. Yet, I'd be interested in hearing more
>> > about this profiling you have done (things like, how you captured
>> > the data, what workloads you are exactly considering, how you
>> > determined what is the bottleneck, etc).
>> Let me try to explain this.
>> 
>> Workload: a. Blue Screen in Windows Guests
>>    b. some client using Windows to do some video
>> processing
>>   which need precise timestamp (we are not sure the
>> core
>>   reason but we see the system is very slow)
>>
>Do you mind sharing just a bit more, such as:
> - number of pcpus
> - number of vcpus of the various VMs

160 pcpus
16 vcpus in VM and 8 VMs

>
>I also am not sure what "a. Blue screen in Windows guests" above
>means... is there a workload called "Blue Screen"? Or is it that you
>are hitting a BSOD in some of your guests (which ones, what were they
>doing)? Or is it that you _want_ to provoke a BSOD on some of your
>guests? Or is that something else? :-P
>

Yes, the "Blue Screen" is what we want to mimic the behavior from client.

The "Blue Screen" will force the hypervisor to do load balance in my mind.

>> Capture the data: lock profile
>> Bottleneck Check: The schedule lock wait time is really high  
>> 
>Ok, cool. Interesting that lock profiling works on 4.1! :-O
>
>> > The scheduler tries to see whether the v->processor of the waking
>> > vcpu can be re-used, but that's not at all guaranteed, and again,
>> > on a very loaded system, not even that likely!
>> > 
>> Hmm... I may missed something.
>> 
>> Took your assumption below for example. 
>> In my mind, the process looks like this:
>> 
>> csched_vcpu_wake()
>> __runq_insert(),  insert the vcpu in pcpu's 6 runq
>> __runq_tickle(),  raise SCHEDULE_SOFTIRQ on pcpu 6 or other
>> (1)
>> 
>Well, yes. More precisely, at least in current staging,
>SCHEDULE_SOFTIRQ is raised for pcpu 6:
> - if pcpu 6 is idle, or
> - if pcpu 6 is not idle but there actually isn't any idle vcpu, and
>   the waking vcpu is higher in priority than what is running on pcpu 6
>
>> __do_softirq()
>> schedule()
>> csched_schedule(),  for pcpu 6, it may wake up d2v2 based
>> on it's
>> priority
>>
>Yes, but it is pcpu 6 that will run csched_schedule() only if the
>conditions mentioned above are met. If not, it will be some other pcpu
>that will do so (or none!).
>
>But I just realized that the fact that you are actually working on 4.1
>is going to be an issue. In fact, the code that it is now in Xen has
>changed **a lot** since 4.1. In fact, you're missing all the soft-
>affinity work (which you may or may not find useful) but also
>improvements and bugfixing.
>

That's true...

>I'll have a quick look at how __runq_tickle() looks like in Xen 4.1,
>but there's very few chances I'll be available to provide detailed
>review, advice, testing, etc., on such an old codebase. :-(
>
>> By looking at the code, what I missed may be in __runq_tickle(),
>> which in case
>> there are idle pcpu, schedule_softirq is also raised on them. By
>> doing so,
>> those idle pcpus would steal other busy tasks and may in chance get
>> d2v2?
>> 
>Yes, it looks like, in Xen 4.1, this is more or less what happens. The
>idea is to always tickle pcpu 6, if the priority of the waking vcpu is
>higher than what is running there. 
>

Hmm... in case there are idle pcpus and the priority of the waking vcpu is
higher than what is running on pcpu 6, would pcpu 6 have more chance to run it?
or other idle pcup would steal it from pcpu6? or they have equal chance?

>If pcpu 6 was not idle, we also tickle one or more idle pcpus so that:
> - if the waking vcpu preempted what was running on pcpu 6, an idler
>   can pick-up ("steal") such preempted vcpu and run it;
> - if the waking vcpu ended up in the runqueue, an idler can steal it
>

Hmm... I don't get the difference between these two cases.

Looks both are an idler steal the vcpu.

>> BTW, if the system is in heavy load, how much chance would we have
>> idle pcpu?
>> 
>It's not very likely that there will be idle pcpus in a very busy
>system, I agree.
>
>> > > We can divide this in three steps:
>> > > a) Send IPI to the target CPU and tell it which vcpu we want it
>> > > to 
>> > > wake up.
>> > > b) The interrupt handler on target cpu insert vcpu to a percpu
>> > > queue 
>> > > and raise
>> > >    softirq.
>> > > c) softirq will dequeue the vcpu and wake it up.
>> > > 
>> > I'm not sure I see how you think this would improve the situation.
>> > 
>> > So, quickly, right now:
>> > 
>> > - let's say vcpu 2 of domain 2 (from now d2v2) is waking up
>> > - let's assume d2v2->processor = 6
>> > - let's assume the wakeup is 

Re: [Xen-devel] [Help] Trigger Watchdog when adding an IPI in vcpu_wake

[Dario Faggioli]

[using xendevel correct address]

On Tue, 2016-09-13 at 16:54 +0800, Wei Yang wrote:
> On Fri, 2016-09-09 at 17:41 +0800, Wei Yang wrote:
> > 
> > I'm not surprised by that. Yet, I'd be interested in hearing more
> > about this profiling you have done (things like, how you captured
> > the data, what workloads you are exactly considering, how you
> > determined what is the bottleneck, etc).
> Let me try to explain this.
> 
> Workload: a. Blue Screen in Windows Guests
>     b. some client using Windows to do some video
> processing
>    which need precise timestamp (we are not sure the
> core
>    reason but we see the system is very slow)
>
Do you mind sharing just a bit more, such as:
 - number of pcpus
 - number of vcpus of the various VMs

I also am not sure what "a. Blue screen in Windows guests" above
means... is there a workload called "Blue Screen"? Or is it that you
are hitting a BSOD in some of your guests (which ones, what were they
doing)? Or is it that you _want_ to provoke a BSOD on some of your
guests? Or is that something else? :-P

> Capture the data: lock profile
> Bottleneck Check: The schedule lock wait time is really high  
> 
Ok, cool. Interesting that lock profiling works on 4.1! :-O

> > The scheduler tries to see whether the v->processor of the waking
> > vcpu can be re-used, but that's not at all guaranteed, and again,
> > on a very loaded system, not even that likely!
> > 
> Hmm... I may missed something.
> 
> Took your assumption below for example. 
> In my mind, the process looks like this:
> 
> csched_vcpu_wake()
> __runq_insert(),  insert the vcpu in pcpu's 6 runq
> __runq_tickle(),  raise SCHEDULE_SOFTIRQ on pcpu 6 or other
> (1)
> 
Well, yes. More precisely, at least in current staging,
SCHEDULE_SOFTIRQ is raised for pcpu 6:
 - if pcpu 6 is idle, or
 - if pcpu 6 is not idle but there actually isn't any idle vcpu, and
   the waking vcpu is higher in priority than what is running on pcpu 6

> __do_softirq()
> schedule()
> csched_schedule(),  for pcpu 6, it may wake up d2v2 based
> on it's
> priority
>
Yes, but it is pcpu 6 that will run csched_schedule() only if the
conditions mentioned above are met. If not, it will be some other pcpu
that will do so (or none!).

But I just realized that the fact that you are actually working on 4.1
is going to be an issue. In fact, the code that it is now in Xen has
changed **a lot** since 4.1. In fact, you're missing all the soft-
affinity work (which you may or may not find useful) but also
improvements and bugfixing.

I'll have a quick look at how __runq_tickle() looks like in Xen 4.1,
but there's very few chances I'll be available to provide detailed
review, advice, testing, etc., on such an old codebase. :-(

> By looking at the code, what I missed may be in __runq_tickle(),
> which in case
> there are idle pcpu, schedule_softirq is also raised on them. By
> doing so,
> those idle pcpus would steal other busy tasks and may in chance get
> d2v2?
> 
Yes, it looks like, in Xen 4.1, this is more or less what happens. The
idea is to always tickle pcpu 6, if the priority of the waking vcpu is
higher than what is running there. 

If pcpu 6 was not idle, we also tickle one or more idle pcpus so that:
 - if the waking vcpu preempted what was running on pcpu 6, an idler
   can pick-up ("steal") such preempted vcpu and run it;
 - if the waking vcpu ended up in the runqueue, an idler can steal it

> BTW, if the system is in heavy load, how much chance would we have
> idle pcpu?
> 
It's not very likely that there will be idle pcpus in a very busy
system, I agree.

> > > We can divide this in three steps:
> > > a) Send IPI to the target CPU and tell it which vcpu we want it
> > > to 
> > > wake up.
> > > b) The interrupt handler on target cpu insert vcpu to a percpu
> > > queue 
> > > and raise
> > >    softirq.
> > > c) softirq will dequeue the vcpu and wake it up.
> > > 
> > I'm not sure I see how you think this would improve the situation.
> > 
> > So, quickly, right now:
> > 
> > - let's say vcpu 2 of domain 2 (from now d2v2) is waking up
> > - let's assume d2v2->processor = 6
> > - let's assume the wakeup is happening on pcpu 4
> > 
> > Right now:
> > 
> > - on pcpu 4, vcpu_wake(d2v2) takes the scheduler lock of d2v2,
> >   which is the runqueue lock of pcpu 6 (in Credit1, there is 1
> >   runqueue per pcpu, and locks are per-cpu already)
> > - in csched_vcpu_wake(d2v2), d2v2 is inserted in pcpu's 6 runqueue
> > - still executing on pcpu 4, __runq_tickle() is called, and it
> >   determines on which pcpu d2v2 should run
> > - it raises the SCHEDULE_SOFTIRQ for such pcpu. Let's look at the
> >   following two cases:
> >    a) if it was determined that d2v2 should run on pcpu 6:
> >    - in a (little, hopefully) while, pcpu 6 will schedule
> >    - it takes its own runqueue lock
> >    - it finds d2v2 

Re: [Xen-devel] help -- A question about ' XENFEAT_auto_translated_physmap':

[Jan Beulich]

>>> On 10.08.16 at 15:19,  wrote:
> On August 09, 2016 9:02 PM, < jbeul...@suse.com > wrote:
>> >>> On 09.08.16 at 14:36,  wrote:
>> > Hi Jan,
>> >
>> > A question about ' XENFEAT_auto_translated_physmap':
>> >
>> > In linux code, in arch/x86/xen/mmu.c,
>>
>> I assume you know that I'm not a maintainer of the Linux code.
>>
>> >  __xen_pgd_walk()
>> > {
>> > 
>> >
>> > if (xen_feature(XENFEAT_auto_translated_physmap))
>> > return 0;
>> > 
>> > }
>> >
>> >
>> >
>> > Why xen_feature(XENFEAT_auto_translated_physmap) is true, then return
>> > directly?
>> > If not return directly, is there any potential risk?
>>
>> Well, the function is specifically there for operations (pinning/unpinning) 
> which are required only for the not-auto-translated case.
>> Why would anyone want to traverse a page table tree just to do nothing on 
> each of the entries?
> 
> Jan, thank you!!
> As I am struggling with a dom0 crash, the kernel is old, 3.0.X..
> Now there is a crash in 
> 
> [] _pin_lock+0x165/0x2a0<*crash*
> 
> unable to handle kernel paging request at 8b1021826000

Which appears to have pretty little to do with the original question,
as here you appear to be dealing with a forward ported XenoLinux
kernel.

> static void _pin_lock(struct mm_struct *mm, int lock)
> {
> 171 pgd_t *pgd = mm->pgd;
> 172 unsigned g;
> 173
> 174 for (g = 0; g <= ((TASK_SIZE_MAX-1) / PGDIR_SIZE); g++, 
> pgd++) {
> 175 pud_t *pud;
> 176 unsigned u;
> 177
> 178 if (pgd_none(*pgd))
> 179 continue;
> 180 pud = pud_offset(pgd, 0);
> 181 for (u = 0; u < PTRS_PER_PUD; u++, pud++) {
> 182 pmd_t *pmd;
> 183 unsigned m;
> 184
> 185 if (pud_none(*pud))
> 186 continue;
> 187 pmd = pmd_offset(pud, 0);
> 188 for (m = 0; m < PTRS_PER_PMD; m++, pmd++) 
> {
> 189 spinlock_t *ptl;
> 190
> 191 if (pmd_none(*pmd))  
> <-*crash*

Well, fine, it crashes here. But without further detail I don't think
you really expect anyone to be able to help (at the first glance
this looks like a follow-on effect from some earlier problem, as
presumably you've hit a ballooned out page, which obviously
shouldn't be referenced by any page tables)? And for this old a
kernel, and a non-upstream one, such a discussion presumably
doesn't belong on this list anyway. If this is a SUSE kernel, and
if the kernel is up-to-date wrt the distro it comes from, then either
open a ticket with our support folks, or (less reasonable, as there
never was an openSUSE version with a 3.0 based kernel) provide
much greater detail on a list like opensuse-virt...@opensuse.org.

Jan

> 192 continue;
> 193 ptl = pte_lockptr(0, pmd);
> 194 if (lock)
> 195 spin_lock(ptl);
> 196 else
> 197 spin_unlock(ptl);
> 198 }
> 199 }
> 200 }
> 201 }
> 202 #endif
> }
> 
> 
> 
> Quan




___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] help -- A question about ' XENFEAT_auto_translated_physmap':

[Xuquan (Euler)]

On August 09, 2016 9:02 PM, < jbeul...@suse.com > wrote:
> >>> On 09.08.16 at 14:36,  wrote:
> > Hi Jan,
> >
> > A question about ' XENFEAT_auto_translated_physmap':
> >
> > In linux code, in arch/x86/xen/mmu.c,
>
> I assume you know that I'm not a maintainer of the Linux code.
>
> >  __xen_pgd_walk()
> > {
> > 
> >
> > if (xen_feature(XENFEAT_auto_translated_physmap))
> > return 0;
> > 
> > }
> >
> >
> >
> > Why xen_feature(XENFEAT_auto_translated_physmap) is true, then return
> > directly?
> > If not return directly, is there any potential risk?
>
> Well, the function is specifically there for operations (pinning/unpinning) 
> which are required only for the not-auto-translated case.
> Why would anyone want to traverse a page table tree just to do nothing on 
> each of the entries?

Jan, thank you!!
As I am struggling with a dom0 crash, the kernel is old, 3.0.X..
Now there is a crash in 

[] _pin_lock+0x165/0x2a0<*crash*

unable to handle kernel paging request at 8b1021826000

static void _pin_lock(struct mm_struct *mm, int lock)
{
171 pgd_t *pgd = mm->pgd;
172 unsigned g;
173
174 for (g = 0; g <= ((TASK_SIZE_MAX-1) / PGDIR_SIZE); g++, 
pgd++) {
175 pud_t *pud;
176 unsigned u;
177
178 if (pgd_none(*pgd))
179 continue;
180 pud = pud_offset(pgd, 0);
181 for (u = 0; u < PTRS_PER_PUD; u++, pud++) {
182 pmd_t *pmd;
183 unsigned m;
184
185 if (pud_none(*pud))
186 continue;
187 pmd = pmd_offset(pud, 0);
188 for (m = 0; m < PTRS_PER_PMD; m++, pmd++) {
189 spinlock_t *ptl;
190
191 if (pmd_none(*pmd))  
<-*crash*
192 continue;
193 ptl = pte_lockptr(0, pmd);
194 if (lock)
195 spin_lock(ptl);
196 else
197 spin_unlock(ptl);
198 }
199 }
200 }
201 }
202 #endif
}



Quan




___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] help -- A question about ' XENFEAT_auto_translated_physmap':

[Jan Beulich]

>>> On 09.08.16 at 14:36,  wrote:
> Hi Jan,
> 
> A question about ' XENFEAT_auto_translated_physmap':
> 
> In linux code, in arch/x86/xen/mmu.c,

I assume you know that I'm not a maintainer of the Linux code.

>  __xen_pgd_walk()
> {
> 
> 
> if (xen_feature(XENFEAT_auto_translated_physmap))
> return 0;
> 
> }
> 
> 
> 
> Why xen_feature(XENFEAT_auto_translated_physmap) is true, then return 
> directly?
> If not return directly, is there any potential risk?

Well, the function is specifically there for operations (pinning/unpinning)
which are required only for the not-auto-translated case. Why would
anyone want to traverse a page table tree just to do nothing on each
of the entries?

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] help

[Doug Goldstein]

On 3/15/16 4:01 PM, Safa Hamza wrote:
> i solve the problem by degugging the linux kernel .. the problem was the
> instruction omap_smc1 ..i'm not sure about what it did but when i delete
> this instruction it works ... but i face another problem .. the execution
> now stops in



> [2.610379] VFS: Mounted root (ext4 filesystem) on device 179:2.
> [2.623192] devtmpfs: mounted
> [2.623584] Freeing unused kernel memory: 340K (c079e000 - c07f3000)
> INIT: version 2.88 booting
> Starting udev
> [3.169771] udev[871]: starting version 164
> [5.205065] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some
> data may be corrupt. Please run fsck.
> [5.400187] EXT4-fs (mmcblk1p4): recovery complete
> [5.400309] EXT4-fs (mmcblk1p4): mounted filesystem with ordered data
> mode. Opts: (null)
> [5.492162] random: nonblocking pool is initialized
> ERROR: could not open directory
> /lib/modules/3.15.0-rc2-00199-g0c0a3e5-dirty: No such file or directory
> WARNING: -e needs -E or -F
> ERROR: could not open directory
> /lib/modules/3.15.0-rc2-00199-g0c0a3e5-dirty: No such file or directory
> FATAL: could not search modules: No such file or directory
> Starting Bootlog daemon: bootlogd: cannot allocate pseudo tty: No such file
> or directory
> bootlogd.
> [5.831227] EXT4-fs (mmcblk0p2): re-mounted. Opts: data=ordered
> ALSA: Restoring mixer settings...
> Configuring network interfaces... /usr/sbin/alsactl: load_state:1686: No
> soundcards found...
> ifconfig: SIOCGIFFLAGS: No such device
> done.
> hwclock: can't open '/dev/misc/rtc': No such file or directory
> Mon Jul  1 15:13:00 UTC 2013
> hwclock: can't open '/dev/misc/rtc': No such file or directory
> INIT: Entering runlevel: 5
> Starting telnet daemon.
> hwclock: can't open '/dev/misc/rtc': No such file or directory
> Starting syslogd/klogd: done
> Starting thttpd.
> Stopping Bootlog daemon: bootlogd.
> 
> ***
>  no console appears .. i guess the problem in the file system ... but this
> file works fine with the linux kernel in omap5 without  the xen hypervisor
> 
> should i make any changement in the rootfs so it can works with xen
> hypervisor !! or any other solution !!
> Regards
> 

The issue is likely that you don't have a getty setup on /dev/hvc0. Edit
your /etc/inittab and add one. Maybe something like:

h0:2345:respawn:/sbin/agetty 38400 hvc0 linux

But honestly there's other problems with the image you are using so
you'll likely run into more but those look like Linux/distro related
problems.

Please make sure you reply to the list if you do reply to this.

-- 
Doug Goldstein



signature.asc
Description: OpenPGP digital signature
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help

[Safa Hamza]

i solve the problem by degugging the linux kernel .. the problem was the
instruction omap_smc1 ..i'm not sure about what it did but when i delete
this instruction it works ... but i face another problem .. the execution
now stops in
 
***
U-Boot SPL 2013.10-rc2 (Mar 08 2016 - 14:23:51)
OMAP5432 ES2.0
SPL: Please implement spl_start_uboot() for your board
SPL: Direct Linux boot not active!
reading u-boot.img
reading u-boot.img


U-Boot 2013.10-rc2 (Mar 08 2016 - 14:23:51)

CPU  : OMAP5432 ES2.0
Board: OMAP5432 uEVM
I2C:   ready
DRAM:  2 GiB
MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
Net:   No ethernet found.
Hit any key to stop autoboot:  0
mmc0 is current device
reading boot.scr
972 bytes read in 5 ms (189.5 KiB/s)
Running bootscript from mmc0 ...
## Executing script at 8200
reading omap5-uevm.dtb
44836 bytes read in 8 ms (5.3 MiB/s)
reading xen-uImage
787372 bytes read in 44 ms (17.1 MiB/s)
reading zImage
4283720 bytes read in 215 ms (19 MiB/s)
## Booting kernel from Legacy Image at 9000 ...
   Image Name:
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:787308 Bytes = 768.9 KiB
   Load Address: 8020
   Entry Point:  8020
   Verifying Checksum ... OK
## Flattened Device Tree blob at 825f
   Booting using the fdt blob at 0x825f
   Loading Kernel Image ... OK
   reserving fdt memory region: addr=825f size=c000
   Using Device Tree in place at 825f, end 825fefff

Starting kernel ...

- UART enabled -
- CPU  booting -
- Xen starting in Hyp mode -
- Zero BSS -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) Checking for initrd in /chosen
(XEN) RAM: 8000 - feff
(XEN)
(XEN) MODULE[0]: 825f - 825fc000 Device Tree
(XEN) MODULE[1]: a000 - a0a0 Kernel
(XEN)  RESVD[0]: 825f - 825fc000
(XEN)
(XEN) Command line: sync_console console=dtuart dtuart=serial2
(XEN) Placing Xen at 0xfee0-0xff00
(XEN) Update BOOTMOD_XEN from 8020-80305701 =>
fee0-fef05701
(XEN) Xen heap: fa00-fe00 (16384 pages)
(XEN) Dom heap: 503808 pages
(XEN) Domain heap initialised
(XEN) Platform: TI OMAP5
(XEN) Looking for dtuart at "serial2", options ""
 Xen 4.7-unstable
(XEN) Xen version 4.7-unstable (root@) (arm-linux-gnueabihf-gcc (Linaro GCC
2014.11) 4.9.3 20141031 (prerelease)) debug=y Tue Mar  1 15:02:30 CET 2016
(XEN) Latest ChangeSet: Sun Feb 28 16:03:51 2016 -0500 git:42391c6
(XEN) Console output is synchronous.
(XEN) Processor: 412fc0f2: "ARM Limited", variant: 0x2, part 0xc0f, rev 0x2
(XEN) 32-bit Execution:
(XEN)   Processor Features: 1131:00011011
(XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 ThumbEE Jazelle
(XEN) Extensions: GenericTimer Security
(XEN)   Debug Features: 02010555
(XEN)   Auxiliary Features: 
(XEN)   Memory Model Features: 10201105 2000 0124 02102211
(XEN)  ISA Features: 02101110 13112111 21232041 2131 10011142 
(XEN) Set AuxCoreBoot1 to fee0004c (0020004c)
(XEN) Set AuxCoreBoot0 to 0x20
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 6144 KHz
(XEN) GICv2: WARNING: The GICC size is too small: 0x1000 expected 0x2000
(XEN) GICv2 initialization:
(XEN) gic_dist_addr=48211000
(XEN) gic_cpu_addr=48212000
(XEN) gic_hyp_addr=48214000
(XEN) gic_vcpu_addr=48216000
(XEN) gic_maintenance_irq=25
(XEN) GICv2: 192 lines, 2 cpus, secure (IID 043b).
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Allocated console ring of 16 KiB.
(XEN) VFP implementer 0x41 architecture 4 part 0x30 variant 0xf rev 0x0
(XEN) Bringing up CPU1
- CPU 0001 booting -
- Xen starting in Hyp mode -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU 1 booted.
(XEN) Brought up 2 CPUs
(XEN) P2M: 40-bit IPA
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80003558
(XEN) I/O virtualisation disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading kernel from boot module @ a000
(XEN) Allocating 1:1 mappings totalling 128MB for dom0:
(XEN) BANK[0] 0x00a800-0x00b000 (128MB)
(XEN) Grant table range: 0x00fee0-0x00fee63000
(XEN) Loading zImage from a000 to
afa0-afe15d48
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading dom0 DTB to 0xaf80-0xaf80aece
(XEN) Scrubbing Free RAM on 1 nodes using 2 CPUs
(XEN) done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) **
(XEN) *** WARNING: CONSOLE OUTPUT IS SYNCHRONOUS
(XEN) *** This option is intended to aid debugging of Xen by ensuring
(XEN) 

Re: [Xen-devel] help

[Julien Grall]

On 15/03/16 18:21, Konrad Rzeszutek Wilk wrote:

On Tue, Mar 15, 2016 at 06:16:44PM +, Julien Grall wrote:



On 15/03/16 18:09, Konrad Rzeszutek Wilk wrote:

On Tue, Mar 15, 2016 at 05:56:46PM +, Julien Grall wrote:

Hi Safa,

On 11/03/16 17:16, Wei Liu wrote:

Add back xen-devel

On Fri, Mar 11, 2016 at 05:23:22PM +0100, Safa Hamza wrote:

ok .. can u tell me how compile xen with debug symbols !!  i have xen-syms
after compiling xen with "make dist-xen XEN_TARGET_ARCH=arm32
CROSS_COMPILE=arm-linux-gnueabihf- CONFIG_EARLY_PRINTK=omap5432"   is this
the kernel  with  debug symbols



If you are using Xen unstable, the hypervisor will be compiled with debug
enabled by default. If not you can add "debug=y" on the build command line.

xen/xen-syms contains the debug symbols, you can use tools such as addr2line
to get more debug information. However, it's not suitable to boot xen on


s/xen/xen-syms/ ?


s/xen/the hypervisor/


You are saying you can't boot Xen on this board?


I meant, xen-syms is not suitable to boot the hypervisor on this board. 
You have to use xen/xen.


Sorry for the confusion.

Regards,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help

[Konrad Rzeszutek Wilk]

On Tue, Mar 15, 2016 at 06:16:44PM +, Julien Grall wrote:
> 
> 
> On 15/03/16 18:09, Konrad Rzeszutek Wilk wrote:
> >On Tue, Mar 15, 2016 at 05:56:46PM +, Julien Grall wrote:
> >>Hi Safa,
> >>
> >>On 11/03/16 17:16, Wei Liu wrote:
> >>>Add back xen-devel
> >>>
> >>>On Fri, Mar 11, 2016 at 05:23:22PM +0100, Safa Hamza wrote:
> ok .. can u tell me how compile xen with debug symbols !!  i have xen-syms
> after compiling xen with "make dist-xen XEN_TARGET_ARCH=arm32
> CROSS_COMPILE=arm-linux-gnueabihf- CONFIG_EARLY_PRINTK=omap5432"   is this
> the kernel  with  debug symbols
> 
> >>
> >>If you are using Xen unstable, the hypervisor will be compiled with debug
> >>enabled by default. If not you can add "debug=y" on the build command line.
> >>
> >>xen/xen-syms contains the debug symbols, you can use tools such as addr2line
> >>to get more debug information. However, it's not suitable to boot xen on
> >
> >s/xen/xen-syms/ ?
> 
> s/xen/the hypervisor/

You are saying you can't boot Xen on this board?

> 
> Regards,
> 
> -- 
> Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help

[Julien Grall]

On 15/03/16 18:09, Konrad Rzeszutek Wilk wrote:

On Tue, Mar 15, 2016 at 05:56:46PM +, Julien Grall wrote:

Hi Safa,

On 11/03/16 17:16, Wei Liu wrote:

Add back xen-devel

On Fri, Mar 11, 2016 at 05:23:22PM +0100, Safa Hamza wrote:

ok .. can u tell me how compile xen with debug symbols !!  i have xen-syms
after compiling xen with "make dist-xen XEN_TARGET_ARCH=arm32
CROSS_COMPILE=arm-linux-gnueabihf- CONFIG_EARLY_PRINTK=omap5432"   is this
the kernel  with  debug symbols



If you are using Xen unstable, the hypervisor will be compiled with debug
enabled by default. If not you can add "debug=y" on the build command line.

xen/xen-syms contains the debug symbols, you can use tools such as addr2line
to get more debug information. However, it's not suitable to boot xen on


s/xen/xen-syms/ ?


s/xen/the hypervisor/

Regards,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help

[Konrad Rzeszutek Wilk]

On Tue, Mar 15, 2016 at 05:56:46PM +, Julien Grall wrote:
> Hi Safa,
> 
> On 11/03/16 17:16, Wei Liu wrote:
> >Add back xen-devel
> >
> >On Fri, Mar 11, 2016 at 05:23:22PM +0100, Safa Hamza wrote:
> >>ok .. can u tell me how compile xen with debug symbols !!  i have xen-syms
> >>after compiling xen with "make dist-xen XEN_TARGET_ARCH=arm32
> >>CROSS_COMPILE=arm-linux-gnueabihf- CONFIG_EARLY_PRINTK=omap5432"   is this
> >>the kernel  with  debug symbols
> >>
> 
> If you are using Xen unstable, the hypervisor will be compiled with debug
> enabled by default. If not you can add "debug=y" on the build command line.
> 
> xen/xen-syms contains the debug symbols, you can use tools such as addr2line
> to get more debug information. However, it's not suitable to boot xen on

s/xen/xen-syms/ ?
> your board.
> 
> xen/xen is the actual hypervisor binary.
> 
> Regards,
> 
> -- 
> Julien Grall
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help

[Julien Grall]

Hi Safa,

On 11/03/16 17:16, Wei Liu wrote:

Add back xen-devel

On Fri, Mar 11, 2016 at 05:23:22PM +0100, Safa Hamza wrote:

ok .. can u tell me how compile xen with debug symbols !!  i have xen-syms
after compiling xen with "make dist-xen XEN_TARGET_ARCH=arm32
CROSS_COMPILE=arm-linux-gnueabihf- CONFIG_EARLY_PRINTK=omap5432"   is this
the kernel  with  debug symbols



If you are using Xen unstable, the hypervisor will be compiled with 
debug enabled by default. If not you can add "debug=y" on the build 
command line.


xen/xen-syms contains the debug symbols, you can use tools such as 
addr2line to get more debug information. However, it's not suitable to 
boot xen on your board.


xen/xen is the actual hypervisor binary.

Regards,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help

[Wei Liu]

Add back xen-devel

On Fri, Mar 11, 2016 at 05:23:22PM +0100, Safa Hamza wrote:
> ok .. can u tell me how compile xen with debug symbols !!  i have xen-syms
> after compiling xen with "make dist-xen XEN_TARGET_ARCH=arm32
> CROSS_COMPILE=arm-linux-gnueabihf- CONFIG_EARLY_PRINTK=omap5432"   is this
> the kernel  with  debug symbols
> 

I'm not sure if you did the right thing because I've never done any ARM
development. I'll let other people answer your question.

Wei.

> On Fri, Mar 11, 2016 at 5:09 PM, Wei Liu  wrote:
> 
> > On Fri, Mar 11, 2016 at 11:02:26AM -0500, Konrad Rzeszutek Wilk wrote:
> > > On Fri, Mar 11, 2016 at 04:47:47PM +0100, Safa Hamza wrote:
> > > > now i did just like u said ...  a new error appears
> > >
> > > Adding XEn-devel back. Please reply all.
> > >
> > > >
> > **
> > > > U-Boot# fdt addr $dtb_addr_r
> > > > U-Boot# fdt resize
> > > > U-Boot# fdt set /chosen xen,xen-bootargs \"$xen_bootargs\"
> > > > U-Boot# fdt resize
> > > > U-Boot# fdt set /chosen xen,dom0-bootargs \"$dom0_bootargs\"
> > > > U-Boot# fdt resize
> > > > U-Boot# fdt mknode /chosen modules
> > > > U-Boot# fdt set /chosen/modules '#address-cells' <1>
> > > > U-Boot# fdt set /chosen/modules '#size-cells' <1>
> > > > U-Boot# fdt mknode /chosen/modules module@0
> > > > libfdt fdt_add_subnode(): FDT_ERR_NOSPACE
> > > >
> > **
> > > > but when i wrote  fdt resize before  fdt mknode /chosen/modules
> > module@0
> > > > this error disappear but still the execution stops as i mentioned
> > before
> > > >
> >
> > The message seems quite straight-forward to me -- one of the libfdt
> > function call failed with some error.
> >
> > I'm afraid you need to do some manual debugging to figure out what went
> > wrong.
> >
> > Wei.
> >
> > > > On Fri, Mar 11, 2016 at 4:20 PM, Konrad Rzeszutek Wilk <
> > > > konrad.w...@oracle.com> wrote:
> > > >
> > > > > On Fri, Mar 11, 2016 at 10:20:01AM -0500, Konrad Rzeszutek Wilk
> > wrote:
> > > > > > On Fri, Mar 11, 2016 at 04:05:58PM +0100, Safa Hamza wrote:
> > > > >
> > > > > And please do not drop Xen-devel. Adding it back on.
> > > > >
> > > > > > > i did like u said but nothing change ..
> > > > > > >
> > > > > >
> > > > > > No you didn't. See below:
> > > > > > > U-Boot# setenv dom0_bootargs 'console=hvc0,115200n8
> > earlyprintk=xen
> > > > > debug'
> > > > > >
> > > > > > You still have 115200n8
> > > > >
> > >
> > > ___
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > http://lists.xen.org/xen-devel
> >

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help

[Wei Liu]

On Fri, Mar 11, 2016 at 11:02:26AM -0500, Konrad Rzeszutek Wilk wrote:
> On Fri, Mar 11, 2016 at 04:47:47PM +0100, Safa Hamza wrote:
> > now i did just like u said ...  a new error appears
> 
> Adding XEn-devel back. Please reply all.
> 
> > **
> > U-Boot# fdt addr $dtb_addr_r
> > U-Boot# fdt resize
> > U-Boot# fdt set /chosen xen,xen-bootargs \"$xen_bootargs\"
> > U-Boot# fdt resize
> > U-Boot# fdt set /chosen xen,dom0-bootargs \"$dom0_bootargs\"
> > U-Boot# fdt resize
> > U-Boot# fdt mknode /chosen modules
> > U-Boot# fdt set /chosen/modules '#address-cells' <1>
> > U-Boot# fdt set /chosen/modules '#size-cells' <1>
> > U-Boot# fdt mknode /chosen/modules module@0
> > libfdt fdt_add_subnode(): FDT_ERR_NOSPACE
> > **
> > but when i wrote  fdt resize before  fdt mknode /chosen/modules module@0
> > this error disappear but still the execution stops as i mentioned before
> > 

The message seems quite straight-forward to me -- one of the libfdt
function call failed with some error.

I'm afraid you need to do some manual debugging to figure out what went
wrong.

Wei.

> > On Fri, Mar 11, 2016 at 4:20 PM, Konrad Rzeszutek Wilk <
> > konrad.w...@oracle.com> wrote:
> > 
> > > On Fri, Mar 11, 2016 at 10:20:01AM -0500, Konrad Rzeszutek Wilk wrote:
> > > > On Fri, Mar 11, 2016 at 04:05:58PM +0100, Safa Hamza wrote:
> > >
> > > And please do not drop Xen-devel. Adding it back on.
> > >
> > > > > i did like u said but nothing change ..
> > > > >
> > > >
> > > > No you didn't. See below:
> > > > > U-Boot# setenv dom0_bootargs 'console=hvc0,115200n8 earlyprintk=xen
> > > debug'
> > > >
> > > > You still have 115200n8
> > >
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help

[Safa Hamza]

i did just like u said ...  a new error appears
**
U-Boot# fdt addr $dtb_addr_r
U-Boot# fdt resize
U-Boot# fdt set /chosen xen,xen-bootargs \"$xen_bootargs\"
U-Boot# fdt resize
U-Boot# fdt set /chosen xen,dom0-bootargs \"$dom0_bootargs\"
U-Boot# fdt resize
U-Boot# fdt mknode /chosen modules
U-Boot# fdt set /chosen/modules '#address-cells' <1>
U-Boot# fdt set /chosen/modules '#size-cells' <1>
U-Boot# fdt mknode /chosen/modules module@0
libfdt fdt_add_subnode(): FDT_ERR_NOSPACE
**
but when i wrote  fdt resize before  fdt mknode /chosen/modules module@0
this error disappear but still the execution stops as i mentioned before

On Fri, Mar 11, 2016 at 5:02 PM, Konrad Rzeszutek Wilk <
konrad.w...@oracle.com> wrote:

> On Fri, Mar 11, 2016 at 04:47:47PM +0100, Safa Hamza wrote:
> > now i did just like u said ...  a new error appears
>
> Adding XEn-devel back. Please reply all.
>
> >
> **
> > U-Boot# fdt addr $dtb_addr_r
> > U-Boot# fdt resize
> > U-Boot# fdt set /chosen xen,xen-bootargs \"$xen_bootargs\"
> > U-Boot# fdt resize
> > U-Boot# fdt set /chosen xen,dom0-bootargs \"$dom0_bootargs\"
> > U-Boot# fdt resize
> > U-Boot# fdt mknode /chosen modules
> > U-Boot# fdt set /chosen/modules '#address-cells' <1>
> > U-Boot# fdt set /chosen/modules '#size-cells' <1>
> > U-Boot# fdt mknode /chosen/modules module@0
> > libfdt fdt_add_subnode(): FDT_ERR_NOSPACE
> >
> **
> > but when i wrote  fdt resize before  fdt mknode /chosen/modules module@0
> > this error disappear but still the execution stops as i mentioned before
> >
> > On Fri, Mar 11, 2016 at 4:20 PM, Konrad Rzeszutek Wilk <
> > konrad.w...@oracle.com> wrote:
> >
> > > On Fri, Mar 11, 2016 at 10:20:01AM -0500, Konrad Rzeszutek Wilk wrote:
> > > > On Fri, Mar 11, 2016 at 04:05:58PM +0100, Safa Hamza wrote:
> > >
> > > And please do not drop Xen-devel. Adding it back on.
> > >
> > > > > i did like u said but nothing change ..
> > > > >
> > > >
> > > > No you didn't. See below:
> > > > > U-Boot# setenv dom0_bootargs 'console=hvc0,115200n8 earlyprintk=xen
> > > debug'
> > > >
> > > > You still have 115200n8
> > >
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help

[Konrad Rzeszutek Wilk]

On Fri, Mar 11, 2016 at 04:47:47PM +0100, Safa Hamza wrote:
> now i did just like u said ...  a new error appears

Adding XEn-devel back. Please reply all.

> **
> U-Boot# fdt addr $dtb_addr_r
> U-Boot# fdt resize
> U-Boot# fdt set /chosen xen,xen-bootargs \"$xen_bootargs\"
> U-Boot# fdt resize
> U-Boot# fdt set /chosen xen,dom0-bootargs \"$dom0_bootargs\"
> U-Boot# fdt resize
> U-Boot# fdt mknode /chosen modules
> U-Boot# fdt set /chosen/modules '#address-cells' <1>
> U-Boot# fdt set /chosen/modules '#size-cells' <1>
> U-Boot# fdt mknode /chosen/modules module@0
> libfdt fdt_add_subnode(): FDT_ERR_NOSPACE
> **
> but when i wrote  fdt resize before  fdt mknode /chosen/modules module@0
> this error disappear but still the execution stops as i mentioned before
> 
> On Fri, Mar 11, 2016 at 4:20 PM, Konrad Rzeszutek Wilk <
> konrad.w...@oracle.com> wrote:
> 
> > On Fri, Mar 11, 2016 at 10:20:01AM -0500, Konrad Rzeszutek Wilk wrote:
> > > On Fri, Mar 11, 2016 at 04:05:58PM +0100, Safa Hamza wrote:
> >
> > And please do not drop Xen-devel. Adding it back on.
> >
> > > > i did like u said but nothing change ..
> > > >
> > >
> > > No you didn't. See below:
> > > > U-Boot# setenv dom0_bootargs 'console=hvc0,115200n8 earlyprintk=xen
> > debug'
> > >
> > > You still have 115200n8
> >

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help

[Konrad Rzeszutek Wilk]

On Fri, Mar 11, 2016 at 10:20:01AM -0500, Konrad Rzeszutek Wilk wrote:
> On Fri, Mar 11, 2016 at 04:05:58PM +0100, Safa Hamza wrote:

And please do not drop Xen-devel. Adding it back on.

> > i did like u said but nothing change ..
> > 
> 
> No you didn't. See below:
> > U-Boot# setenv dom0_bootargs 'console=hvc0,115200n8 earlyprintk=xen debug'
> 
> You still have 115200n8

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help

[Konrad Rzeszutek Wilk]

On Thu, Mar 10, 2016 at 09:04:22PM +0100, Safa Hamza wrote:
> hello
> i'm trying to run xen on omap5 following
> this
> http://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/OMAP5432_uEVM
> 
> the execution stops at this point
> 
> **
> U-Boot SPL 2013.10-rc2 (Mar 08 2016 - 14:23:51)
> OMAP5432 ES2.0
> SPL: Please implement spl_start_uboot() for your board
> SPL: Direct Linux boot not active!
> reading u-boot.img
> reading u-boot.img
> 
> 
> U-Boot 2013.10-rc2 (Mar 08 2016 - 14:23:51)
> 
> CPU  : OMAP5432 ES2.0
> Board: OMAP5432 uEVM
> I2C:   ready
> DRAM:  2 GiB
> MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
> Using default environment
> 
> Net:   No ethernet found.
> Hit any key to stop autoboot:  0
> mmc0 is current device
> reading boot.scr
> ** Unable to read file boot.scr **
> reading uEnv.txt
> ** Unable to read file uEnv.txt **
> ** File not found /boot/zImage **
> U-Boot# setenv dtb_addr_r 0x825f
> U-Boot# setenv xen_addr_r 0x9000
> U-Boot# setenv kernel_addr_r 0xa000
> U-Boot# setenv xen_bootargs 'sync_console console=dtuart dtuart=serial2'
> U-Boot# setenv dom0_bootargs 'console=hvc0,115200n8 earlyprintk=xen debug

That does not look right.

console=hvc0 earlyprintk=xen debug 

is more right.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Help in fixing a issue

[Wei Liu]

On Fri, Feb 05, 2016 at 09:15:52PM +1100, PREETI MISHRA wrote:
> I have a patch in Xen which stores some information of VM process. I have
> another program running in Dom0 which intercept this information.
> 

I'm not sure I can parse this sentence. It's too vague for what you want
to do. For one, I don't know what "VM process" is.

> i) I want to configure my patch running in Xen to send the alert
> notification to program running in Dom0 to read data, probably using event
> channels. How to configure event channel?
> ii) Which API or command will be used to read the data from patch buffer.
> 

If you're writing user space program, code in QEMU might be a good
reference.

Please search for files like "xen_nic.c" in QEMU source code and start
from there.

You can also explore various Xen driver code inside Linux if you're
keen.

Another thing that might be of interest to you is a program lives in
xen.git called xentrace. It extracts buffer from hypervisor, which might
be what you need.

Wei.

> I am running xen 4.6 in ubuntu 14.04
> 
> regards,
> 
> Preeti

> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Help in fixing a issue

[Wei Liu]

Add back xen-devel, please use "reply-all" in the future.

And please don't top-post.

On Fri, Feb 05, 2016 at 10:01:57PM +1100, PREETI MISHRA wrote:
> Thanks for the reply,
> 
> actually, I have a virtual machine in which some processes are running. I
> want to analysis their behavior using VMI at xen.
> 

Have you checked out libvmi in this case?

> My tool has two components:i) xen patch running at hypervisor ii) analyzing
> component running at Dom0
> 
> 1. Xen patch is responsible for collecting the system call information of a
> monitored process. It can be syscall no and process id.
>   - how to take this information? //particularly i want to trap any
> execution of monitored programs.
>   - how to store this information in buffer?
>-how to send alert to my another daemon (analyzing component)
> running in Dom0.

See xentrace / xenanalyze in xen.git.

> 2. On receiving alert, analyzing component perform analysis over the
> collected data.
>   - how to send the buffer information (pid and system call no) to
> analyzing component?
> 

Anyway, I'm not expert on this. I will let other people to have a look.

My gut feeling is that libvmi should be the way to go.

Wei.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Help in fixing a issue

[Wei Liu]

BTW please check out:

 http://wiki.xen.org/wiki/Asking_Xen_Devel_Questions

Wei.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help for dirty bitmap

[Ian Campbell]

On Sun, 2015-11-15 at 13:13 +0800, 高强 wrote:
> Hi,alls
> 
>     As we know,xen 4+ maintains a dirty bitmap. Snapshot or migration
> will use this bitmap. When the guest domain changes the memory page， VMM
> can capture it and set dirty bitmap to 1. 
> 
>     But I can't know how VMM to maintain this dirty bitmap. Under what
> circumstances VMM will set the dirty page bitmap to 1, and how to track
> the page is be dirty?

The keyword you are looking for is "logdirty". Grepping the source base for
that should get you on the right track.

Ian.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help for snapshot

[Ian Campbell]

On Wed, 2015-10-21 at 11:06 +0800, 高强 wrote:
> Hi,alls
> 
> I 'm modifying the source code of snapshot in xen-4.4.1.

That is now 3 Xen releases ago.

I suggest that if you want help with new developments you should target the
master branch in xen.git (which is targetted becoming 4.7) or as an
absolute minimum the latest release branch (stable-4.6).

A lot has changed WRT migration and related technologies in the 4.6
release, working on new functionality for such things against 4.4 at this
is not of much interest to the community I'm afraid.

Ian.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help for snapshot

[Dario Faggioli]

On Wed, 2015-10-21 at 10:07 +0100, Ian Campbell wrote:

> I suggest that if you want help with new developments you should
> target the
> master branch in xen.git (which is targetted becoming 4.7) or as an
> absolute minimum the latest release branch (stable-4.6).
> 
> A lot has changed WRT migration and related technologies in the 4.6
> release, working on new functionality for such things against 4.4 at
> this
> is not of much interest to the community I'm afraid.
> 
Not to mention that the new migration code you'll find in the code
branches that Ian mentioned, is most likely going to be better
structured and easier to understand and modify.

Regards,
Dario
-- 
<> (Raistlin Majere)
-
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK)



signature.asc
Description: This is a digitally signed message part
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help for snapshot

[Wei Liu]

On Wed, Oct 21, 2015 at 11:06:07AM +0800, 高强 wrote:
> Hi,alls
> 
> I 'm modifying the source code of snapshot in xen-4.4.1. I know the memory
> copy and cpu state copy are all in xc_domain_save.c--xc_domain_save().
> 
> But I don't sure where is the device state copy. Who can tell me the
> position of device copy in source code.
> 

Please try using grep to search for relevant strings inside
tools/libx{c,l} and ask more specific questions.

http://wiki.xenproject.org/wiki/Asking_Developer_Questions

Wei.


> Thanks

> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help for python log in xen-4.4.1

[Wei Liu]

Please don't top-post.

On Tue, Oct 20, 2015 at 10:58:56AM +0800, 高强 wrote:
> Hi,
> 
> If you know, could you tell me the call relationship of command "xl save".
> Example, when I execute "xl save",which function will be called first,and
> then call which function? I mainly want to know which python function will
> be called.
> 

No python code will be called if you're using xl toolstack.

The code for xl and the underly library libxl is in tools/libxl directory.
Most of the time, you should be able to get the call chain by simply
inspecting the code.

Wei.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help for python log in xen-4.4.1

[Wei Liu]

On Mon, Oct 19, 2015 at 10:22:52AM +0800, 高强 wrote:
> Hi,alls
> 
> 
> There are some python code in xen-4.4.1,and in the python code there are
> lots debug information such as
> log.debug("***"),log.exception("***"),XendError("*")and so on.
> 
> I want to get this kind of information but I don't know where to get them.
> Can someone tell me?
> 
> else,I try to use the followed method to tracing program execution process:
> 
> import sys
> f=open("/home/*/python.log","a+")
> print >> f, "Save.py is start\n"
> f.close()
> 
> But,I can't find the file python.log while executed end,who can tell me why?
> 

Xen 4.4 is using xl toolstack. Xend is not used so no wonder you
couldn't find any logs.

If you describe your end goal we might provide more concrete
suggestions.


Wei.

> Thanks!

> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help for python log in xen-4.4.1

[高强]

Hi,

If you know, could you tell me the call relationship of command "xl save".
Example, when I execute "xl save",which function will be called first,and
then call which function? I mainly want to know which python function will
be called.

Thanks

2015-10-20 10:22 GMT+08:00 高强 :

> Hi,
>
> Thank you for your help!
>
> I'm modifying the source code of xen-4.4.1,I hope throurh printing debug
> information to track the execution order of functions and check the value
> of some variables. This is my ultimate goal.
>
> But I don't know how to do it in the python code. I tried to print debug
> information into files,but I couldn't find the file I created.
>
> Thanks
>
> 2015-10-19 17:04 GMT+08:00 Wei Liu :
>
>> On Mon, Oct 19, 2015 at 10:22:52AM +0800, 高强 wrote:
>> > Hi,alls
>> >
>> >
>> > There are some python code in xen-4.4.1,and in the python code there are
>> > lots debug information such as
>> > log.debug("***"),log.exception("***"),XendError("*")and so
>> on.
>> >
>> > I want to get this kind of information but I don't know where to get
>> them.
>> > Can someone tell me?
>> >
>> > else,I try to use the followed method to tracing program execution
>> process:
>> >
>> > import sys
>> > f=open("/home/*/python.log","a+")
>> > print >> f, "Save.py is start\n"
>> > f.close()
>> >
>> > But,I can't find the file python.log while executed end,who can tell me
>> why?
>> >
>>
>> Xen 4.4 is using xl toolstack. Xend is not used so no wonder you
>> couldn't find any logs.
>>
>> If you describe your end goal we might provide more concrete
>> suggestions.
>>
>>
>> Wei.
>>
>> > Thanks!
>>
>> > ___
>> > Xen-devel mailing list
>> > Xen-devel@lists.xen.org
>> > http://lists.xen.org/xen-devel
>>
>>
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help for python log in xen-4.4.1

[高强]

Hi,

Thank you for your help!

I'm modifying the source code of xen-4.4.1,I hope throurh printing debug
information to track the execution order of functions and check the value
of some variables. This is my ultimate goal.

But I don't know how to do it in the python code. I tried to print debug
information into files,but I couldn't find the file I created.

Thanks

2015-10-19 17:04 GMT+08:00 Wei Liu :

> On Mon, Oct 19, 2015 at 10:22:52AM +0800, 高强 wrote:
> > Hi,alls
> >
> >
> > There are some python code in xen-4.4.1,and in the python code there are
> > lots debug information such as
> > log.debug("***"),log.exception("***"),XendError("*")and so
> on.
> >
> > I want to get this kind of information but I don't know where to get
> them.
> > Can someone tell me?
> >
> > else,I try to use the followed method to tracing program execution
> process:
> >
> > import sys
> > f=open("/home/*/python.log","a+")
> > print >> f, "Save.py is start\n"
> > f.close()
> >
> > But,I can't find the file python.log while executed end,who can tell me
> why?
> >
>
> Xen 4.4 is using xl toolstack. Xend is not used so no wonder you
> couldn't find any logs.
>
> If you describe your end goal we might provide more concrete
> suggestions.
>
>
> Wei.
>
> > Thanks!
>
> > ___
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
>
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help for snapshot

[Ian Campbell]

On Tue, 2015-10-13 at 09:18 +0800, 高强 wrote:
> Hi,alls
> 
> I'm modifying the source code of xen-4.4.1. Specifically, I'm modifying
> the memory copy part of snapshot. I first put all memory pages to write
> protection, and then rewrite the write protection exception part of the
> code. After capture the exception  then save the memory page , and all
> memory page copy is complete, call the function xenaccess_teardown(xch,
> xenaccess), but there is a mistake:
> 
> libxl: error: libxl_utils.c:396:libxl_read_exactly: file/stream truncated
> reading ipc msg header from domain 3 save/restore helper stdout pipe
> libxl: error: libxl_exec.c:129:libxl_report_child_exitstatus: domain 3
> save/restore helper [-1] died due to fatal signal Segmentation fault
> (core dumped)
> Failed to save domain, resuming domain
> 
> Has anyone ever meet this kind of problem, and who can tell me the cause
> of the error.

Without knowing exactly what you have changed, probably not.

Ian.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help for snapshot

[高强]

I'm modifying the source code of snapshot,in the function xc_domain_save()
of xc_domain_save.c,through calling wrexact() to copy the memory pages to
the local dist.

I'm not sure if it is the answer you wang.

2015-10-13 13:49 GMT+08:00 Gohar Irfan :

> I'm also working on a similar thing - can you please tell me how you copy
> the memory pages from one part of the memory to another?
>
> On Tue, Oct 13, 2015 at 10:04 AM 高强  wrote:
>
>> Hi,alls
>>
>> I'm modifying the source code of xen-4.4.1. Specifically, I'm modifying
>> the memory copy part of snapshot. I first put all memory pages to write
>> protection, and then rewrite the write protection exception part of the
>> code. After capture the exception  then save the memory page , and all
>> memory page copy is complete, call the function xenaccess_teardown(xch,
>> xenaccess), but there is a mistake:
>>
>> libxl: error: libxl_utils.c:396:libxl_read_exactly: file/stream truncated
>> reading ipc msg header from domain 3 save/restore helper stdout pipe
>> libxl: error: libxl_exec.c:129:libxl_report_child_exitstatus: domain 3
>> save/restore helper [-1] died due to fatal signal Segmentation fault (core
>> dumped)
>> Failed to save domain, resuming domain
>>
>> Has anyone ever meet this kind of problem, and who can tell me the cause
>> of the error.
>>
>> Thanks
>> ___
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>>
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help for snapshot

[Gohar Irfan]

I'm also working on a similar thing - can you please tell me how you copy
the memory pages from one part of the memory to another?

On Tue, Oct 13, 2015 at 10:04 AM 高强  wrote:

> Hi,alls
>
> I'm modifying the source code of xen-4.4.1. Specifically, I'm modifying
> the memory copy part of snapshot. I first put all memory pages to write
> protection, and then rewrite the write protection exception part of the
> code. After capture the exception  then save the memory page , and all
> memory page copy is complete, call the function xenaccess_teardown(xch,
> xenaccess), but there is a mistake:
>
> libxl: error: libxl_utils.c:396:libxl_read_exactly: file/stream truncated
> reading ipc msg header from domain 3 save/restore helper stdout pipe
> libxl: error: libxl_exec.c:129:libxl_report_child_exitstatus: domain 3
> save/restore helper [-1] died due to fatal signal Segmentation fault (core
> dumped)
> Failed to save domain, resuming domain
>
> Has anyone ever meet this kind of problem, and who can tell me the cause
> of the error.
>
> Thanks
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] help: about arch/x86/xen/multicalls.c

[Konrad Rzeszutek Wilk]

On Thu, Aug 06, 2015 at 02:21:04PM +0800, 黄先生 wrote:
 hi all:
   My linux kernel verison is 2.6.32-15, and I make kernel with xen 
 compileroptions. But when my virual machine start on AWS,it show these log:
 does anyone know how to do?
 

that is soo ancient I am sorry to say we can't help you there.

Can you update to a newer kernel please.
May I no compiler  arch/x86/xen/multicalls.c? How do I can?
 
 1 multicall(s) failed: cpu 0
 Pid: 1026, comm: cut_demo_li Tainted: P   2.6.32.15-hermes-1 #10
 Call Trace:
  [c1003284] xen_mc_flush+0xad/0x169
  [c1005307] ? xen_restore_fl_direct_end+0x0/0x1
  [c10037ad] xen_leave_lazy_mmu+0x8/0xf
  [c10641c3] remap_pfn_range+0x311/0x392
  [c1060f9e] ? __do_fault+0x420/0x454
  [c1169c87] mmap_mem+0x5b/0x72
  [c1067739] mmap_region+0x1d2/0x355
  [c1067bc9] do_mmap_pgoff+0x232/0x294
  [c105def4] sys_mmap_pgoff+0x7e/0xe8
  [c1006ed8] sysenter_do_call+0x12/0x2c
 [ cut here ]
 WARNING: at arch/x86/xen/multicalls.c:182 xen_mc_flush+0x161/0x169()
 Modules linked in: ncfm_load_lic(P) ncfm_syscfg(P) ncfm_ev(P) ncfm_igb 
 ncfm_ixgbe ncfm_e1000 ncfm_e1000e ncfm_e100 ncfm_r8168 nedrv(P) ncsfkrnl(P)
 Pid: 1026, comm: cut_demo_li Tainted: P   2.6.32.15-hermes-1 #10
 Call Trace:
  [c1003338] ? xen_mc_flush+0x161/0x169
  [c102bee2] warn_slowpath_common+0x60/0x77
  [c102bf06] warn_slowpath_null+0xd/0x10
  [c1003338] xen_mc_flush+0x161/0x169
  [c1005307] ? xen_restore_fl_direct_end+0x0/0x1
  [c10037ad] xen_leave_lazy_mmu+0x8/0xf
  [c10641c3] remap_pfn_range+0x311/0x392
  [c1060f9e] ? __do_fault+0x420/0x454
  [c1169c87] mmap_mem+0x5b/0x72
  [c1067739] mmap_region+0x1d2/0x355
  [c1067bc9] do_mmap_pgoff+0x232/0x294
  [c105def4] sys_mmap_pgoff+0x7e/0xe8
  [c1006ed8] sysenter_do_call+0x12/0x2c
 ---[ end trace 96d9d060aff577eb ]---
 1 multicall(s) failed: cpu 0
 Pid: 1026, comm: cut_demo_li Tainted: PW  2.6.32.15-hermes-1 #10
 Call Trace:
  [c1003284] xen_mc_flush+0xad/0x169
  [c1005307] ? xen_restore_fl_direct_end+0x0/0x1
  [c10037ad] xen_leave_lazy_mmu+0x8/0xf
  [c10641c3] remap_pfn_range+0x311/0x392
  [c1169c87] mmap_mem+0x5b/0x72
  [c1067739] mmap_region+0x1d2/0x355
  [c1067bc9] do_mmap_pgoff+0x232/0x294
  [c105def4] sys_mmap_pgoff+0x7e/0xe8
  [c1006ed8] sysenter_do_call+0x12/0x2c
 [ cut here ]
 WARNING: at arch/x86/xen/multicalls.c:182 xen_mc_flush+0x161/0x169()
 Modules linked in: ncfm_load_lic(P) ncfm_syscfg(P) ncfm_ev(P) ncfm_igb 
 ncfm_ixgbe ncfm_e1000 ncfm_e1000e ncfm_e100 ncfm_r8168 nedrv(P) ncsfkrnl(P)
 Pid: 1026, comm: cut_demo_li Tainted: PW  2.6.32.15-hermes-1 #10
 Call Trace:
  [c1003338] ? xen_mc_flush+0x161/0x169
  [c102bee2] warn_slowpath_common+0x60/0x77
  [c102bf06] warn_slowpath_null+0xd/0x10
  [c1003338] xen_mc_flush+0x161/0x169
  [c1005307] ? xen_restore_fl_direct_end+0x0/0x1
  [c10037ad] xen_leave_lazy_mmu+0x8/0xf
  [c10641c3] remap_pfn_range+0x311/0x392
  [c1169c87] mmap_mem+0x5b/0x72
  [c1067739] mmap_region+0x1d2/0x355
  [c1067bc9] do_mmap_pgoff+0x232/0x294
  [c105def4] sys_mmap_pgoff+0x7e/0xe8
  [c1006ed8] sysenter_do_call+0x12/0x2c
 ---[ end trace 96d9d060aff577ec ]---
 BUG: unable to handle kernel paging request at d42da180
 IP: [c1003c9d] xen_set_pte+0x14/0x1b
 *pdpt = 14055027 *pde = 00173067 *pte = 8000142da061
 Oops: 0003 [#1] SMP
 last sysfs file:
 Modules linked in: ncfm_load_lic(P) ncfm_syscfg(P) ncfm_ev(P) ncfm_igb 
 ncfm_ixgbe ncfm_e1000 ncfm_e1000e ncfm_e100 ncfm_r8168 nedrv(P) ncsfkrnl(P)
 
 Pid: 1026, comm: cut_demo_li Tainted: PW  (2.6.32.15-hermes-1 #10)
 EIP: 0061:[c1003c9d] EFLAGS: 00010246 CPU: 0
 EIP is at xen_set_pte+0x14/0x1b
 EAX:  EBX: 2127a227 ECX: 000c EDX: 
 ESI: 000c EDI: d42da180 EBP: d4145ee8 ESP: d4145edc
  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
 Process cut_demo_li (pid: 1026, ti=d4144000 task=d403c080 task.ti=d4144000)
 Stack:
  b7830df4   d4145f00 c1004ae3 d42da180  
 0  d4145f84 c1062fc6 2127a227 000c d40f d4145f50 b7830df4
 0 d414a6fc c3bc9e40 2127a227 000c   0180 d41c5de0
 Call Trace:
  [c1004ae3] ? xen_set_pte_at+0xbd/0xc3
  [c1062fc6] ? handle_mm_fault+0x476/0xb15
  [c104f600] ? handle_level_irq+0xb3/0xbb
  [c1009579] ? handle_irq+0x3b/0x4a
  [c101e24a] ? do_page_fault+0x17f/0x1e3
  [c101e0cb] ? do_page_fault+0x0/0x1e3
  [c12de08e] ? error_code+0x66/0x6c
  [c101e0cb] ? do_page_fault+0x0/0x1e3
 Code: 00 c7 42 10 00 00 00 00 c7 42 14 f0 7f 00 00 83 c4 0c 5b 5e 5f 5d c3 55 
 89 e5 57 89 c7 56 89 ce 53 89 d3 e8 3d 90 01 00 89 77 04 89 1f 5b 5e 5f 5d 
 c3 55 89 e5 56 89 ce 53 89 c3 83 38 00 0f 84
 EIP: [c1003c9d] xen_set_pte+0x14/0x1b SS:ESP 0069:d4145edc
 CR2: d42da180
 ---[ end trace 96d9d060aff577ed ]---
 note: cut_demo_li[1026] exited with preempt_count 1
 

 ___
 Xen-devel mailing list
 Xen-devel@lists.xen.org
 http://lists.xen.org/xen-devel

Re: [Xen-devel] Help with reading from hvc console device on a PV guest

[Wei Liu]

Please don't top-post.

On Fri, Jul 24, 2015 at 10:05:20AM -0700, sainath grandhi wrote:
 Hi Wei,
Thanks for your reply. Sorry I did not change the comment properly.
 These are the two methods I tried separately.
 
 First method:
  I see /dev/hvc1 and /dev/hvc2 being created when I give two
 channels in the guest xl config file. So I wrote a program to open
 hvc1 and read like below.
 fd=open(/dev/hvc1, O_RDWR|O_NONBLOCK)
 
 
 Second method:
  I also notice the devices are created under
 /sys/bus/xen/devices/console-1 and console-2. And the corresponding
 devices are in /dev/hvc1 and /dev/hvc2.
  So I manually created devices similar to following udev rules like
 below for console 1.
 
 HVC_MAJOR=229
 
 NAME_PATH=device/console/1/name
 
 NAME=$(xenstore-read $NAME_PATH)
 
 mknod /dev/xenconsole/$NAME c $HVC_MAJOR 1
 
 ln -s /dev/xenconsole/$NAME /dev/xenconsole/console-1
 
 NAME in my case is xen.channel.1 Hence I tried to open and read using
 fd=open(/dev/xenconsole/xen.channel.1, O_RDWR|O_NONBLOCK)
 

Looks like that you have all udev rules in place. Good.

 
 But both the attempts failed. And they do not read the contents I sent
 from dom0. What I notice from the code review of hvc driver is that
 any data that the hvc driver gets from the backend is used ONLY for
 terminal consumption and not for file operations?
 

Not sure, I've never used that myself.

I've CC'ed Dave Scott who might have better idea about how channel is
supposed to be used.

 When I tried to create a console login on the hvc2, and do a socat on
 the socket path from dom0, I am able to send commands like ls, pwd
 etc. and receive responses fine.
 

This means the channel is created OK.

Wei.

 
 Thanks
 -Sainath
 
 On Fri, Jul 24, 2015 at 12:54 AM, Wei Liu wei.l...@citrix.com wrote:
  On Thu, Jul 23, 2015 at 10:14:57AM -0700, sainath grandhi wrote:
  Hello,
 
  Please let me know if you have some information about issue below.
 
  I created a channel device for PV guest using
 
  channel= [connection=pty, name=xen.channel.0,
  path=/var/lib/xen/ctl-socket,connection=socket, name=xen.channel.1,
  path=/var/lib/xen/ctl-socket-data]
 
 Xen toolstack uses qemu backend for additional consoles other than 
  console 0.
 
  /usr/local/lib/xen/bin/qemu-system-i386 -xen-domid 505 -chardev
  socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-505,server,nowait
  -no-shutdown -mon chardev=libxl-cmd,mode=control -chardev
  socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-505,server,nowait
  -mon chardev=libxenstat-cmd,mode=control -chardev
  pty,id=libxl-channel0 -chardev
  socket,id=libxl-channel1,path=/var/lib/xen/ctl-socket-data,server,nowait
  -nodefaults -xen-attach -name ubuntupvhchannel -vnc none -display none
  -nographic -machine xenpv -m 2048
 
 I see additional console devices in the guest under /dev/hvc1 and 
  /dev/hvc2.
 
I write to the other end of the channel i.e. from dom0,   but when I
  read the console device /dev/hvc1 or /dev/hvc2, using cat or an
  application using read(), I do not get the data out. But I could print
  out the data using printk (the string I passed from dom0) inside the
  hvc driver code where it reads from backend.
 
Looks like I am missing something in the userspace creation of
  device node or some flag for opening device file in the read system
  call of my application. Any suggestions?
 
  Application I wrote to read from console device:
 
memset(buf, 0, sizeof(buf));
 
fd=open(/dev/xenconsole/xen.channel.1, O_RDWR|O_NONBLOCK);
 
 
  Have you checked if there is such device under the said directory?
 
  You need to have udev rules in guest to create those devices for you.
 
  See the discussion thread at
 
 alpine.deb.2.02.1506241241110.18...@kaball.uk.xensource.com
 
if(fd  0)
 
printf(could not open dev hvc2\n);
 
 
  I'm confused because the code says hvc2 here but open() has another path
  in it.
 
  Wei.
 
size = read(fd, (void *)buf, sizeof(buf));
 
printf(size read %d\n,size);
 
  Thanks
 
  ___
  Xen-devel mailing list
  Xen-devel@lists.xen.org
  http://lists.xen.org/xen-devel

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Help

[Wei Liu]

On Thu, Jul 23, 2015 at 08:35:20PM +0530, Akash Talole wrote:
 Hello,
 
 Issue in tapdisk-vbd.c
 
 Why td_queue_write(parent,treq); is called in  static void
 __tapdisk_vbd_reissue_td_request(td_vbd_t *vbd,td_image_t *image,
 td_request_t treq) function as we can't write in parent vhd because it is
 read only.
 
 I have attached the code of tapdisk-vbd.c and block-vhd.c.
 

There is no need to attach the code. We all have that in our repository.
:-)

 And please tell me about how to write into child vhd after reading from
 parent vhd in block-vhd.c in case of VHD_BM_BIT_CLEAR in function
 vhd_queue_read().
 

Please describe what your end goal is so that developers with knowledge
in that area can help you better.

  http://wiki.xenproject.org/wiki/Asking_Developer_Questions

Wei.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Help with reading from hvc console device on a PV guest

[Wei Liu]

On Thu, Jul 23, 2015 at 10:14:57AM -0700, sainath grandhi wrote:
 Hello,
 
 Please let me know if you have some information about issue below.
 
 I created a channel device for PV guest using
 
 channel= [connection=pty, name=xen.channel.0,
 path=/var/lib/xen/ctl-socket,connection=socket, name=xen.channel.1,
 path=/var/lib/xen/ctl-socket-data]
 
Xen toolstack uses qemu backend for additional consoles other than console 
 0.
 
 /usr/local/lib/xen/bin/qemu-system-i386 -xen-domid 505 -chardev
 socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-505,server,nowait
 -no-shutdown -mon chardev=libxl-cmd,mode=control -chardev
 socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-505,server,nowait
 -mon chardev=libxenstat-cmd,mode=control -chardev
 pty,id=libxl-channel0 -chardev
 socket,id=libxl-channel1,path=/var/lib/xen/ctl-socket-data,server,nowait
 -nodefaults -xen-attach -name ubuntupvhchannel -vnc none -display none
 -nographic -machine xenpv -m 2048
 
I see additional console devices in the guest under /dev/hvc1 and 
 /dev/hvc2.
 
   I write to the other end of the channel i.e. from dom0,   but when I
 read the console device /dev/hvc1 or /dev/hvc2, using cat or an
 application using read(), I do not get the data out. But I could print
 out the data using printk (the string I passed from dom0) inside the
 hvc driver code where it reads from backend.
 
   Looks like I am missing something in the userspace creation of
 device node or some flag for opening device file in the read system
 call of my application. Any suggestions?
 
 Application I wrote to read from console device:
 
   memset(buf, 0, sizeof(buf));
 
   fd=open(/dev/xenconsole/xen.channel.1, O_RDWR|O_NONBLOCK);
 

Have you checked if there is such device under the said directory?

You need to have udev rules in guest to create those devices for you.

See the discussion thread at

   alpine.deb.2.02.1506241241110.18...@kaball.uk.xensource.com

   if(fd  0)
 
   printf(could not open dev hvc2\n);
 

I'm confused because the code says hvc2 here but open() has another path
in it.

Wei.

   size = read(fd, (void *)buf, sizeof(buf));
 
   printf(size read %d\n,size);
 
 Thanks
 
 ___
 Xen-devel mailing list
 Xen-devel@lists.xen.org
 http://lists.xen.org/xen-devel

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Help with reading from hvc console device on a PV guest

[sainath grandhi]

Hi Wei,
   Thanks for your reply. Sorry I did not change the comment properly.
These are the two methods I tried separately.

First method:
 I see /dev/hvc1 and /dev/hvc2 being created when I give two
channels in the guest xl config file. So I wrote a program to open
hvc1 and read like below.
fd=open(/dev/hvc1, O_RDWR|O_NONBLOCK)


Second method:
 I also notice the devices are created under
/sys/bus/xen/devices/console-1 and console-2. And the corresponding
devices are in /dev/hvc1 and /dev/hvc2.
 So I manually created devices similar to following udev rules like
below for console 1.

HVC_MAJOR=229

NAME_PATH=device/console/1/name

NAME=$(xenstore-read $NAME_PATH)

mknod /dev/xenconsole/$NAME c $HVC_MAJOR 1

ln -s /dev/xenconsole/$NAME /dev/xenconsole/console-1

NAME in my case is xen.channel.1 Hence I tried to open and read using
fd=open(/dev/xenconsole/xen.channel.1, O_RDWR|O_NONBLOCK)


But both the attempts failed. And they do not read the contents I sent
from dom0. What I notice from the code review of hvc driver is that
any data that the hvc driver gets from the backend is used ONLY for
terminal consumption and not for file operations?

When I tried to create a console login on the hvc2, and do a socat on
the socket path from dom0, I am able to send commands like ls, pwd
etc. and receive responses fine.


Thanks
-Sainath

On Fri, Jul 24, 2015 at 12:54 AM, Wei Liu wei.l...@citrix.com wrote:
 On Thu, Jul 23, 2015 at 10:14:57AM -0700, sainath grandhi wrote:
 Hello,

 Please let me know if you have some information about issue below.

 I created a channel device for PV guest using

 channel= [connection=pty, name=xen.channel.0,
 path=/var/lib/xen/ctl-socket,connection=socket, name=xen.channel.1,
 path=/var/lib/xen/ctl-socket-data]

Xen toolstack uses qemu backend for additional consoles other than 
 console 0.

 /usr/local/lib/xen/bin/qemu-system-i386 -xen-domid 505 -chardev
 socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-505,server,nowait
 -no-shutdown -mon chardev=libxl-cmd,mode=control -chardev
 socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-505,server,nowait
 -mon chardev=libxenstat-cmd,mode=control -chardev
 pty,id=libxl-channel0 -chardev
 socket,id=libxl-channel1,path=/var/lib/xen/ctl-socket-data,server,nowait
 -nodefaults -xen-attach -name ubuntupvhchannel -vnc none -display none
 -nographic -machine xenpv -m 2048

I see additional console devices in the guest under /dev/hvc1 and 
 /dev/hvc2.

   I write to the other end of the channel i.e. from dom0,   but when I
 read the console device /dev/hvc1 or /dev/hvc2, using cat or an
 application using read(), I do not get the data out. But I could print
 out the data using printk (the string I passed from dom0) inside the
 hvc driver code where it reads from backend.

   Looks like I am missing something in the userspace creation of
 device node or some flag for opening device file in the read system
 call of my application. Any suggestions?

 Application I wrote to read from console device:

   memset(buf, 0, sizeof(buf));

   fd=open(/dev/xenconsole/xen.channel.1, O_RDWR|O_NONBLOCK);


 Have you checked if there is such device under the said directory?

 You need to have udev rules in guest to create those devices for you.

 See the discussion thread at

alpine.deb.2.02.1506241241110.18...@kaball.uk.xensource.com

   if(fd  0)

   printf(could not open dev hvc2\n);


 I'm confused because the code says hvc2 here but open() has another path
 in it.

 Wei.

   size = read(fd, (void *)buf, sizeof(buf));

   printf(size read %d\n,size);

 Thanks

 ___
 Xen-devel mailing list
 Xen-devel@lists.xen.org
 http://lists.xen.org/xen-devel

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Help

[Ian Campbell]

On Thu, 2015-06-25 at 07:51 +0300, Luc Pierard de Maujouy wrote:

You should try and use a descriptive subject in the future so that
people who know about the area are inspired to read the mail.

Also in the first instance this seems more suitable for the xen-users
list.

 domU is expected to request dom0 for additionnal RAM before starting to
 use swap.

I don't know much about tmem systems, but isn't there some daemon
required in the guest to achieve that? Or perhaps that is what
CONFIG_XEN_SELFBALLOONING in the domU config is for? Is that enabled
along with things like FRONTSWAP and friends?

Some more details of exactly what you've added to which command lines
and some logs might allow someone to spot where things are not correct.

Ian.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

[Singhal, Upanshu]

Hello Don,

Thanks a lot for your help. I am able to create the PVSCSI type disks on my XEN 
VM, below method is perfect.

Appreciate your response, wish you a very happy holidays.

Thanks,
-Upanshu

-Original Message-
From: Don Slutz [mailto:dsl...@verizon.com] 
Sent: Wednesday, December 24, 2014 7:09 PM
To: Singhal, Upanshu
Cc: Don Slutz; xen-devel@lists.xen.org
Subject: Re: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

On 12/23/14 05:51, Singhal, Upanshu wrote:

 Hello Don,

 I am not trying to configure VMW PVSCSI type of device but not able to 
 do so. Though, PVSCSI is available on the distribution I am using. Any 
 inputs on how to configure PVSCSI type disk device?


device_model_args_hvm = [
-device,
pvscsi,id=scsi1,
-drive,
if=none,id=disk1,file=/home/don/qemu-img/C63-min-disk1.raw,
-device,
scsi-disk,bus=scsi1.0,scsi-id=0,drive=disk1,
]

-Don Slutz

 Thanks,

 -Upanshu

 *From:*Singhal, Upanshu
 *Sent:* Monday, December 22, 2014 12:35 PM
 *To:* 'Don Slutz'
 *Cc:* 'xen-devel@lists.xen.org'
 *Subject:* RE: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

 Hello Don,

 xen_emul_unplug=unnecessarydoes the trick, I am able to see the
 vmxnet3 driver using lspci and ethtool -I eth0. Thanks a lot for your 
 help, much appreciated.

 Performance between vmxnet3 running on XEN is about 1/3 of vmxnet3 
 running on ESXi. Similar performance number I get between vmxnet3 on 
 ESXi and vif on XEN. Do you know what could be the reason for this?

 Thanks,

 -Upanshu

 *From:*Don Slutz [mailto:dsl...@verizon.com]
 *Sent:* Saturday, December 20, 2014 3:59 AM
 *To:* Singhal, Upanshu
 *Cc:* xen-devel@lists.xen.org mailto:xen-devel@lists.xen.org
 *Subject:* Re: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

 xen_emul_unplug=unnecessary (kernel arg) may help you here.

 Also udev likes to rename your devices.

 Here is a lspci from a guest:


 [root@C63-min-tools ~]# lspci
 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 
 (rev 02)
 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton 
 II]
 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE 
 [Natoma/Triton II]
 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
 00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device 
 (rev 01)
 00:03.0 VGA compatible controller: Cirrus Logic GD 5446
 00:04.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 
 01)
 00:05.0 Ethernet controller: Intel Corporation 82540EM Gigabit 
 Ethernet Controller (rev 03)
 00:06.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 
 01)
 00:07.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 
 01)
 00:08.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 
 01)
 00:09.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 
 01)
 00:0a.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 
 01)
 00:0b.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 
 01)
 00:0c.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 
 01)
 00:0d.0 Ethernet controller: Intel Corporation 82540EM Gigabit 
 Ethernet Controller (rev 03)

 And to help:

 [root@C63-min-tools ~]# ls -l /sys/class/net/*/device lrwxrwxrwx. 1 
 root root 0 Dec 19 17:23 /sys/class/net/eth0/device -
 ../../../vif-0
 lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth1/device -
 ../../../vif-1
 lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth2/device -
 ../../../vif-2
 lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth3/device -
 ../../../vif-3
 lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth4/device -
 ../../../vif-4
 lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth5/device -
 ../../../vif-5
 lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth6/device -
 ../../../vif-6
 lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth7/device -
 ../../../vif-7
 lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth8/device -
 ../../../vif-8
 lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth9/device -
 ../../../vif-9
 lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic0/device
 - ../../../:00:04.0
 lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic1/device
 - ../../../:00:05.0
 lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic2/device
 - ../../../:00:06.0
 lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic3/device
 - ../../../:00:07.0
 lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic4/device
 - ../../../:00:08.0
 lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic5/device
 - ../../../:00:09.0
 lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic6/device
 - ../../../:00:0a.0
 lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic7/device
 - ../../../:00:0b.0
 lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic8/device
 - ../../../:00:0c.0
 lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net

Re: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

[Don Slutz]

On 12/23/14 05:51, Singhal, Upanshu wrote:


Hello Don,

I am not trying to configure VMW PVSCSI type of device but not able to 
do so. Though, PVSCSI is available on the distribution I am using. Any 
inputs on how to configure PVSCSI type disk device?




device_model_args_hvm = [
-device,
pvscsi,id=scsi1,
-drive,
if=none,id=disk1,file=/home/don/qemu-img/C63-min-disk1.raw,
-device,
scsi-disk,bus=scsi1.0,scsi-id=0,drive=disk1,
]

-Don Slutz


Thanks,

-Upanshu

*From:*Singhal, Upanshu
*Sent:* Monday, December 22, 2014 12:35 PM
*To:* 'Don Slutz'
*Cc:* 'xen-devel@lists.xen.org'
*Subject:* RE: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

Hello Don,

xen_emul_unplug=unnecessarydoes the trick, I am able to see the 
vmxnet3 driver using lspci and ethtool –I eth0. Thanks a lot for your 
help, much appreciated.


Performance between vmxnet3 running on XEN is about 1/3 of vmxnet3 
running on ESXi. Similar performance number I get between vmxnet3 on 
ESXi and vif on XEN. Do you know what could be the reason for this?


Thanks,

-Upanshu

*From:*Don Slutz [mailto:dsl...@verizon.com]
*Sent:* Saturday, December 20, 2014 3:59 AM
*To:* Singhal, Upanshu
*Cc:* xen-devel@lists.xen.org mailto:xen-devel@lists.xen.org
*Subject:* Re: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

xen_emul_unplug=unnecessary (kernel arg) may help you here.

Also udev likes to rename your devices.

Here is a lspci from a guest:


[root@C63-min-tools ~]# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 
(rev 02)

00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE 
[Natoma/Triton II]

00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device 
(rev 01)

00:03.0 VGA compatible controller: Cirrus Logic GD 5446
00:04.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:05.0 Ethernet controller: Intel Corporation 82540EM Gigabit 
Ethernet Controller (rev 03)

00:06.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:07.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:08.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:09.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0a.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0b.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0c.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0d.0 Ethernet controller: Intel Corporation 82540EM Gigabit 
Ethernet Controller (rev 03)


And to help:

[root@C63-min-tools ~]# ls -l /sys/class/net/*/device
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth0/device - 
../../../vif-0
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth1/device - 
../../../vif-1
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth2/device - 
../../../vif-2
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth3/device - 
../../../vif-3
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth4/device - 
../../../vif-4
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth5/device - 
../../../vif-5
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth6/device - 
../../../vif-6
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth7/device - 
../../../vif-7
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth8/device - 
../../../vif-8
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth9/device - 
../../../vif-9
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic0/device 
- ../../../:00:04.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic1/device 
- ../../../:00:05.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic2/device 
- ../../../:00:06.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic3/device 
- ../../../:00:07.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic4/device 
- ../../../:00:08.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic5/device 
- ../../../:00:09.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic6/device 
- ../../../:00:0a.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic7/device 
- ../../../:00:0b.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic8/device 
- ../../../:00:0c.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic9/device 
- ../../../:00:0d.0



-Don Slutz

On 12/19/14 07:04, Singhal, Upanshu wrote:

Hello,

In one of my experiment, I am building a Linux VM with Network
interface model as “vmxnet3”. I am able to create the VM
successfully, but I see that the driver loaded is “vif” and not
“vmxnet3”. I am using the following option for the network
interface: *vif = [ 'type=ioemu, mac=00:16:3e:00:00:22,
bridge=xenbr0, model=vmxnet3' ]*

**

I tried with model as e1000

Re: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

[Don Slutz]

On 12/22/14 02:04, Singhal, Upanshu wrote:


Hello Don,

xen_emul_unplug=unnecessary does the trick, I am able to see the 
vmxnet3 driver using lspci and ethtool --I eth0. Thanks a lot for your 
help, much appreciated.


Performance between vmxnet3 running on XEN is about 1/3 of vmxnet3 
running on ESXi. Similar performance number I get between vmxnet3 on 
ESXi and vif on XEN. Do you know what could be the reason for this?




Not sure, but mtu does have a factor here.

And the full network layout, other uses of network, etc. all have impacts.

Have you checked out:

http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide

I have not been investigating network bandwidth.

   -Don Slutz


Thanks,

-Upanshu

*From:*Don Slutz [mailto:dsl...@verizon.com]
*Sent:* Saturday, December 20, 2014 3:59 AM
*To:* Singhal, Upanshu
*Cc:* xen-devel@lists.xen.org
*Subject:* Re: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

xen_emul_unplug=unnecessary (kernel arg) may help you here.

Also udev likes to rename your devices.

Here is a lspci from a guest:


[root@C63-min-tools ~]# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 
(rev 02)

00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE 
[Natoma/Triton II]

00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device 
(rev 01)

00:03.0 VGA compatible controller: Cirrus Logic GD 5446
00:04.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:05.0 Ethernet controller: Intel Corporation 82540EM Gigabit 
Ethernet Controller (rev 03)

00:06.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:07.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:08.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:09.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0a.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0b.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0c.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0d.0 Ethernet controller: Intel Corporation 82540EM Gigabit 
Ethernet Controller (rev 03)


And to help:

[root@C63-min-tools ~]# ls -l /sys/class/net/*/device
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth0/device - 
../../../vif-0
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth1/device - 
../../../vif-1
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth2/device - 
../../../vif-2
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth3/device - 
../../../vif-3
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth4/device - 
../../../vif-4
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth5/device - 
../../../vif-5
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth6/device - 
../../../vif-6
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth7/device - 
../../../vif-7
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth8/device - 
../../../vif-8
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth9/device - 
../../../vif-9
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic0/device 
- ../../../:00:04.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic1/device 
- ../../../:00:05.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic2/device 
- ../../../:00:06.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic3/device 
- ../../../:00:07.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic4/device 
- ../../../:00:08.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic5/device 
- ../../../:00:09.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic6/device 
- ../../../:00:0a.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic7/device 
- ../../../:00:0b.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic8/device 
- ../../../:00:0c.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic9/device 
- ../../../:00:0d.0



   -Don Slutz

On 12/19/14 07:04, Singhal, Upanshu wrote:

Hello,

In one of my experiment, I am building a Linux VM with Network
interface model as vmxnet3. I am able to create the VM
successfully, but I see that the driver loaded is vif and not
vmxnet3. I am using the following option for the network
interface: *vif = [ 'type=ioemu, mac=00:16:3e:00:00:22,
bridge=xenbr0, model=vmxnet3' ]*

**

I tried with model as e1000 and it works fine. Lspci command also
does not show vmxnet3. Though, qemu device help shows that
vmxnet3 is supported on XEN 4.4.1 version I am using.

I tried searching on internet about the right configuration for
vmxnet3 with XEN, but not able to find right information. Can
someone please help me on how to create a VM with vmxnet3

Re: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

[Singhal, Upanshu]

Hello Don,

xen_emul_unplug=unnecessary  does the trick, I am able to see the vmxnet3 
driver using lspci and ethtool -I eth0. Thanks a lot for your help, much 
appreciated.

Performance between vmxnet3 running on XEN is about 1/3 of vmxnet3 running on 
ESXi. Similar performance number I get between vmxnet3 on ESXi and vif on XEN. 
Do you know what could be the reason for this?

Thanks,
-Upanshu

From: Don Slutz [mailto:dsl...@verizon.com]
Sent: Saturday, December 20, 2014 3:59 AM
To: Singhal, Upanshu
Cc: xen-devel@lists.xen.org
Subject: Re: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

xen_emul_unplug=unnecessary (kernel arg) may help you here.

Also udev likes to rename your devices.

Here is a lspci from a guest:


[root@C63-min-tools ~]# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
00:03.0 VGA compatible controller: Cirrus Logic GD 5446
00:04.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:05.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet 
Controller (rev 03)
00:06.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:07.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:08.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:09.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0a.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0b.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0c.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0d.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet 
Controller (rev 03)

And to help:

[root@C63-min-tools ~]# ls -l /sys/class/net/*/device
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth0/device - 
../../../vif-0
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth1/device - 
../../../vif-1
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth2/device - 
../../../vif-2
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth3/device - 
../../../vif-3
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth4/device - 
../../../vif-4
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth5/device - 
../../../vif-5
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth6/device - 
../../../vif-6
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth7/device - 
../../../vif-7
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth8/device - 
../../../vif-8
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth9/device - 
../../../vif-9
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic0/device - 
../../../:00:04.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic1/device - 
../../../:00:05.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic2/device - 
../../../:00:06.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic3/device - 
../../../:00:07.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic4/device - 
../../../:00:08.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic5/device - 
../../../:00:09.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic6/device - 
../../../:00:0a.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic7/device - 
../../../:00:0b.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic8/device - 
../../../:00:0c.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic9/device - 
../../../:00:0d.0


   -Don Slutz

On 12/19/14 07:04, Singhal, Upanshu wrote:
Hello,

In one of my experiment, I am building a Linux VM with Network interface model 
as vmxnet3. I am able to create the VM successfully, but I see that the 
driver loaded is vif and not vmxnet3. I am using the following option for 
the network interface: vif = [ 'type=ioemu, mac=00:16:3e:00:00:22, 
bridge=xenbr0, model=vmxnet3' ]

I tried with model as e1000 and it works fine. Lspci command also does not show 
vmxnet3. Though, qemu device help shows that vmxnet3 is supported on XEN 
4.4.1 version I am using.

I tried searching on internet about the right configuration for vmxnet3 with 
XEN, but not able to find right information. Can someone please help me on how 
to create a VM with vmxnet3?

This experiment I am doing to compare the difference between vmxnet3 bandwidth 
on VMWARE ESXi vs. XEN Server. My sample VM configuration file is:

# This configures an HVM rather than PV guest
builder = hvm

# Guest name
name = rhel-vmxnet3-xen-2.hvm

# 128-bit UUID for the domain as a hexadecimal number.
# Use uuidgen to generate one if required.
# The default behavior is to generate a new UUID each

Re: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

[Don Slutz]

xen_emul_unplug=unnecessary (kernel arg) may help you here.

Also udev likes to rename your devices.

Here is a lspci from a guest:


[root@C63-min-tools ~]# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE 
[Natoma/Triton II]

00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device 
(rev 01)

00:03.0 VGA compatible controller: Cirrus Logic GD 5446
00:04.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:05.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet 
Controller (rev 03)

00:06.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:07.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:08.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:09.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0a.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0b.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0c.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0d.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet 
Controller (rev 03)


And to help:

[root@C63-min-tools ~]# ls -l /sys/class/net/*/device
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth0/device - 
../../../vif-0
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth1/device - 
../../../vif-1
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth2/device - 
../../../vif-2
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth3/device - 
../../../vif-3
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth4/device - 
../../../vif-4
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth5/device - 
../../../vif-5
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth6/device - 
../../../vif-6
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth7/device - 
../../../vif-7
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth8/device - 
../../../vif-8
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth9/device - 
../../../vif-9
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic0/device - 
../../../:00:04.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic1/device - 
../../../:00:05.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic2/device - 
../../../:00:06.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic3/device - 
../../../:00:07.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic4/device - 
../../../:00:08.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic5/device - 
../../../:00:09.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic6/device - 
../../../:00:0a.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic7/device - 
../../../:00:0b.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic8/device - 
../../../:00:0c.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic9/device - 
../../../:00:0d.0



   -Don Slutz


On 12/19/14 07:04, Singhal, Upanshu wrote:


Hello,

In one of my experiment, I am building a Linux VM with Network 
interface model as vmxnet3. I am able to create the VM successfully, 
but I see that the driver loaded is vif and not vmxnet3. I am 
using the following option for the network interface: *vif = [ 
'type=ioemu, mac=00:16:3e:00:00:22, bridge=xenbr0, model=vmxnet3' ]*


**

I tried with model as e1000 and it works fine. Lspci command also does 
not show vmxnet3. Though, qemu device help shows that vmxnet3 is 
supported on XEN 4.4.1 version I am using.


I tried searching on internet about the right configuration for 
vmxnet3 with XEN, but not able to find right information. Can someone 
please help me on how to create a VM with vmxnet3?


This experiment I am doing to compare the difference between vmxnet3 
bandwidth on VMWARE ESXi vs. XEN Server. My sample VM configuration 
file is:


# This configures an HVM rather than PV guest

builder = hvm

# Guest name

*name = rhel-vmxnet3-xen-2.hvm*

# 128-bit UUID for the domain as a hexadecimal number.

# Use uuidgen to generate one if required.

# The default behavior is to generate a new UUID each time the guest 
is started.


#uuid = ----

# Enable Microsoft Hyper-V compatibile paravirtualisation /

# enlightenment interfaces. Turning this on can improve Windows guest

# performance and is therefore recommended

#viridian = 1

# Initial memory allocation (MB)

*memory = 8192*

# Maximum memory (MB)

# If this is greater than `memory' then the slack will start ballooned

# (this assumes guest kernel support for ballooning)

#maxmem = 512

# Number of VCPUS

*vcpus = 8*

# Network devices

# A list of 'vifspec' entries as 

Re: [Xen-devel] Help: VMXNET3 support with XEN 4.4.1

[Don Slutz]

P.S. here is the vif line I used:

vif = [
 model=vmxnet3,bridge=xenbr0,mac=00:0c:29:86:44:a0,
 model=e1000,bridge=xenbr0,mac=00:0c:29:86:44:aa,
 model=vmxnet3,bridge=xenbr0,mac=00:0c:29:86:44:b4,
 model=vmxnet3,bridge=xenbr0,mac=00:0c:29:86:44:be,
 model=vmxnet3,bridge=xenbr0,mac=00:0c:29:86:44:c8,
 model=vmxnet3,bridge=xenbr0,mac=00:0c:29:86:44:d2,
 model=vmxnet3,bridge=xenbr0,mac=00:0c:29:86:44:dc,
 model=vmxnet3,bridge=xenbr0,mac=00:0c:29:86:44:e6,
 model=vmxnet3,bridge=xenbr0,mac=00:0c:29:86:44:f0,
 model=e1000,bridge=xenbr0,mac=00:0c:29:86:44:fa,
]

   -Don Slutz

On 12/19/14 17:29, Don Slutz wrote:

xen_emul_unplug=unnecessary (kernel arg) may help you here.

Also udev likes to rename your devices.

Here is a lspci from a guest:


[root@C63-min-tools ~]# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 
(rev 02)

00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE 
[Natoma/Triton II]

00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device 
(rev 01)

00:03.0 VGA compatible controller: Cirrus Logic GD 5446
00:04.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:05.0 Ethernet controller: Intel Corporation 82540EM Gigabit 
Ethernet Controller (rev 03)

00:06.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:07.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:08.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:09.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0a.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0b.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0c.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
00:0d.0 Ethernet controller: Intel Corporation 82540EM Gigabit 
Ethernet Controller (rev 03)


And to help:

[root@C63-min-tools ~]# ls -l /sys/class/net/*/device
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth0/device - 
../../../vif-0
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth1/device - 
../../../vif-1
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth2/device - 
../../../vif-2
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth3/device - 
../../../vif-3
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth4/device - 
../../../vif-4
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth5/device - 
../../../vif-5
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth6/device - 
../../../vif-6
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth7/device - 
../../../vif-7
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth8/device - 
../../../vif-8
lrwxrwxrwx. 1 root root 0 Dec 19 17:23 /sys/class/net/eth9/device - 
../../../vif-9
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic0/device 
- ../../../:00:04.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic1/device 
- ../../../:00:05.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic2/device 
- ../../../:00:06.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic3/device 
- ../../../:00:07.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic4/device 
- ../../../:00:08.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic5/device 
- ../../../:00:09.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic6/device 
- ../../../:00:0a.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic7/device 
- ../../../:00:0b.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic8/device 
- ../../../:00:0c.0
lrwxrwxrwx. 1 root root 0 Dec 19 17:21 /sys/class/net/pci-nic9/device 
- ../../../:00:0d.0



   -Don Slutz


On 12/19/14 07:04, Singhal, Upanshu wrote:


Hello,

In one of my experiment, I am building a Linux VM with Network 
interface model as vmxnet3. I am able to create the VM 
successfully, but I see that the driver loaded is vif and not 
vmxnet3. I am using the following option for the network interface: 
*vif = [ 'type=ioemu, mac=00:16:3e:00:00:22, bridge=xenbr0, 
model=vmxnet3' ]*


**

I tried with model as e1000 and it works fine. Lspci command also 
does not show vmxnet3. Though, qemu device help shows that vmxnet3 
is supported on XEN 4.4.1 version I am using.


I tried searching on internet about the right configuration for 
vmxnet3 with XEN, but not able to find right information. Can someone 
please help me on how to create a VM with vmxnet3?


This experiment I am doing to compare the difference between vmxnet3 
bandwidth on VMWARE ESXi vs. XEN Server. My sample VM configuration 
file is:


# This configures an HVM rather than PV guest

builder = hvm

# Guest name

*name = rhel-vmxnet3-xen-2.hvm*

# 128-bit UUID for the domain as a hexadecimal number.

# Use uuidgen to generate one if required.

# The