Re: [CentOS-virt] LZ4 Kernel Decompression not supported

[Sarah Newman]

On 10/12/20 3:24 AM, Dr Francis Greaves wrote:

I am running XEN on CentOS Linux release 7.8.2003

I have a Debian 7, and two Gentoo DomU Guests running fine.

I am trying to get an Ubuntu 20.04 DomU Guest running but I get this error when 
trying to run the netboot kernel image



xc_dom_probe_bzimage_kernel unable to LZ4 decompress kernel



I understand that CentOS Xen does not support LZ4 decompression.



Is there a workaround for this?


Use HVM and a bootloader.
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] CentOS 8 Install as DOMU in PV Environment

[Sarah Newman]

On 9/27/20 1:23 PM, 9f9dcad3f78905b03201--- via CentOS-virt wrote:

Christoph,

I understand this is the better option if HVM is available.. Im not sure how to 
make use of this kernel in a non-HVM, PV environment. I'm a little disturbed by 
the fact that there isnt PV support in CentOS 8, I hope it comes later so at 
least PVH becomes an option.


It's existentially possible for there to be PVH support without PV support. I 
don't know about the current implementation.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] xen hvm domU and > 3 nic's... bug?

[Sarah Newman]

On 2/27/20 5:12 AM, Christoph wrote:

It seems to be a problem with kernel... I use the kernel-ml there
(5.5.6)... with kernel from centos-virt repo (4.9.*) it works without
this problem...


Have you made some pass to look for related problems on the xen-users or xen-devel mailing list or in the git commit history of mainline since 5.5.6 
was released?


If you can't find any evidence the bug has been reported or fixed already, I would encourage you to do so on the xen-devel mailing list since Linux 
4.9 won't be supported forever.


But in addition to the example configuration file, you may need to provide details about exactly what happens with 4 NICs to get a useful response. 
The output from running the xl create command, the kernel log, or the xen-hotplug log would likely contain more information.


--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] centos-virt CPU microcode updates?

[Sarah Newman]

On 5/16/19 2:12 AM, Karel Hendrych wrote:
> Hi,
> 
> is there any guide for CPU microcode updates on CentOS6, Xen 4.10, kernel 4.9 
> ?

I'm not sure off the top of my head.

You can add ucode=scan to the xen command line, add a file 
/path/to/microcode_file as the last item in your boot list, and generate that 
microcode
file using something like:

iucode_tool --write-earlyfw "${OUT_BLOB_FILE}" \
"${TEMP_DIR_NAME}/intel-ucode" \
"${TEMP_DIR_NAME}/intel-ucode-with-caveats"

Where the last two items have been extracted from the intel microcode tarball.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Xen-kernel: Update to 4.14 or 4.19?

[Sarah Newman]

On 3/7/19 10:30 AM, Akemi Yagi wrote:
> On Thu, Mar 7, 2019 at 9:42 AM George Dunlap  wrote:
>>
>> Hey all,
>>
>> We've been on 4.9 for some time now, and while it's still supported, I
>> think it's time to start thinking about upgrading, and I'd like input
>> from the community about which version to move up to.
>>
>> 4.19 has been out for almost 5 months now.  It will include PVH domU
>> support, and PVH dom0 support in what _is believed_ to be the final
>> form; so when the Virt SIG moves to a version of Xen that supports PVH
>> dom0, the kernel will already be in place with no need to upgrade.
>>
>> The other option would be to move to 4.14: Probably more stable (as
>> it's been out for over a year now), but doesn't have either PVH domU
>> or PVH dom0 support.
>>
>> I'd suggest 4.19. Any other opinions?
>>
>>  -George
> 
> You may also want to consider each version's EOL:
> 
> 4.9   Jan, 2023
> 4.14   Jan, 2020
> 4.19   Dec, 2020

Regardless of EOL date, I think it's worth trying to upgrade when Xen has 
stable PVH dom0 support.

I am pretty sure historically that there have been difficulties backporting 
some of the side channel mitigations as they can be quite invasive. That
may be another reason to upgrade sooner rather than later.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] how to update ucode with xen

[Sarah Newman]

On 09/18/2018 11:55 PM, Christoph wrote:
> 
> Hi
> 
> can someone say me how to update the µcode of the cpu with xen?
> 
> I have added the ucode=scan parameter to xen but it does not seem to work...
> 
> the µcode version of my xeon is really old :/
> 
> model name    : Intel(R) Xeon(R) CPU E3-1271 v3 @ 3.60GHz
> microcode    : 0x10
> 

We use ucode=-1 and provide it as a module for grub.
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] ANNOUNCE: centos-release-xen switching to 4.8 next week

[Sarah Newman]

On 08/02/2018 03:58 AM, T.Weyergraf wrote:
> Hi
> 
> Thanks for providing updated Packages, they are much appreciated. At work, we 
> are currently running an entire production infrastructure on Xen4CentOS,
> with quite some success.
> 
> We are looking into a refresh towards CentOS 7 along with newer Xen and Dom0 
> Kernel packages. However, even the updated packages are quite old. Xen
> 4.8 is out of active support since June and will see the end of security 
> support in less than a year.

https://wiki.xenproject.org/wiki/Xen_Project_Release_Features says December 
2019, so about a year and 4 months.

> Likewise, a newer LTS kernel (4.14) exists for
> quite some time, while the Xen4CentOS effort currently uses 4.9.

The end support date for that is much sooner: January 2019. Based on 
https://www.kernel.org/category/releases.html I'm not sure if it makes sense to
move until a new LTS is available, as I suspect that will happen before 4.9 
support ends. If you wanted to contribute an experimental 4.14 kernel it
might be accepted.

> Are there any short to mid-term plans to bump both versions to more current 
> ones (i.e.: 4.10 and 4.14)?

Anthony's original email said "that update will come with a new package 
centos-release-xen-410 to install Xen 4.10." So yes on Xen.

> As a side note, is there anything reasonable, people like me could to, to 
> support the speed-up of that process? I would consider testing to be
> important, but are there any regression test-suites, one could use? I am 
> aware, there are such tests, but I have not found something to actually try
> in our test-infrastructure.

My understanding is that would be most welcome if you have anything to 
contribute. There are occasional meetings on freenode in #centos-devel for the
virt-sig but you can also try the #centos-virt irc room.

> Finally a big shout-out and kudos to the Xen community and Xen4CentOS. Your 
> work is used and much appreciated.

+1, we haven't given back enough ourselves.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] custom Xen on custom kernel on CentOS 7

[Sarah Newman]

On 02/27/2018 07:50 AM, John Vetter wrote:
> Hi,
> I'm trying to run an arbitrary Xen version (4.7.x) on a recent kernel (say, 
> 4.13.x) on CentOS 7.
> What is the recommended way for doing this? (I am new to Xen and 
> virtualization).
> I tried the following:
> 1. installed xen4centos.
> 2. built linux kernel 4.13.x and installed it (using make install)
> 3. built xen 4.7.x and installed it (using make install).
> grub2-mkconfig and grub-bootxen.sh don't seem to be picking up the 
> combination 
> of new kernel and new xen and making an entry in the grub.cfg file.
> It did make an entry for the new Xen with the kernel installed with 
> xen4centos.
> I looked at the grub-bootxen.sh script but was unable to figure a way out. 
> How 
> do I get the installation scripts to make an entry for my new kernel and new 
> Xen?

I would suggest one of
1. Run the scripts with -x to figure out where they're quitting early
2. Make an rpm for the kernel you're building, or
3. Manually edit the grub files and run dracut. It's not hard to do what those 
scripts do by hand.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Xen 4.4 Immediate EOL

[Sarah Newman]

On 01/18/2018 09:56 AM, Kevin Stange wrote:
> Apparently I failed to do proper due diligence before making this
> recommendation.  The Xen 4.4 repo does not have vixen build because of a
> dependency upon grub2 which isn't available under CentOS 6.  Your best
> bet would be to use Vixen for PV domains, so if you think that's
> something you want to do, we need some volunteers to help with packaging
> and testing.  Otherwise, use HVM domains or upgrade to a newer version
> of Xen.  Sorry for this error on my part.
>

We have a SPEC file available for grub2: https://github.com/prgmrcom/grub2 you 
will need epel installed.

___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Centos 6 2.6.32-696.18.7.el6.x86_64 does not boot in Xen PV mode

[Sarah Newman]

On 01/05/2018 05:26 PM, Shaun Reitan wrote:
> I can confirm the issue with 2.6.32-696.18.7.el6.x86_64, but mine looks 
> alittle different... Maybe because i'm using pvgrub.
> 
> = Init TPM Front 
> Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront 
> initialization! error = ENOENT
> Tpmfront:Info Shutting down tpmfront
> close blk: backend=/local/domain/0/backend/vbd/14/51712 node=device/vbd/51712
> close blk: backend=/local/domain/0/backend/vbd/14/51728 node=device/vbd/51728
> [root@devhost1]#
> 
> -- 
> Shaun Reitan
> NDCHost.com

You need to add earlyprintk=xen.

___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

[CentOS-virt] Centos 6 2.6.32-696.18.7.el6.x86_64 does not boot in Xen PV mode

[Sarah Newman]

Problems start before any of the kaiser code executes, though it could still be 
related to CONFIG_KAISER since that has effects beyond kaiser.c.

---
(early) Initializing cgroup subsys cpuset
(early) Initializing cgroup subsys cpu
(early) Linux version 2.6.32-696.18.7.el6.x86_64 
(mockbu...@c1bl.rdu2.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) 
(GCC) ) #1 SMP Thu
Jan 4 17:31:22 UTC 2018
(early) Command line: ro console=hvc0 rootflags=barrier=0 crashkernel=auto 
SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=us earlyprintk=xen
(early) KERNEL supported cpus:
(early)   Intel GenuineIntel
(early)   AMD AuthenticAMD
(early)   Centaur CentaurHauls
(early) 1 multicall(s) failed: cpu 0
(early) Pid: 0, comm: swapper Not tainted 2.6.32-696.18.7.el6.x86_64 #1
(early) Call Trace:
(early)  [] ? xen_mc_flush+0x1c3/0x250
(early)  [] ? xen_extend_mmu_update+0xde/0x1b0
(early)  [] ? xen_set_pmd_hyper+0x9d/0xc0
(early)  [] ? early_ioremap_init+0x98/0x133
(early)  [] ? setup_arch+0x40/0xca6
(early)  [] ? vprintk_default+0xe/0x10
(early)  [] ? printk+0x4f/0x52
(early)  [] ? start_kernel+0xdc/0x43b
(early)  [] ? reserve_early+0x30/0x39
(early)  [] ? x86_64_start_reservations+0x125/0x129
(early)  [] ? xen_start_kernel+0x4fe/0x505
(early) [ cut here ]
(early) WARNING: at arch/x86/xen/multicalls.c:182 xen_mc_flush+0x21f/0x250() 
(Not tainted)
(early) Modules linked in:
(early) Pid: 0, comm: swapper Not tainted 2.6.32-696.18.7.el6.x86_64 #1
(early) Call Trace:
(early)  [] ? warn_slowpath_common+0x91/0xe0
(early)  [] ? __raw_callee_save_xen_restore_fl+0x11/0x1e
(early)  [] ? warn_slowpath_null+0x1a/0x20
(early)  [] ? xen_mc_flush+0x21f/0x250
(early)  [] ? xen_extend_mmu_update+0xde/0x1b0
(early)  [] ? xen_extend_mmu_update+0xde/0x1b0
(early)  [] ? xen_set_pmd_hyper+0x9d/0xc0
(early)  [] ? early_ioremap_init+0x98/0x133
(early)  [] ? setup_arch+0x40/0xca6
(early)  [] ? vprintk_default+0xe/0x10
(early)  [] ? printk+0x4f/0x52
(early)  [] ? start_kernel+0xdc/0x43b
(early)  [] ? reserve_early+0x30/0x39
(early)  [] ? x86_64_start_reservations+0x125/0x129
(early)  [] ? xen_start_kernel+0x4fe/0x505
(early) ---[ end trace a7919e7f17c0a725 ]---
(early) ACPI in unprivileged domain disabled
(early) released 0 pages of unused memory
(early) BIOS-provided physical RAM map:
(early)  Xen:  - 000a (usable)
(early)  Xen: 000a - 0010 (reserved)
(early)  Xen: 0010 - 2000 (usable)
(early) bootconsole [xenboot0] enabled
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] CentOS-virt - Kernel Side-Channel Attacks

[Sarah Newman]

On 01/04/2018 10:49 AM, Akemi Yagi wrote:
> On Thu, Jan 4, 2018 at 9:51 AM,  wrote:
> 
>> Please patch the CentOS-virt Kernel to fix the
>> Kernel Side-Channel Attacks vulnerabilities.
>>
>> The latest CentOS-virt kernel was released in November, as seen below.
>>
>> kernel-4.9.63-29.el7.x86_64.rpm 2017-11-21 13:30
>>
>> https://access.redhat.com/security/vulnerabilities/speculativeexecution
>> http://mirror.centos.org/centos/7/virt/x86_64/xen/
>>
> 
> ​As far as I can see, the patches for ​
> KAISER (Kernel Address
> ​ ​Isolation to have Side-channels Efficiently Removed) will appear in
> kernel 4.9.75. Looks like it will be released soon upstream (kernel.org).
> 

To my best knowledge KAISER doesn't matter for Xen Dom0's given they run in PV 
mode, and KAISER isn't enabled for PV guests.
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Live migration haswell, broadwell

[Sarah Newman]

On 11/14/2017 03:13 AM, T.Weyergraf wrote:
> Hi
> 
> I wonder, if live migration (back and forth) is possible on mixed Haswell 
> (Xeon V3) and Broadwell (Xeon V4) installations. The only notable difference
> between the two is apparently a working TSX implementation on V4, which got 
> disabled on V3 due to bugs. The rest (VMCS-shadowing, posted interrupts)
> should not apply to our environment, as we do not run nested-vmx nor 
> device-passthrough on our Xen servers
> 
> Now, I found no sane way to disable TSX on a given system but I cannot rule 
> out, that some (Linux-)software, such as Postgres will use them
> eventually. Also, I have a hard time trying to assess, if TSX can be disabled 
> on V4 to enable seamless migration.
> 
> Any hint would be greately appreciated.

Have you looked at CPUID masking? I haven't tried this but someone wrote a 
handy calculator for it. https://github.com/twizted/xen_maskcalc
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

[CentOS-virt] Crash in network stack under Xen

[Sarah Newman]

Hi,

We had a potentially network related crash on a dom0 with Linux 4.9.39 / Xen 
4.8 and as of today I can't find any fixes in stable/linux-4.9.y,
xen/staging-4.8, or CPU microcode updates that look like a smoking gun. I can't 
rule out that it's Xen related. The backtraces are:

 [ cut here ]
 WARNING: CPU: 0 PID: 0 at net/ipv4/af_inet.c:1473 inet_gro_complete+0xbb/0xd0
 Call Trace:
  dump_stack+0x63/0x8e
   __warn+0xd1/0xf0
   warn_slowpath_null+0x1d/0x20
   inet_gro_complete+0xbb/0xd0
   napi_gro_complete+0x73/0xa0
   napi_gro_flush+0x5f/0x80
   napi_complete_done+0x6a/0xb0
   igb_poll+0x38d/0x720 [igb]
   ? igb_msix_ring+0x2e/0x40 [igb]
   ? __handle_irq_event_percpu+0x4b/0x1a0
   net_rx_action+0x158/0x360
   __do_softirq+0xd1/0x283
   irq_exit+0xe9/0x100
   xen_evtchn_do_upcall+0x35/0x50
   xen_do_hypervisor_callback+0x1e/0x40
   ? xen_hypercall_sched_op+0xa/0x20
   ? xen_hypercall_sched_op+0xa/0x20
   ? xen_safe_halt+0x10/0x20
   ? default_idle+0x1e/0xd0
   ? arch_cpu_idle+0xf/0x20
   ? default_idle_call+0x2c/0x40
   ? cpu_startup_entry+0x1ac/0x240
   ? rest_init+0x77/0x80
   ? start_kernel+0x4a7/0x4b4
   ? set_init_arg+0x55/0x55
   ? x86_64_start_reservations+0x24/0x26
   ? xen_start_kernel+0x555/0x561

 general protection fault:  [#1] SMP
 Call Trace:
  ? napi_gro_complete+0x5e/0xa0
   skb_release_all+0x24/0x30
   kfree_skb+0x32/0x90
   napi_gro_complete+0x5e/0xa0
   napi_gro_flush+0x5f/0x80
   napi_complete_done+0x6a/0xb0
   igb_poll+0x38d/0x720 [igb]
   ? igb_msix_ring+0x2e/0x40 [igb]
   ? __handle_irq_event_percpu+0x4b/0x1a0
   net_rx_action+0x158/0x360
   __do_softirq+0xd1/0x283
   irq_exit+0xe9/0x100
   xen_evtchn_do_upcall+0x35/0x50
   xen_do_hypervisor_callback+0x1e/0x40
   ? xen_hypercall_sched_op+0xa/0x20
   ? xen_hypercall_sched_op+0xa/0x20
   ? xen_safe_halt+0x10/0x20
   ? default_idle+0x1e/0xd0
   ? arch_cpu_idle+0xf/0x20
   ? default_idle_call+0x2c/0x40
   ? cpu_startup_entry+0x1ac/0x240
   ? rest_init+0x77/0x80
   ? start_kernel+0x4a7/0x4b4
   ? set_init_arg+0x55/0x55
   ? x86_64_start_reservations+0x24/0x26
   ? xen_start_kernel+0x555/0x561
 RIP   skb_release_data+0x73/0xf0
 Kernel panic - not syncing: Fatal exception in interrupt
 Kernel Offset: disabled
(XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds.

If anyone has had a similar backtrace or knows of a potential fix please 
respond.

This server has ECC and there were no ECC or other errors in the BIOS event 
log, nor were there any indications of any problems in the serial console
log leading up to the warning.

This particular server had an uptime of about a month and a half, and so far 
we've had this error exactly once across all our servers since switching
to 4.9.39 in August, so I don't think it's going to be easy to reproduce.

---

It looks to me like in the first backtrace, this check from inet_gro_complete 
failed:

ops = rcu_dereference(inet_offloads[proto]);

Which I'm guessing means the packet didn't have a valid layer 4 protocol 
definition, or we don't have that protocol enabled. Then when attempting to
handle that failure there was a GPF, I believe by accessing invalid data in 
shinfo->frag_list . "skb_release_data+0x73" is in __read_once_size, which
I think is generated by "kfree_skb: if (likely(atomic_read(>users) == 1))" 
.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Stability issues since moving to 4.6 - Kernel paging request bug + VM left in null state

[Sarah Newman]

On 11/07/2017 04:57 PM, Sarah Newman wrote:
> On 11/07/2017 03:12 PM, Nathan March wrote:
>> Since moving from 4.4 to 4.6, I've been seeing an increasing number of
>> stability issues on our hypervisors. I'm not clear if there's a singular
>> root cause here, or if I'm dealing with multiple bugs.
>>
>>  
>>
>> One of the more common ones I've seen, is a VM on shutdown will remain in
>> the null state and a kernel bug is thrown:
>>
>>  
>>
>> xen001 log # xl list
>>
>> NameID   Mem VCPUs  State
>> Time(s)
>>
>> Domain-0 0  614424 r-
>> 6639.7
>>
>> (null)   3 0 1 --pscd
>> 36.3
>>
>>  
>>
>> [89920.839074] BUG: unable to handle kernel paging request at
>> 88020ee9a000
>>
> 
> 
>> This is on xen 4.6.6-4.el6 with 4.9.58-29.el6.x86_64. I see these issues
>> across a wide number of systems with from both Dell and Supermicro, although
>> we run the same Intel x540 10gb nic's in each system with the same netapp
>> nfs backend storage.
> 
> We don't use NFS and have not seen the exact same issue.

Additionally we aren't using xen 4.6 any more, we're using 4.8, but we didn't 
see issues like that when we were using xen 4.6. We're also still on
4.9.39. You might try an older kernel or a newer version of xen in addition to 
looking for nfs specific issues.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Stability issues since moving to 4.6 - Kernel paging request bug + VM left in null state

[Sarah Newman]

On 11/07/2017 03:12 PM, Nathan March wrote:
> Since moving from 4.4 to 4.6, I've been seeing an increasing number of
> stability issues on our hypervisors. I'm not clear if there's a singular
> root cause here, or if I'm dealing with multiple bugs.
> 
>  
> 
> One of the more common ones I've seen, is a VM on shutdown will remain in
> the null state and a kernel bug is thrown:
> 
>  
> 
> xen001 log # xl list
> 
> NameID   Mem VCPUs  State
> Time(s)
> 
> Domain-0 0  614424 r-
> 6639.7
> 
> (null)   3 0 1 --pscd
> 36.3
> 
>  
> 
> [89920.839074] BUG: unable to handle kernel paging request at
> 88020ee9a000
> 


> This is on xen 4.6.6-4.el6 with 4.9.58-29.el6.x86_64. I see these issues
> across a wide number of systems with from both Dell and Supermicro, although
> we run the same Intel x540 10gb nic's in each system with the same netapp
> nfs backend storage.

We don't use NFS and have not seen the exact same issue.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Xen 4.6.3-15 packages, including XSAs 216-219, 221-225 on their way through the build system

[Sarah Newman]

On 06/20/2017 05:06 AM, George Dunlap wrote:
> Xen 4.6.3-15 packages for CentOS 6 and CentOS 7 are on their way
> through the build system.  They should show up in centos-virt-testing
> in a few hours, and in the main mirrors tomorrow morning (God
> willing).
> 
> These contain several critical updates; users are encouraged to update
> as soon as possible.
> 

There are also Linux kernel changes in XSA-216. They apply cleanly to 4.9.31 
and presumably also 4.9.25.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] xen4centos kernel version / debuginfo

[Sarah Newman]

On 06/12/2017 03:25 PM, Kevin Stange wrote:
> On 06/12/2017 05:17 PM, Sarah Newman wrote:
>> Is there any problem moving to 4.9.31? This contains upstream commits 
>> f2e767bb5d6ee0d9 for mpt3sas and
>> 69861e0a52f87333 for dom0 memory mappings.
> 
> 4.9.31-27 is tagged for testing presently along with updated firmware.
> I have it running on one hypervisor on Xen-44 and it has been stable for
> 3 days so far.  We could probably use some more testing before tagging
> it stable.
> 
> I have been having some dom0 page allocation failures on 4.9.25... does
> 69861e0a52f87333 address that issue?  I was having trouble determining
> if it was related.

I don't know if it addresses it. I have also been having similar problems.

> 
> https://buildlogs.centos.org/centos/6/virt/x86_64/xen/
> 
>> I am also trying to build a debuginfo package but it's not correct. 
>> Specifically I'm able to get vmlinux but the .debug files for modules are
>> stripped, which isn't right. Does anyone know how to fix this?
> 
> How are you building?  CentOS builds with mock, but I'm not super
> familiar with the environment.  rpmbuild -b doesn't seem to always work
> that well.

I am building with mock.

There are currently no debuginfo packages, this is a capability I'm trying to 
add.

>> I'm also not sure how you're supposed to build the kernel-firmware package 
>> with the current spec file definition.
> 
> I'm also not sure about how the firmware builds work but here's the Koji
> builds from the last update:
> 
> http://cbs.centos.org/koji/buildinfo?buildID=17333
> http://cbs.centos.org/koji/buildinfo?buildID=17332
> 

Not from the kernel spec file, apparently, even though there's tons of 
references in there. I wonder why this changed.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

[CentOS-virt] Crash in CentOS 7 kernel-3.10.0-514.16.1.el7.x86_64 in Xen PV mode

[Sarah Newman]

I experienced a bug that is likely the same as 
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1350373 . Commit
b7dd0e350e0bd4c0fddcc9b8958342700b00b168 , which is supposed to fix it, doesn't 
appear in this kernel and doesn't apply cleanly either.
Is there any point in trying to backport the patch?

The backtrace is as follows:

[   32.304666] [ cut here ]
[   32.304679] kernel BUG at arch/x86/kernel/paravirt.c:252!
[   32.304683] invalid opcode:  [#1] SMP
[   32.304687] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 
ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat
ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 
nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle 
iptable_security iptable_raw ebtable_filter ebtables ip6table_filter
ip6_tables iptable_filter intel_powerclamp coretemp pcspkr ip_tables ext4 
mbcache jbd2 xen_netfront xen_blkfront crc32c_intel
[   32.304734] CPU: 0 PID: 3901 Comm: dracut Not tainted 
3.10.0-514.16.1.el7.x86_64 #1
[   32.304739] task: 880002598000 ti: 88001b728000 task.ti: 
88001b728000
[   32.304743] RIP: e030:[]  [] 
enter_lazy.part.0+0x4/0x6
[   32.304755] RSP: e02b:88001f803aa8  EFLAGS: 00010002
[   32.304758] RAX: 0001 RBX: 88001eacd640 RCX: 3000
[   32.304761] RDX: 88000640 RSI: c90c8000 RDI: 0001
[   32.304765] RBP: 88001f803aa8 R08: 88001f803b78 R09: 813d50f9
[   32.304771] R10: 88001e801e00 R11: ea093dc0 R12: c90c9000
[   32.304777] R13: c90c8000 R14:  R15: 88001d150340
[   32.304787] FS:  7f64425b0740() GS:88001f80() 
knlGS:
[   32.304796] CS:  e033 DS:  ES:  CR0: 8005003b
[   32.304801] CR2: 006de2c8 CR3: 1b405000 CR4: 2660
[   32.304807] DR0:  DR1:  DR2: 
[   32.304813] DR3:  DR6: 0ff0 DR7: 0400
[   32.304818] Stack:
[   32.304823]  88001f803ab8 81061857 88001f803b60 
811b1fe2
[   32.304833]  c90c8fff c90c9000 819bac90 
c90c8fff
[   32.304843]  c90c9000 88001eacb000 810206b0 

[   32.304854] Call Trace:
[   32.304858]  
[   32.304861]  [] paravirt_enter_lazy_mmu+0x27/0x30
[   32.304879]  [] apply_to_page_range+0x282/0x460
[   32.304888]  [] ? map_pte_fn+0x60/0x60
[   32.304894]  [] arch_gnttab_map_status+0x3b/0x70
[   32.304904]  [] gnttab_map_frames_v2+0xd6/0x150
[   32.304910]  [] gnttab_map+0xa1/0x140
[   32.304917]  [] get_free_entries+0x100/0x2e0
[   32.304923]  [] gnttab_alloc_grant_references+0x15/0x30
[   32.304933]  [] do_blkif_request+0x6bf/0x8a0 [xen_blkfront]
[   32.304945]  [] ? __freed_request+0x92/0xa0
[   32.304951]  [] __blk_run_queue+0x33/0x40
[   32.304957]  [] blk_start_queue+0x29/0x40
[   32.304964]  [] kick_pending_request_queues+0x21/0x30 
[xen_blkfront]
[   32.304975]  [] blkif_interrupt+0x76e/0x820 [xen_blkfront]
[   32.304986]  [] ? kmem_cache_free+0x1bb/0x1f0
[   32.304995]  [] handle_irq_event_percpu+0x3e/0x1e0
[   32.305003]  [] handle_irq_event+0x3d/0x60
[   32.305004]  [] handle_edge_irq+0x77/0x130
[   32.305004]  [] __xen_evtchn_do_upcall+0x227/0x350
[   32.305004]  [] xen_evtchn_do_upcall+0x33/0x50
[   32.305004]  [] xen_do_hypervisor_callback+0x1e/0x30
[   32.305004]  
[   32.305004]  [] ? copy_pte_range+0x2b6/0x5a0
[   32.305004]  [] ? copy_pte_range+0x286/0x5a0
[   32.305004]  [] ? copy_page_range+0x312/0x490
[   32.305004]  [] ? dup_mm+0x362/0x680
[   32.305004]  [] ? copy_process+0x144e/0x1960
[   32.305004]  [] ? do_fork+0x91/0x2c0
[   32.305004]  [] ? SyS_clone+0x16/0x20
[   32.305004]  [] ? stub_clone+0x69/0x90
[   32.305004]  [] ? system_call_fastpath+0x16/0x1b
[   32.305004] Code: 20 e9 2f ff ff ff 44 89 fa 44 89 ee 48 c7 c7 10 45 8c 81 
31 c0 e8 9d 14 00 00 58 5a 5b 41 5c 41 5d 41 5e 41 5f 5d c3 55 48 89 e5
<0f> 0b 66 66 66 66 90 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48
[   32.305004] RIP  [] enter_lazy.part.0+0x4/0x6
[   32.305004]  RSP 
[   32.305004] ---[ end trace 49f67f0d85e1ac69 ]---
[   32.305004] Kernel panic - not syncing: Fatal exception in interrupt

Thanks, Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] 4.9 kernel fails to boot because it didn't have the mpt3sas module

[Sarah Newman]

On 05/17/2017 01:30 PM, Jerry wrote:

> For some reason each of the 5, 6 & 7 releases have issues with the mpt2sas
> driver (there isn't enough swiotlb allocated by default for the driver to
> work properly, and the units for that parameter changed between 5 and 6).
> It's like this driver has bad luck or something.

To my best knowledge they shouldn't use swiotlb if you give the dom0 4GiB+
of RAM. I posted the exact same problem before and that was the recommendation
I got. It works.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] 4.9 kernel fails to boot because it didn't have the mpt3sas module

[Sarah Newman]

On 05/16/2017 09:04 PM, Jerry wrote:

> 
> Turns out dracut was unable to mount the root file system.  So I went back
> into the 3.10 kernel again to see if the mpt2sas or mpt3sas driver was in
> its initramfs file... and it wasn't:
> 
> 
> $ sudo lsinitrd -k 4.9.25-27.el7.x86_64 | grep mpt
> -rw-r--r--   1 root root0 May 16 12:39 etc/fstab.empty
> -rw-r--r--   1 root root   22 Nov  5  2016
> usr/lib/kbd/unimaps/empty.uni
> 
> For comparison:
> 
> $ sudo lsinitrd -k 3.10.0-514.16.1.el7.x86_64 | grep mpt
> -rw-r--r--   1 root root0 May 16 04:37 etc/fstab.empty
> -rw-r--r--   1 root root   22 Nov  5  2016
> usr/lib/kbd/unimaps/empty.uni
> drwxr-xr-x   2 root root0 May 16 04:37
> usr/lib/modules/3.10.0-514.16.1.el7.x86_64/kernel/drivers/scsi/mpt3sas
> -rw-r--r--   1 root root   379021 Apr 12 08:51
> usr/lib/modules/3.10.0-514.16.1.el7.x86_64/kernel/drivers/scsi/mpt3sas/mpt2sas.ko
> 
> 
> So I added it:
> 
> 
> $ sudo dracut --force --add-drivers mpt3sas --kver=4.9.25-27.el7.x86_64
> $ sudo lsinitrd -k 4.9.25-27.el7.x86_64 | grep mpt
> Arguments: --force --add-drivers 'mpt3sas' --kver '4.9.25-27.el7.x86_64'
> -rw-r--r--   1 root root0 May 16 12:57 etc/fstab.empty
> -rw-r--r--   1 root root   22 Nov  5  2016
> usr/lib/kbd/unimaps/empty.uni
> drwxr-xr-x   2 root root0 May 16 12:57
> usr/lib/modules/4.9.25-27.el7.x86_64/kernel/drivers/scsi/mpt3sas
> -rwxr--r--   1 root root   374152 May 16 12:57
> usr/lib/modules/4.9.25-27.el7.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko
> 
> 
> After this I was able to get the 4.9 kernel to boot and Xen is now working.

You should be able to make this change permanently from /etc/dracut.conf with
the add_drivers+= line, or build a generic initrd with hostonly="no" .

I don't know why it wasn't originally added.

--Sarah

___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

[CentOS-virt] centos-virt IRC meetings

[Sarah Newman]

Hi,

We were hoping to attend an IRC meeting this morning but it looks like that 
didn't happen. Has this been moved to once a month or was this a special week?

Thanks, Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] NIC Stability Problems Under Xen 4.4 / CentOS 6 / Linux 3.18

[Sarah Newman]

On 03/16/2017 04:22 PM, Kevin Stange wrote:

>> I still can't rest assured the NIC issue is fixed, but no 4.4 or 4.9
>> server has yet had a NIC issue, with some being up almost a full month.
>> It looks promising! (I'm knocking on all the wood everywhere, though.)
> 
> I'm ready to call this conclusive.  The problems I was having across the
> board seemed to be caused by something seriously broken in 3.18.  Most
> of my servers are now on 4.9.13 or newer and everything has been working
> very well.
> 
> I'm not going to post any further updates unless something breaks.
> Thanks to everyone that provided tips and suggestions along the way.
> 

Do you mind sharing what hardware have you been running the 4.9 kernel on other 
than "Supermicro X9DRT, Dual Xeon E5-2650, 2x I350, 2x 82571EB" and
"Supermicro X9DRD-iF/LF, Dual Xeon E5-2630, 2x I350, 2x 82575EB" if any? Are 
you using any SATA/SAS controllers?

Thanks, Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Xen C6 kernel 4.9.13 and testing 4.9.15 only reboots.

[Sarah Newman]

On 03/24/2017 11:35 AM, PJ Welsh wrote:
> As a follow up I was able to test fresh install on Dell R710 and a Dell
> R620 with success on CentOS 7.3 without issue on the new kernel.  My new
> plan will be to just move this C6 to one of the C7 I just created.

That sounds like a compiler problem, since I think the C6 and C7 kernels are 
built from the same source.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] xen 4.7 or 4.8

[Sarah Newman]

On 02/18/2017 06:23 AM, Johnny Hughes wrote:
> On 02/18/2017 02:07 AM, Christoph wrote:
>> 
>> Hi
>> 
>> are there somewhere pkgs with xen 4.7 or 4.8 for centos7?
>> 
> 
> The SIG has agreed to maintain every other release (even numbered release) of 
> Xen.  So there will be a 4.8 set of packages at some point.
> 
> There are currently no released 4.8 packages.
> 
> I do see that we have some testing packages here:
> 
> http://cbs.centos.org/repos/virt7-xen-48-testing/x86_64/os/Packages/
> 
> George Dunlap can talk to the planned schedule for 4.8 and if those packages 
> are really ready for testing.

Any updates or asks on this?

Thanks, Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Selinux Problem

[Sarah Newman]

On 01/30/2017 03:22 AM, George Dunlap wrote:

> 
> I think that comment may be a little old.  I do try to support SELinux
> -- the smoke tests I use before pushing changes have it enabled by
> default, and they use both qemu-xen and blktap.
> 
> But it's difficult to help debug problems when you haven't even said
> what problem(s) you're having. :-)
> 
> Please be sure to include the output of `dmesg`, `xl dmesg`, your
> xl.cfg, and /var/log/audit/audit.log.
> 
> Thanks,
>  -George

George,

I appreciate you try to keep SELinux working and thank you. If SELinux isn't 
appropriate for an environment, disabling it is easy. But if it is needed
for whatever reason, adding support is hard.

Looking through our ansible role, it turns out that for xenconsoled to be able 
to work with oxenstored I had to make a policy change. I hesitate to
publish that policy as-is because I used audit2allow without taking enough time 
to tune it and the policy is probably too permissive.

But running xenconsoled with oxenstored on CentOS 6 should allow you to 
duplicate. If you don't have time to duplicate, I should be able to do that
and get you the original audit.log messages.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Selinux Problem

[Sarah Newman]

On 01/26/2017 08:45 AM, Sarah Newman wrote:
> On 01/26/2017 08:06 AM, Günther J. Niederwimmer wrote:
>> Hello,
>>
>> CentOS 7.(3) Xen 4.4,
>>
>> Can I find any Doc for selinux with XEN, I found many Problems with selinux 
>> on 
>> Dom0 ?
>>
>> Or have I to disable selinux when I install XEN.
>>
>> Thank's for a answer.
>>
> 
> What problems and what version of CentOS?
> 
> We leave selinux enabled.

Sorry I'm blind, should have had more coffee.

I would like to know what problems you're having specifically. We aren't on 
CentOS 7 yet unfortunately.

___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Selinux Problem

[Sarah Newman]

On 01/26/2017 08:06 AM, Günther J. Niederwimmer wrote:
> Hello,
> 
> CentOS 7.(3) Xen 4.4,
> 
> Can I find any Doc for selinux with XEN, I found many Problems with selinux 
> on 
> Dom0 ?
> 
> Or have I to disable selinux when I install XEN.
> 
> Thank's for a answer.
> 

What problems and what version of CentOS?

We leave selinux enabled.

___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

[CentOS-virt] Xen4CentOS kernel-debuginfo

[Sarah Newman]

I think this may have been asked before, but what would it take to get 
debuginfo packages built for the Xen4CentOS kernels? If it's just a patch for
kernel.spec file that nobody has gotten around to, what is the best starting 
place for making that patch?

Thanks, Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] how to enable auto-start on newer versions of Xen ?

[Sarah Newman]

On 08/23/2016 01:10 PM, Sarah Newman wrote:
> On 08/23/2016 12:59 PM, Craig Thompson wrote:
> 
>> I was thinking that in iterations past, it was possible just to run the 
>> hypervisor, put a symlink in /etc/xen/auto, and VMs would be auto-started 
>> upon reboot.  Maybe I'm wrong, and maybe those older boxes had libvirt 
>> installed on them, too.
>>
> 
> We haven't had issues on xen 4.6. We don't use libvirt.
> 
> Are you sure xendomains is enabled? Maybe in the past it was enabled by 
> default but isn't now.
> 
> If it is enabled, have you tried running the xendomains init script manually 
> with bash -x to see where it's going wrong? You need to add 'set -x' to
> /usr/lib64/xen/bin/xendomains as well for full output.
> 

Sorry, I missed the CentOS 7 bit.

Looking at https://github.com/CentOS-virt7/xen/blob/xen-46/SPECS/xen.spec#L613 
it looks like xendomains is not enabled by default. It looks like that
happened in 
https://github.com/CentOS-virt7/xen/commit/01a31207ef50ec8c664fa0807ac85075dca3aa53.
 No clue as to why.

--Sarah

___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] how to enable auto-start on newer versions of Xen ?

[Sarah Newman]

On 08/23/2016 12:59 PM, Craig Thompson wrote:

> I was thinking that in iterations past, it was possible just to run the 
> hypervisor, put a symlink in /etc/xen/auto, and VMs would be auto-started 
> upon reboot.  Maybe I'm wrong, and maybe those older boxes had libvirt 
> installed on them, too.
> 

We haven't had issues on xen 4.6. We don't use libvirt.

Are you sure xendomains is enabled? Maybe in the past it was enabled by default 
but isn't now.

If it is enabled, have you tried running the xendomains init script manually 
with bash -x to see where it's going wrong? You need to add 'set -x' to
/usr/lib64/xen/bin/xendomains as well for full output.

___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Soft lockups with Xen4CentOS 3.18.25-18.el6.x86_64

[Sarah Newman]

On 03/10/2016 12:05 AM, Sarah Newman wrote:
> On 03/09/2016 08:15 PM, Sarah Newman wrote:
>> I've been running 3.18.25-18.el6.x86_64 + our build of xen 4.4.3-9 on one 
>> host for the last couple of weeks and have gotten several soft lockups
>> within the last 24 hours. I am posting here first in case anyone else has 
>> experienced the same issue.
>>
> 
> Here is mpstat from around the time of the issue:
> 
> 0:08:56 PM  CPU%usr   %nice%sys %iowait%irq   %soft  %steal  
> %guest   %idle
> 10:09:10 PM  all0.000.00   66.670.000.00   33.330.00
> 0.000.00
> 10:09:11 PM  all2.170.005.43   32.610.00   58.701.09
> 0.000.00
> 10:09:12 PM  all0.000.001.150.000.00   85.060.00
> 0.00   13.79
> 10:09:13 PM  all0.000.001.080.000.00   83.870.00
> 0.00   15.05
> 10:09:14 PM  all0.000.001.100.000.00   83.520.00
> 0.00   15.38
> 10:09:15 PM  all1.090.001.090.000.00   85.870.00
> 0.00   11.96
> 10:09:51 PM  all0.000.001.090.000.00   84.781.09
> 0.00   13.04
> Message from syslogd at Mar  9 22:09:51 ...
>  kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
> 10:10:02 PM  all0.000.00   33.33   50.000.00   16.670.00
> 0.000.00
> 10:10:03 PM  all3.160.00   10.538.420.002.111.05
> 0.00   74.74
> 10:10:04 PM  all0.000.003.23   38.710.001.081.08
> 0.00   55.91
> 10:10:05 PM  all0.000.004.30   11.830.003.231.08
> 0.00   79.57
> 
> Typical load:
> 
> 10:22:15 PM  CPU%usr   %nice%sys %iowait%irq   %soft  %steal  
> %guest   %idle
> 10:22:16 PM  all0.000.001.020.000.001.020.00
> 0.00   97.96
> 10:22:17 PM  all0.000.000.000.000.000.001.04
> 0.00   98.96
> 10:22:18 PM  all0.000.000.000.000.001.011.01
> 0.00   97.98
> 10:22:19 PM  all0.000.001.010.000.001.010.00
> 0.00   97.98
> 10:22:20 PM  all0.000.000.000.000.000.001.02
> 0.00   98.98
> 10:22:21 PM  all0.000.001.020.000.001.020.00
> 0.00   97.96
> 10:22:22 PM  all0.000.000.000.000.001.011.01
> 0.00   97.98
> 
> 
> I reverted to an older kernel since the older kernel had run for a couple of 
> months without issues.


This did not fix it. I isolated the issue to a vif rate limit of 100Mb/s being 
applied to one of the guests and am now able to reproduce on a
different machine.

I will look into whether this has been fixed already; if so I will submit a 
pull request for the Xen4CentOS kernel and if not I will take it up with
the xen-devel list.

___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] Soft lockups with Xen4CentOS 3.18.25-18.el6.x86_64

[Sarah Newman]

On 03/09/2016 08:15 PM, Sarah Newman wrote:
> I've been running 3.18.25-18.el6.x86_64 + our build of xen 4.4.3-9 on one 
> host for the last couple of weeks and have gotten several soft lockups
> within the last 24 hours. I am posting here first in case anyone else has 
> experienced the same issue.
> 

Here is mpstat from around the time of the issue:

0:08:56 PM  CPU%usr   %nice%sys %iowait%irq   %soft  %steal  %guest 
  %idle
10:09:10 PM  all0.000.00   66.670.000.00   33.330.00
0.000.00
10:09:11 PM  all2.170.005.43   32.610.00   58.701.09
0.000.00
10:09:12 PM  all0.000.001.150.000.00   85.060.00
0.00   13.79
10:09:13 PM  all0.000.001.080.000.00   83.870.00
0.00   15.05
10:09:14 PM  all0.000.001.100.000.00   83.520.00
0.00   15.38
10:09:15 PM  all1.090.001.090.000.00   85.870.00
0.00   11.96
10:09:51 PM  all0.000.001.090.000.00   84.781.09
0.00   13.04
Message from syslogd at Mar  9 22:09:51 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
10:10:02 PM  all0.000.00   33.33   50.000.00   16.670.00
0.000.00
10:10:03 PM  all3.160.00   10.538.420.002.111.05
0.00   74.74
10:10:04 PM  all0.000.003.23   38.710.001.081.08
0.00   55.91
10:10:05 PM  all0.000.004.30   11.830.003.231.08
0.00   79.57

Typical load:

10:22:15 PM  CPU%usr   %nice%sys %iowait%irq   %soft  %steal  
%guest   %idle
10:22:16 PM  all0.000.001.020.000.001.020.00
0.00   97.96
10:22:17 PM  all0.000.000.000.000.000.001.04
0.00   98.96
10:22:18 PM  all0.000.000.000.000.001.011.01
0.00   97.98
10:22:19 PM  all0.000.001.010.000.001.010.00
0.00   97.98
10:22:20 PM  all0.000.000.000.000.000.001.02
0.00   98.98
10:22:21 PM  all0.000.001.020.000.001.020.00
0.00   97.96
10:22:22 PM  all0.000.000.000.000.001.011.01
0.00   97.98


I reverted to an older kernel since the older kernel had run for a couple of 
months without issues.
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

[CentOS-virt] Soft lockups with Xen4CentOS 3.18.25-18.el6.x86_64

[Sarah Newman]

I've been running 3.18.25-18.el6.x86_64 + our build of xen 4.4.3-9 on one host 
for the last couple of weeks and have gotten several soft lockups
within the last 24 hours. I am posting here first in case anyone else has 
experienced the same issue.

Here is the first instance:

sched: RT throttling activated
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
Modules linked in: ebt_arp xen_pciback xen_gntalloc ebt_ip ebtable_filter 
ebtables ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4
iptable_filter ip_tables xt_physdev br_netfilter bridge stp llc ip6t_REJECT 
nf_reject_ipv6 nf_c
onntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables 
ipv6 xen_acpi_processor blktap xen_netback xen_blkback xen_gntdev
xen_evtchn xenfs xen_privcmd joydev sg 8250_fintek serio_raw gpio_ich iTCO_wdt 
iTCO_vendor_su
pport coretemp intel_powerclamp crct10dif_pclmul crc32_pclmul crc32c_intel 
pcspkr i2c_i801 lpc_ich igb ptp pps_core hwmon ioatdma dca i7core_edac
edac_core shpchp ext3 jbd mbcache raid10 raid1 sd_mod mptsas mptscsih mptbase 
scsi_transpor
t_sas aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 ahci 
libahci mgag200 ttm drm_kms_helper dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.18.25-18.el6.x86_64 #1
Hardware name: Supermicro X8DTN+-F/X8DTN+-F, BIOS 080016  08/03/2011
task: 81c1b4c0 ti: 81c0 task.ti: 81c0
RIP: e030:[]  [] 
xenvif_tx_build_gops+0xa5/0x890 [xen_netback]
RSP: e02b:88013f403ca8  EFLAGS: 0206
RAX: 003c RBX: c90012c28000 RCX: c90012c280d0
RDX: 00071ea1 RSI: 0040 RDI: c90012c28000
RBP: 88013f403e38 R08:  R09: 0001
R10:  R11:  R12: 00071e65
R13: 88013f403e50 R14: 003c R15: 0032
FS:  7fe942ac7980() GS:88013f40() knlGS:
CS:  e033 DS:  ES:  CR0: 8005003b
CR2: 807f6800 CR3: bcd4c000 CR4: 2660
Stack:
 88013f403d30 880006de6800 8800adfee1c0 88013f403e54
 00015786e75a 0040a0351878 c90012c2def0 00015786e73c
 c90012c280d0 c90012c2def0 a03516c0 8800bf29e000
Call Trace:
 
 [] ? br_handle_frame_finish+0x3f0/0x3f0 [bridge]
 [] ? __netif_receive_skb_core+0x1ee/0x640
 [] ? __netif_receive_skb+0x27/0x70
 [] ? netif_receive_skb_internal+0x2d/0x90
 [] ? igb_alloc_rx_buffers+0x63/0xe0 [igb]
 [] xenvif_tx_action+0x4d/0xa0 [xen_netback]
 [] xenvif_poll+0x35/0x68 [xen_netback]
 [] net_rx_action+0x112/0x2a0
 [] __do_softirq+0xfc/0x2b0
 [] irq_exit+0xbd/0xd0
 [] xen_evtchn_do_upcall+0x3c/0x50
 [] xen_do_hypervisor_callback+0x1e/0x40
 
 [] ? xen_hypercall_sched_op+0xa/0x20
 [] ? xen_hypercall_sched_op+0xa/0x20
 [] ? xen_safe_halt+0x10/0x20
 [] ? default_idle+0x24/0xc0
 [] ? arch_cpu_idle+0xf/0x20
 [] ? cpuidle_idle_call+0xd6/0x1d0
 [] ? __atomic_notifier_call_chain+0x12/0x20
 [] ? cpu_idle_loop+0x135/0x1e0
 [] ? cpu_startup_entry+0x1b/0x70
 [] ? cpu_startup_entry+0x60/0x70
 [] ? rest_init+0x77/0x80
 [] ? start_kernel+0x441/0x448
 [] ? set_init_arg+0x5d/0x5d
 [] ? x86_64_start_reservations+0x2a/0x2c
 [] ? xen_start_kernel+0x5ef/0x5f1
Code: 00 0f 87 06 07 00 00 44 8b b3 b8 00 00 00 44 03 b3 c0 00 00 00 45 29 e6 
41 39 c6 44 0f 47 f0 45 85 f6 0f 84 8f 00 00 00 0f ae e8 <8b> 83 c0 00
00 00 83 e8 01 44 21 e0 48 8d 04 40 48 c1 e0 02 48

Of the remaining lockups, here is the common backtrace with the exception that 
there have been two instances of RIP being in net_rx_action:

 [] net_rx_action+0x112/0x2a0
 [] __do_softirq+0xfc/0x2b0
 [] irq_exit+0xbd/0xd0
 [] xen_evtchn_do_upcall+0x3c/0x50
 [] xen_do_hypervisor_callback+0x1e/0x40
 
 [] ? xen_hypercall_sched_op+0xa/0x20
 [] ? xen_hypercall_sched_op+0xa/0x20
 [] ? xen_safe_halt+0x10/0x20
 [] ? default_idle+0x24/0xc0
 [] ? arch_cpu_idle+0xf/0x20
 [] ? cpuidle_idle_call+0xd6/0x1d0
 [] ? __atomic_notifier_call_chain+0x12/0x20
 [] ? cpu_idle_loop+0x135/0x1e0
 [] ? cpu_startup_entry+0x1b/0x70
 [] ? cpu_startup_entry+0x60/0x70
 [] ? rest_init+0x77/0x80
 [] ? start_kernel+0x441/0x448
 [] ? set_init_arg+0x5d/0x5d
 [] ? x86_64_start_reservations+0x2a/0x2c
 [] ? xen_start_kernel+0x5ef/0x5f1

I can post more complete backtraces if that information would be useful to 
someone.
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] XSAs 170 and 154, repository layouts, and centos-release-xen 8-1

[Sarah Newman]

On 02/17/2016 04:30 AM, George Dunlap wrote:
> I have the following packages going through the CBS:
> * A CentOS 7 xen-4.6.1-2, with XSAs 170 and 154
> * A CentOS 6 xen-4.6.1-2, with XSAs 170 and 154
> * A CentOS 6 xen-4.4.3-11, with XSAs 170
> 
> All these should show up in mirrors hopefully sometime later today.
> As usual, please report any problems here.

Domains using the distribution provided pvgrub won't boot after upgrade.

Old location of pvgrub:

/usr/lib/xen/boot/pv-grub-x86_32.gz
/usr/lib/xen/boot/pv-grub-x86_64.gz

New location of pvgrub:

/usr/lib64/xen/boot/pv-grub-x86_32.gz
/usr/lib64/xen/boot/pv-grub-x86_64.gz




___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] CentOS 6 Virt SIG Xen 4.6 packages available in centos-virt-xen-testing

[Sarah Newman]

On 01/21/2016 04:32 AM, George Dunlap wrote:

> I'm a developer, not a server admin, so I can't gauge how important
> this issue is.  Before making such a change, I'd like to hear opinions
> from other people in the community about how important (or not) it is
> to avoid breaking xm, given the ample warning (>1 year) users have
> had.
> 
> On the other hand, explicitly moving to a "xen${VER}" (both for C6 and
> C7) would make it simpler for people to step up and maintain older
> versions in parallel if anybody wanted to do so.

My inclination is towards a naming scheme like xen46, xen48, etc + a meta 
package that always depends on the latest. It should be more obvious when
there's a major upgrade, and those who can't afford a major upgrade can 
uninstall the meta package.

For the record, we have no particular desire for xen 4.4 but haven't done 
enough testing to say xen 4.6 is good yet.
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

[CentOS-virt] Xen4CentOS and XSA-142

[Sarah Newman]

It looks like no XSA-142 patch, which is "libxl fails to honour readonly flag 
on disks with qemu-xen" has been applied to Xen4CentOS. I assume this
was on purpose?

If not, I can have someone try adding the original patch from 
http://xenbits.xen.org/xsa/advisory-142.html and some variant of the commit from
ef6cb76026628e26e3d1ae53c50ccde1c3c78b1b 
http://xen.1045712.n5.nabble.com/xen-master-libxl-relax-readonly-check-introduced-by-XSA-142-fix-td5729704.html
 .

Thanks, Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] 3.18.21-16 upgrade, kernel panic, unable to mount root fs

[Sarah Newman]

On 11/01/2015 12:07 PM, Karel Hendrych wrote:
> Hi, just a heads-up: 3.18.21 didn't boot up on HP ML310e Gen8 v2, SATA drives 
> in AHCI mode, / on software raid 1, no LVM. It ended up in kernel panic
> with unable to mount root fs. Attached. No difference in grub kernel/xen 
> settings.
> 
> I didn't investigate the things deeper so far. 4.4.3-3/3.18.17 is booting 
> fine.
> 
> Any similar experience?

Check the module line for the initramfs is present in grub.conf. It's gone 
missing in all our recent upgrades, but I wasn't sure if it was an upstream
problem or our own.

___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

[CentOS-virt] Changelog for xen-4.4.3-3

[Sarah Newman]

Just a heads up, the changelog for the xen package says

 * Thu Oct 22 2015 George Dunlap  - 4.4.3-2.el6.centos
 - Import XSAs 149-153


The source RPM also has the patch for XSA 148, as shown in the git log.

Regards, Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
https://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] How to kickstart a guest with Centos 6.6 Xen 4.4

[Sarah Newman]

On 07/30/2015 05:29 AM, Elliot Fox wrote:

 The TL;DR is: Everything I've found to kickstart a new vanilla rhel/centos 
 guest points to specifying a kernel  and initrd- for RHEL/Centos 5. But
 where is the xen initrd for Centos 6? The Xen4Quickstart instructions are 
 awesome, but after install you are left with a kernel and an initramfs  no
 initrd for a centos 6 guest in xen.
 
 So is there a better way to kickstart a fresh/vanilla VM post Xen4Quickstart 
 instructions? Shall I build an initrd from the initramfs (and how) or is
 there another way?


If you want do a kickstart install using the PXE initrd.img, use the matching 
PXE vmlinuz. Generally speaking, don't mix and match kernel and
initramfs versions and assume it will work even if they're from the same major 
release.

Where to find the kickstart file needs to be specified in the normal way
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/s1-kickstart2-startinginstall.html
___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] /var/lib/xenstored Xen4CentOS

[Sarah Newman]

On 06/04/2015 08:32 AM, George Dunlap wrote:

 for i in {1..80} ; do xl create null.cfg name=\t$i\
 kernel=\/boot/vmlinuz-3.18.12-11.el7.x86_64\ memory=\40\
 on_crash=\preserve\ ; done
 
 And then used the shell snippet from the bug report to generate load:
 
 while true; do xl list  /dev/null; usleep 5000;done
 
 With xenstored normally the CPU load was about 30%.  Mounted under
 tmpfs, it was slightly smaller, around 20%, but swung around wildly.
 With oxenstored it was about 15%.
 
 This is with xfs as the filesystem.
 
 That doesn't seem big enough to warrant a lot of effort: if you can
 reproduce the 75% numbers from the bug report I'll pursue it further,
 otherwise I'll probably just leave it.

I'm actually more concerned about I/O wait than CPU usage. When shutting down 
or starting a lot of guests the wait time could be significantly increased.

I only started to look into this because I was doing load testing and it was 
taking much longer to shut everything down then it seemed like it should.
Running multiple versions of xl in parallel (especially xl destroy) can make 
many, many backups of tdb such that there's hundreds of MB of backup copies.

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt

[CentOS-virt] /var/lib/xenstored Xen4CentOS

[Sarah Newman]

By default, /var/lib/xenstored is mounted tmpfs in centos 5 but this doesn't 
appear to be true for Xen4CentOS. This can cause performance issues as
mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=446089

I'm not sure if this should be part of xencommons, but if not, adding a tmpfs 
mount should probably be documented on the quick start wiki page.

Thanks, Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] CentOS 6 VM image for paravirtualizaton on CentOS Xen server

[Sarah Newman]

On 03/15/2015 03:25 PM, Nico Kadel-Garcia wrote:
 So I'm right back to my effectively unanswered original questions. So
 please: I asked a very specific pair of questions, and they remain
 unanswered. CentOS 5 Xen server (hypervisor, or Dom0, whatever we want
 to call it this week): Does CentOS 6 work, paravirtualized, on such a
 server? And given my deployment issues, does anyone have a base OS
 image I can get a copy of?

I'm not sure why you need tftp to do a net install assuming you control the 
guest configuration.

How about

kernel = file from 
https://mirrors.kernel.org/centos/6/os/i386/images/pxeboot/vmlinuz
ramdisk = file from 
https://mirrors.kernel.org/centos/6/os/i386/images/pxeboot/initrd.img
extra = console=hvc0
memory = 512 or larger

--Sarah
___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt

Re: [CentOS-virt] CentOS 6 VM image for paravirtualizaton on CentOS Xen server

[Sarah Newman]

On 03/15/2015 07:39 PM, Nico Kadel-Garcia wrote:
 On Sun, Mar 15, 2015 at 8:26 PM, Sarah Newman s...@prgmr.com wrote:
 On 03/15/2015 03:25 PM, Nico Kadel-Garcia wrote:
 So I'm right back to my effectively unanswered original questions. So
 please: I asked a very specific pair of questions, and they remain
 unanswered. CentOS 5 Xen server (hypervisor, or Dom0, whatever we want
 to call it this week): Does CentOS 6 work, paravirtualized, on such a
 server? And given my deployment issues, does anyone have a base OS
 image I can get a copy of?

 I'm not sure why you need tftp to do a net install assuming you control the 
 guest configuration.

 How about

 kernel = file from 
 https://mirrors.kernel.org/centos/6/os/i386/images/pxeboot/vmlinuz
 ramdisk = file from 
 https://mirrors.kernel.org/centos/6/os/i386/images/pxeboot/initrd.img
 extra = console=hvc0
 memory = 512 or larger

 --Sarah
 
 I'll be happy to try this. Thank you for the pointer. Are you
 confident that 'console=hvc0' is the right installation time message
 for this? And will I be able to access a CD or DVD image for actual OS
 installation with these options?

It is net install IE you download packages over the network. No CD/DVD.

console=hvc0 is the xen part, otherwise look at the standard installation 
instructions for centos 6.

 
 I'd really prefer to work from 'virsh' than from hand-writing xl
 configuration files.When I last did this sort of thing, I worked from
 a PXE environment that I controlled and could reserve DHCP settings
 based on MAC addresses, and tune PXE to boot from disk by default but
 allow users to select a clean re-install of the operating system they
 wanted.

virt-install with --location maybe? Never tried it but it looks like what you 
want.

http://linux.die.net/man/1/virt-install



___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt