I've been watching for a while for a solution, or even conformation, of
a potential kernel freeze problem on the Raspberry PI 2/3 platform with
the Xenomai patches. Having not seen anything that appears related, I'm
posting this description of what I've experienced.
I'm interesting in advice on how to solve this problem as well as how to
best debug the issue. I've enabled 'Detect Hard and Soft Lockups' and
'Panic on Soft Lockups', but have not been successful in recovering from
the hung state (other than by a power cycle) nor retrieving information
related to the cause.
The details are below.
Thanks,
john
() Problem:
Under heavy load, a kernel compiled with the Xenomai patches
('CONFIG_XENOMAI=y' in .config) will freeze without warnings or error
messages. The same kernel version compiled with xenomai patches off ('#
CONFIG_XENOMAI is not set' in .config) does not experience such problems.
() Hardware:
Tested on several Raspberry 2 and 3 boards with various sdcards and
power supplies. Using the default system boot configuration and system
clock (i.e. not overclocked).
() Software:
kernel 4.1.21 from https://github.com/raspberrypi/linux/tree/rpi-4.1.y
Mathieu Rondonneau RPI2 patch from this list, Apr 16 2016
ipipe-core-4.1.18-arm-8.patch
xenomai-3-3.0.3
note: I have tested, with similar results, many other kernels and as far
back as kernel 3.18 & xenomai-3-3.0.1
() Conditions to reproduce kernel freeze:
Boot xenomai patched kernel
Launch a kernel build process and unzip process. I commonly do this
through 'screen' as follows:
(I'm leaving out obvious setup steps such as 'mkdir <foobar>', 'make
defconfig', etc)
[]# screen
[]# cd linux-rpi-3.18.y/
[]# for i in `seq 1 5`; do make -j4 bzImage; make clean; done
[]# <ctl-a> c (new screen & shell)
[]# cd tmp
[]# for i in `seq 1 30`; do unzip ../linux-rpi-3.18.y.zip; rm -rf
linux-rpi-3.18.y; done
[]# <ctl-a> c (new screen & shell)
[]# top
With the 'CONFIG_XENOMAI=y' kernel, this will typically create a kernel
freeze in 3 to 20 minutes.
Note that this load does not seem to be enough to use swap space and I
have not launched any Xemonai threads.
() Comments
I have tested this level of load for 18+ hours using the same kernel
compiled with '# CONFIG_XENOMAI is not set' in the configuration file.
There is no failure in this case.
The basic .config file differences are shown below. I can make available
the actual configuration files if needed.
< '# CONFIG_XENOMAI is not set' .config file
> '# CONFIG_XENOMAI=y' .config file
< CONFIG_PREEMPT_RCU=y
> CONFIG_TREE_RCU=y
> CONFIG_OPTPROBES=y
< CONFIG_HAVE_CONTEXT_TRACKING=y
< CONFIG_UNINLINE_SPIN_UNLOCK=y
> CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
> CONFIG_INLINE_READ_UNLOCK=y
> CONFIG_INLINE_READ_UNLOCK_IRQ=y
> CONFIG_INLINE_WRITE_UNLOCK=y
> CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
> CONFIG_XENOMAI=y
> CONFIG_XENO_OPT_STATS=y
> CONFIG_XENO_OPT_RR_QUANTUM=1000
> CONFIG_XENO_OPT_AUTOTUNE=y
> CONFIG_XENO_OPT_TIMER_LIST=y
> CONFIG_XENO_OPT_HOSTRT=y
> CONFIG_XENO_OPT_VFILE=y
> CONFIG_XENO_OPT_REGISTRY_NRSLOTS=512
> CONFIG_XENO_OPT_SYS_HEAPSZ=1024
> CONFIG_XENO_OPT_PRIVATE_HEAPSZ=128
> CONFIG_XENO_OPT_SHARED_HEAPSZ=128
> CONFIG_XENO_OPT_NRTIMERS=128
> CONFIG_XENO_OPT_TIMING_SCHEDLAT=0
> CONFIG_XENO_OPT_TIMING_KSCHEDLAT=0
> CONFIG_XENO_OPT_TIMING_IRQLAT=0
> CONFIG_XENO_OPT_RTDM_COMPAT_DEVNODE=y
> CONFIG_XENO_DRIVERS_AUTOTUNE=y
> CONFIG_XENO_DRIVERS_TIMERBENCH=y
> CONFIG_XENO_DRIVERS_SWITCHTEST=y
> CONFIG_XENO_ARCH_UNLOCKED_SWITCH=y
> CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH=y
> CONFIG_IPIPE_WANT_ACTIVE_MM=y
> CONFIG_XENO_ARCH_WANT_TIP=y
> CONFIG_XENO_ARCH_FPU=y
> CONFIG_XENO_ARCH_OUTOFLINE_XNLOCK=y
> CONFIG_XENO_VERSION_MAJOR=3
> CONFIG_XENO_VERSION_MINOR=0
> CONFIG_XENO_REVISION_LEVEL=3
> CONFIG_XENO_VERSION_STRING="3.0.3"
> CONFIG_XENO_VERSION_NAME="Groovy Cosmic Halo"
> CONFIG_IPIPE_ARM_KUSER_TSC=y
> CONFIG_NEED_KUSER_HELPERS=y
> CONFIG_IPIPE=y
> CONFIG_IPIPE_CORE=y
> CONFIG_IPIPE_CORE_APIREV=2
> CONFIG_IPIPE_WANT_APIREV_2=y
> CONFIG_IPIPE_TARGET_APIREV=2
> CONFIG_IPIPE_HAVE_HOSTRT=y
< CONFIG_PREEMPT=y
< CONFIG_PREEMPT_COUNT=y
> CONFIG_PREEMPT_VOLUNTARY=y
< CONFIG_SECCOMP=y
> CONFIG_IPV6_SUBTREES=y
> CONFIG_BT_HCIUART=m
> CONFIG_BT_HCIUART_H4=y
> CONFIG_BT_HCIUART_3WIRE=y
> CONFIG_BT_HCIUART_BCM=y
> CONFIG_QCA7000=m
> CONFIG_GPIO_BCM_VIRT=y
> CONFIG_DRM=m
> CONFIG_DRM_KMS_HELPER=m
> CONFIG_DRM_KMS_FB_HELPER=y
> CONFIG_DRM_GEM_CMA_HELPER=y
> CONFIG_DRM_KMS_CMA_HELPER=y
> CONFIG_DRM_VC4=m
> CONFIG_HDMI=y
> CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
> CONFIG_SND_BCM2708_SOC_BOOMBERRY_DAC=m
> CONFIG_SND_BCM2708_SOC_BOOMBERRY_DIGI=m
> CONFIG_HID_BETOP_FF=m
> CONFIG_MCP3422=m
< CONFIG_DEBUG_PREEMPT=y
< CONFIG_DYNAMIC_FTRACE=y
< CONFIG_FTRACE_MCOUNT_RECORD=y
_______________________________________________
Xenomai mailing list
[email protected]
https://xenomai.org/mailman/listinfo/xenomai
