Hi, Meng: Julien is correct-- a coworker and I are working on support for Tegra SoCs, and we've made pretty good progress; there's work yet to be done, but we have dom0 and guests booting on the Jetson TK1, Jetson TX1, and the Google Pixel C. We hope to get a patch set out soon-- unfortunately, our employer has to take some time to verify that everything's okay to be open-sourced, so I can't send out our work-in-progress just yet. We'll have an RFC patchset out soon, I hope!
There are two main hardware differences that cause Tegra SoCs to have trouble with Xen: - The primary interrupt controller for those systems isn't a single GIC, as Xen expects. Instead, there's an NVIDIA Legacy Interrupt Controller (LIC, or ICTLR) that gates all peripheral interrupts before passing them to a standard GICv2. This interrupt controller has to be programmed to ensure Xen can receive interrupts from the hardware (e.g. serial), programmed to ensure that interrupts for pass-through devices are correctly unmasked, and virtualized so dom0 can program the "sections" related to interrupts not being routed to Xen or to a domain for hardware passthrough. - The serial controller on the Tegra SoCs doesn't behave in the same was as most NS16550-compatibles; it actually adheres to the NS16550 spec a little more rigidly than most compatible controllers. A coworker (Chris Patterson, cc'd) figured out what was going on; from what I understand, most 16550s generate the "transmit ready" interrupt once, when the device first can accept new FIFO entries. Both the original 16550 and the Tegra implementation generate the "transmit ready" interrupt /continuously/ when there's space available in the FIFO, slewing the CPU with a stream of constant interrupts. What you're seeing is likely a symptom of the first difference. In your logs, you see messages that indicate Xen is having trouble correctly routing IRQ that are parented by the legacy interrupt controller: > irq 0 not connected to primary controller.Connected to > /interrupt-controller@60004000 The issue here is that Xen is currently explicitly opting not to route legacy-interrupt-controller interrupts, as they don't belong to the primary GIC. As a result, these interrupts never make it to dom0. The logic that needs to be tweaked is here: http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/arm/domain_build.c;h=00dc07af637b67153d33408f34331700dff84f93;hb=HEAD#l1137 We re-write this logic in our forthcoming patch-set to be more general. As an interim workaround, you might opt to rewrite that logic so LIC interrupts (which have an interrupt-parent compatible with "tegra124-ictlr", in your case) can be routed by Xen, as well. Off the top of my head, a workaround might look like: /* * Don't map IRQ that have no physical meaning * ie: IRQ whose controller is not the GIC */ - if ( rirq.controller != dt_interrupt_controller ) +if ( (rirq.controller != dt_interrupt_controller) && (!dt_device_is_compatible(rirq.controller, "tegra124-ictlr") ) Of course, that's off-the-cuff code I haven't tried, but hopefully it should help to get you started. --Kyle On Mon, May 16, 2016 at 3:39 PM, Meng Xu <xumengpa...@gmail.com> wrote: > Hi Julien, > > On Mon, May 16, 2016 at 1:33 PM, Julien Grall <julien.gr...@arm.com> wrote: >> (CC Kyle who is also working on Tegra?) >> >> Hi Meng, >> >> Many people are working on Nvidia platform with different issues :/. I have >> CCed another person which IIRC is also working on it. > > Sure. It's good to know others are also interested in this platform. > It will be more useful to fix it... :-) > >> >> On 16/05/16 17:33, Meng Xu wrote: >>> >>> On Mon, May 16, 2016 at 7:33 AM, Julien Grall <julien.gr...@arm.com> >>> wrote: >>>> >>>> >>>> On 15/05/16 20:35, Meng Xu wrote: >>>>> >>>>> >>>>> I'm trying to run Xen on NVIDIA Jetson TK1 board. (Right now, Xen does >>>>> not support the Jetson board officially. But I'm thinking it may be >>>>> very interesting and useful to see it happens, since it has GPU inside >>>>> which is quite popular in automotive.) >>>>> >>>>> Now I encountered some problem to boot dom0 in Xen environment. I want >>>>> to debug the issues and maybe fix the issues, but I'm not so sure how >>>>> I should debug the issue more efficiently. I really appreciate it if >>>>> you advise me a little bit about the method of how to fix the issue. >>>>> :-) >>>>> >>>>> ---Below is the details---- >>>>> >>>>> I noticed the Dushyant from IBM also tried to run Xen on the Jetson >>>>> board. (http://www.gossamer-threads.com/lists/xen/devel/422519). I >>>>> used the same Linux kernel (Jan Kiszka's development tree - >>>>> http://git.kiszka.org/linux.git/, branch queues/assorted) and Ian's >>>>> Xen repo. with the hack for Jetson board. I can see the dom0 kernel >>>>> can boot to some extend and then "stall/spin" before the dom0 kernel >>>>> fully boot up. >>>>> >>>>> In order to figure out the possible issue, I boot the exact same Linux >>>>> kernel in native environment on one CPU and collected the boot log >>>>> information in [1]. I also boot the same Linux kernel as dom0 in Xen >>>>> environment and collected the boot log information in [2]. >>>>> >>>>> In Xen environment, dom0 hangs after the following message >>>>> [ 10.541010] NET: Registered protocol family 10 >>>>> 6mip6: Mobile IPv6 >>>>> [ 10.542510] mi >>>>> >>>>> In native environment, the kernel has the following log after >>>>> initializing NET. >>>>> [ 2.934693] NET: Registered protocol family 10 >>>>> [ 2.940611] mip6: Mobile IPv6 >>>>> [ 2.943645] sit: IPv6 over IPv4 tunneling driver >>>>> [ 2.951303] NET: Registered protocol family 17 >>>>> [ 2.955800] NET: Registered protocol family 15 >>>>> [ 2.960257] can: controller area network core (rev 20120528 abi 9) >>>>> [ 2.966617] NET: Registered protocol family 29 >>>>> [ 2.971098] can: raw protocol (rev 20120528) >>>>> [ 2.975384] can: broadcast manager protocol (rev 20120528 t) >>>>> [ 2.981088] can: netlink gateway (rev 20130117) max_hops=1 >>>>> [ 2.986734] Bluetooth: RFCOMM socket layer initialized >>>>> [ 2.991979] Bluetooth: RFCOMM ver 1.11 >>>>> [ 2.995757] Bluetooth: BNEP (Ethernet Emulation) ver 1.3 >>>>> [ 3.001109] Bluetooth: BNEP socket layer initialized >>>>> [ 3.006089] Bluetooth: HIDP (Human Interface Emulation) ver 1.2 >>>>> [ 3.012052] Bluetooth: HIDP socket layer initialized >>>>> [ 3.017894] Registering SWP/SWPB emulation handler >>>>> [ 3.029675] tegra-pcie 1003000.pcie-controller: 2x1, 1x1 >>>>> configuration >>>>> [ 3.036586] +3.3V_SYS: supplied by +VDD_MUX >>>>> [ 3.040857] +3.3V_LP0: supplied by +3.3V_SYS >>>>> [ 3.045509] +1.35V_LP0(sd2): supplied by +5V_SYS >>>>> [ 3.050201] +1.05V_RUN_AVDD: supplied by +1.35V_LP0(sd2) >>>>> [ 3.057131] tegra-pcie 1003000.pcie-controller: probing port 0, using >>>>> 2 lanes >>>>> [ 3.066479] tegra-pcie 1003000.pcie-controller: Slot present pin >>>>> change, signature: 00000008 >>>>> >>>>> I'm suspecting that my dom0 kernel hangs when it tries to initialize >>>>> "can: controller area network core ". However, from Dushyant's post at >>>>> http://www.gossamer-threads.com/lists/xen/devel/422519, it seems >>>>> Dushyant's dom0 kernel hangs when it tries to initialize pci_bus. (The >>>>> linux config I used may be different form Dushyant's. That could be >>>>> the reason.) >>>>> >>>>> Right now, the system just hangs and has no output indicating what the >>>>> problem could be. Although there are a lot of error message before the >>>>> system hangs, I'm not that sure if I should start with solving all of >>>>> those error messages. Maybe some errors can be ignored? >>>>> >>>>> My questions are: >>>>> 1) Do you have suggestion on how to see more information about the >>>>> reason why the dom0 hangs? >>>> >>>> >>>> >>>> Have you tried to dump the registers using Xen console (CTLR-x 3 times >>>> then 0) and see where it get stucks? >>> >>> >>> >>> I tried to type CTLR -x 3 times and then 0, nothing happens... :-( >>> Just to confirm, once the system got stuck, I directly type Ctrl-x for >>> three times on the host's screen. Am I correct? >> >> >> Sorry, I forgot the default way to switch console is CTLR-a three times. >> >> On my configuration I modified the default character to avoid issue with >> screen. >> > > Ah-ha, typing "Ctrl -a, a" for three times, I can switch to the Xen > console now. :-) > > Thank you for the clarification. :-) > >>> >>> Maybe the serial console is not correctly set up? >> >> >> It's likely to be a problem with the serial drivers in driver. Can you tried >> CTLR-a before the kernel is booting? >> >>> The serial console configuration I used is as follows, could you have >>> a quick look to see if it's because I configure the serial >>> incorrectly? >> >> >> I am not familiar with the Nvidia board. If the serial configuration is >> working for baremetal, then it should work for Xen. >> >>> >>> I used screen program to attach to the serial port. >>> The command I used is $screen /dev/ttyUSB0 115200n8 on the host machine. >>> >>> On the board, I set up the device tree's /chosen node as follows: >>> # >>> fdt print /chosen >>> >>> chosen { >>> >>> xen,xen-bootargs = "console=dtuart dtuart=serial0 >>> dom0_mem=512M loglvl=all guest_loglvl=all dom0_max_vcpus=1 >>> dom0_vcpus_pin maxcpus=1"; >>> >>> bootargs = "console=dtuart dtuart=serial0 dom0_mem=512M >>> loglvl=all guest_loglvl=all dom0_max_vcpus=1 dom0_vcpus_pin >>> maxcpus=1"; >> >> >> FIY, this property is not necessary. >> >>> module { >>> >>> bootargs = "console=hvc0 console=tty1 earlyprintk=xen >>> root=/dev/mmcblk0p1 rw rootwait"; >> >> >> Can you try to add "clk_ignore_unused" on the Linux command line? > > Yes. After trying it, it gives me some different,but more useful log > information. > > In the kernel booting log, it first shows this message: > > ---start of the message---- > > [ 10.607251] Waiting for root device /dev/mmcblk0p1... > > ... /* I omit some other unrelated message */ > > [ 5.347354] sdhci-tegra 700b0400.sdhci: Got WP GPIO > > 3mmc0: Unknown controller version (3). You may experience problems. > > [ 5.347464] mmc0: Unknown controller version (3). You may > experience problems. > > sdhci-tegra 700b0400.sdhci: No vmmc regulator found > > [ 5.347647] sdhci-tegra 700b0400.sdhci: No vmmc regulator found > > 3mmc0: Unknown controller version (3). You may experience problems. > > [ 5.347933] mmc0: Unknown controller version (3). You may > experience problems. > > sdhci-tegra 700b0600.sdhci: No vmmc regulator found > > [ 5.348099] sdhci-tegra 700b0600.sdhci: No vmmc regulator found > > sdhci-tegra 700b0600.sdhci: No vqmmc regulator found > > [ 5.348162] sdhci-tegra 700b0600.sdhci: No vqmmc regulator found > > 4mmc0: Invalid maximum block size, assuming 512 bytes > > [ 5.348222] mmc0: Invalid maximum block size, assuming 512 bytes > > 6mmc0: SDHCI controller on 700b0600.sdhci [700b0600.sdhci] using ADMA 64-bit > > [ 5.395208] mmc0: SDHCI controller on 700b0600.sdhci > [700b0600.sdhci] using ADMA 64-bit > > 6usbcore: registered new interface driver usbhid > > ---end of the message---- > > It seems that mmc0 is not correctly recognized by dom0. > > Later the dom0 kernel keeps printing this message: > > [ 426.405136] mmc0: Timeout waiting for hardware interrupt. > > Dom0 actually hangs here, because it cannot read the eMMC device. :-( > > Is it because the device tree is not properly recreated by Xen? > Do you happen to know how to fix this issue or have some idea about > how to fix it? I can have a look at it. > > BTW, dom0 didn't recognize the mmc1 controller either. > > Thank you very much for your time and help in this! :-) > > Best Regards, > > Meng > ----------- > Meng Xu > PhD Student in Computer and Information Science > University of Pennsylvania > http://www.cis.upenn.edu/~mengxu/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel