Hi Song,
The xCAT probe session output is below:
# xcatprobe osdeploy -n gn001
The install NIC in current server is p2p2
[INFO]
All nodes to be deployed are valid
[ OK ]
-------------------------------------------------------------
Start capturing every message during OS provision process....
-------------------------------------------------------------
[gn001] 10:04:13 Receive DHCPDISCOVER via p2p2
[gn001] 10:04:13 Send DHCPOFFER on 10.1.2.1 back to 00:25:90:6c:a8:a2 via
p2p2
[gn001] 10:04:15 DHCPREQUEST for 10.1.2.1 (10.1.0.254) from
00:25:90:6c:a8:a2 via p2p2
[gn001] 10:04:15 Send DHCPACK on 10.1.2.1 back to 00:25:90:6c:a8:a2 via p2p2
[gn001] 10:04:15 Via TFTP download pxelinux.0
[gn001] 10:04:15 Via TFTP download pxelinux.0
[gn001] 10:04:15 Via TFTP download
pxelinux.cfg/00000000-0000-0000-0000-0025906ca8a2
[gn001] 10:04:15 Via TFTP download pxelinux.cfg/01-00-25-90-6c-a8-a2
[gn001] 10:04:15 Via TFTP download pxelinux.cfg/0A010201
[gn001] 10:04:15 Via TFTP download
xcat/osimage/centos7.6-x86_64-netboot-gpu-compute/kernel
[gn001] 10:04:16 Via TFTP download
xcat/osimage/centos7.6-x86_64-netboot-gpu-compute/initrd-stateless.gz
[gn001] 10:20:10 Receive DHCPDISCOVER via p2p2
[gn001] 10:20:10 Send DHCPOFFER on 10.1.2.1 back to 00:25:90:6c:a8:a2 via
p2p2
[gn001] 10:20:10 DHCPREQUEST for 10.1.2.1 (10.1.0.254) from
00:25:90:6c:a8:a2 via p2p2
[gn001] 10:20:10 Send DHCPACK on 10.1.2.1 back to 00:25:90:6c:a8:a2 via p2p2
[gn001] 10:20:17 INFO =============deployment starting====================
[gn001] 10:20:17 INFO =============deployment starting====================
[gn001] 10:20:17 INFO Executing xcatroot to prepare for netbooting
(dracut_33)...
[gn001] 10:20:17 INFO Sending request to 10.1.0.254:3002 for changing
status to netbooting...
[gn001] 10:20:18 Node status is changed to netbooting
[gn001] 10:20:18 INFO Downloading rootfs image from
http://10.1.0.254:80//install/netboot/centos7.6/x86_64/gpu-...
[gn001] 10:20:17 INFO Executing xcatroot to prepare for netbooting
(dracut_33)...
[gn001] 10:20:17 INFO Sending request to 10.1.0.254:3002 for changing
status to netbooting...
[gn001] 10:20:18 INFO Downloading rootfs image from
http://10.1.0.254:80//install/netboot/centos7.6/x86_64/gpu-...
[gn001] 10:20:18 Via HTTP get
//install/netboot/centos7.6/x86_64/gpu-compute/rootimg.cpio.gz
[gn001] 10:20:27 INFO Setting up RAM-root tmpfs on downloaded
rootimg.cpio.[gz/xz]...
[gn001] 10:20:27 INFO Setting up RAM-root tmpfs on downloaded
rootimg.cpio.[gz/xz]...
[gn001] 10:20:47 INFO Exiting xcatroot...
[gn001] 10:20:47 INFO Exiting xcatroot...
And below, the piece of the DHCP lease file for the node:
host gn001 {
dynamic;
hardware ethernet 00:25:90:6c:a8:a2;
uid 00:25:90:6c:a8:a2;
fixed-address 10.1.2.1;
supersede server.ddns-hostname = "gn001";
supersede host-name = "gn001";
if option vendor-class-identifier = "ScaleMP" {
supersede server.filename = "vsmp/pxelinux.0";
} else {
supersede server.filename = "pxelinux.0";
}
}
Regards
--
Angelo Cavalcanti
br.linkedin.com/in/angelocr
Em sáb, 16 de mar de 2019 às 09:38, Angelo Cavalcanti <
[email protected]> escreveu:
> 1. The status is "powering-on"
>
> 2. Yes, the issue happens in the same node
>
> 3. Ok. I will send the xCAT-probe output session
>
> Angelo Cavalcanti
> br.linkedin.com/in/angelocr
>
>
>
> Em sex, 15 de mar de 2019 às 07:37, Song BJ Yang <[email protected]>
> escreveu:
>
>> Hi,
>>
>> If the console output covers the whole process, seems the the initrd boot
>> up process did not reach the rootimg download phase. And there is a
>> `[2019-03-14T10:20:39-03:00]
>> [ 37.280041] systemd-fstab-generator[261]: Could not find a root= entry
>> on the kernel command line.`,
>>
>> several questions:
>>
>> 1. what is the node status: `lsdef <node> -i status,statustime`? is it
>> changed to "netbooting"?
>> 2. did you provision a batch of nodes with the same osimage? did the
>> issue always appear on the same node?
>> 3. please install xCAT-probe on you MN, run `xcatprobe xcatmn` to check
>> if any configuration issue.
>> and watch `xcatprobe osdeploy -n <failing node>` in 1 terminal session,
>> and then kick off provision. You will find the provision progress in
>> the xcatprobe session. Please provide that session output.
>>
>> ------------------------------------------------------------------------------
>> YANG Song (杨嵩)
>> IBM China System Technology Laboratory
>> Tel: 86-10-82452903
>> Email: [email protected]
>> Address: Building 28, ZhongGuanCun Software Park,
>> No.8, Dong Bei Wang West Road, Haidian District Beijing 100193, PRC
>>
>> 北京市海淀区东北旺西路8号中关村软件园28号楼
>> 邮编: 100193
>>
>>
>>
>> ----- Original message -----
>> From: Angelo Cavalcanti <[email protected]>
>> To: Song BJ Yang <[email protected]>
>> Cc: [email protected]
>> Subject: Re: [xcat-user] [External] Netboot process stuck
>> Date: Fri, Mar 15, 2019 10:37 AM
>>
>> Thanks Song,
>>
>> I added the following kernel parameters:
>>
>> debug ignore_loglevel log_buf_len=10M print_fatal_signals=1
>>
>> The console output file is attached. I noticed that the machine's devices
>> were not found in the udev database.
>>
>> Regards,
>>
>> Angelo Cavalcanti
>> br.linkedin.com/in/angelocr
>>
>> Enviado do Gmail Android App
>>
>> Em qui, 14 de mar de 2019 06:41, Song BJ Yang <[email protected]>
>> escreveu:
>>
>> Hi,
>>
>> We encountered a similar issue
>> https://github.com/xcat2/xcat-core/issues/274 , but in this case the
>> console uncovered the root cause.
>>
>>
>> However, your console output does not show why the boot up process hang.
>> I suggest you add more verbose output during boot up, this is a reference
>> dochttps://www.askapache.com/linux/linux-debugging/ on how to get more
>> debug info during kernel boot up.
>>
>> To apply the kernel options during diskless kernel boot up, you can
>> leverage the `addkcmdline` attribute, an example for addkcmdline usage ,
>>
>> chdef mid05tor12cn05 addkcmdline="debug ignore_loglevel log_buf_len=10M
>> print_fatal_signals=1 LOGLEVEL=8 earlyprintk=vga,keep sched_debug"
>>
>>
>> good luck
>>
>>
>> ------------------------------------------------------------------------------
>> YANG Song (杨嵩)
>> IBM China System Technology Laboratory
>> Tel: 86-10-82452903
>> Email: [email protected]
>> Address: Building 28, ZhongGuanCun Software Park,
>> No.8, Dong Bei Wang West Road, Haidian District Beijing 100193, PRC
>>
>> 北京市海淀区东北旺西路8号中关村软件园28号楼
>> 邮编: 100193
>>
>>
>>
>> ----- Original message -----
>> From: Jarrod Johnson <[email protected]>
>> To: xCAT Users Mailing list <[email protected]>
>> Cc:
>> Subject: Re: [xcat-user] [External] Netboot process stuck
>> Date: Thu, Mar 14, 2019 5:33 AM
>>
>>
>> What does the boot kernel get command line wise (e.g.
>> /tftpboot/xcat/xnba/nodes/<nodename>)
>>
>>
>>
>> *From:* Angelo Cavalcanti <[email protected]>
>> *Sent:* Wednesday, March 13, 2019 4:23 PM
>> *To:* [email protected]
>> *Subject:* [External] [xcat-user] Netboot process stuck
>>
>>
>>
>> Hi everyone,
>>
>>
>>
>> I've setup various nodes to netboot image but one of them stuck in boot
>> process, below:
>>
>>
>>
>> [ 485.127448] systemd[1]: Reached target Sockets.
>>
>> [ 515.134248] systemd[1]: Started Journal Service.
>>
>> [ 635.252091] RPC: Registered named UNIX socket transport module.
>>
>> [ 635.258163] RPC: Registered udp transport module.
>>
>> [ 635.263004] RPC: Registered tcp transport module.
>>
>> [ 635.267847] RPC: Registered tcp NFSv4.1 backchannel transport module.
>>
>> [ 845.308189] pps_core: LinuxPPS API ver. 1 registered
>>
>> [ 845.313333] pps_core: Software ver. 5.3.6 - Copyright 2005-2007
>> Rodolfo Giometti <[email protected]>
>>
>> [ 845.325629] PTP clock support registered
>>
>> [ 845.332035] dca service started, version 1.12.1
>>
>> [ 845.343852] mlx4_core: Mellanox ConnectX core driver v4.0-0
>>
>> [ 845.349585] mlx4_core: Initializing 0000:04:00.0
>>
>> [ 845.364042] igb: Intel(R) Gigabit Ethernet Network Driver - version
>> 5.4.0-k
>>
>> [ 845.368351] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps
>> 0x3f impl SATA mode
>>
>> [ 845.368355] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio
>> slum part ems apst
>>
>> [ 845.380249] scsi host0: ahci
>>
>> [ 845.380938] scsi host1: ahci
>>
>> [ 845.382757] scsi host2: ahci
>>
>> [ 845.383161] scsi host3: ahci
>>
>> [ 845.386805] scsi host4: ahci
>>
>> [ 845.387739] scsi host5: ahci
>>
>> [ 845.387852] ata1: SATA max UDMA/133 abar m2048@0xde100000 port
>> 0xde100100 irq 39
>>
>> [ 845.387855] ata2: SATA max UDMA/133 abar m2048@0xde100000 port
>> 0xde100180 irq 39
>>
>> [ 845.387857] ata3: SATA max UDMA/133 abar m2048@0xde100000 port
>> 0xde100200 irq 39
>>
>> [ 845.387860] ata4: SATA max UDMA/133 abar m2048@0xde100000 port
>> 0xde100280 irq 39
>>
>> [ 845.387862] ata5: SATA max UDMA/133 abar m2048@0xde100000 port
>> 0xde100300 irq 39
>>
>> [ 845.387865] ata6: SATA max UDMA/133 abar m2048@0xde100000 port
>> 0xde100380 irq 39
>>
>> [ 845.451810] igb: Copyright (c) 2007-2014 Intel Corporation.
>>
>> [ 845.511186] igb 0000:81:00.0: added PHC on eth0
>>
>> [ 845.515869] igb 0000:81:00.0: Intel(R) Gigabit Ethernet Network
>> Connection
>>
>> [ 845.522896] igb 0000:81:00.0: eth0: (PCIe:5.0Gb/s:Width x4)
>> 00:25:90:6c:a8:a2
>>
>> [ 845.530236] igb 0000:81:00.0: eth0: PBA No: 104900-000
>>
>> [ 845.535510] igb 0000:81:00.0: Using MSI-X interrupts. 8 rx queue(s), 8
>> tx queue(s)
>>
>> [ 845.598681] igb 0000:81:00.1: added PHC on eth1
>>
>> [ 845.603379] igb 0000:81:00.1: Intel(R) Gigabit Ethernet Network
>> Connection
>>
>> [ 845.610404] igb 0000:81:00.1: eth1: (PCIe:5.0Gb/s:Width x4)
>> 00:25:90:6c:a8:a3
>>
>> [ 845.617768] igb 0000:81:00.1: eth1: PBA No: 104900-000
>>
>> [ 845.623050] igb 0000:81:00.1: Using MSI-X interrupts. 8 rx queue(s), 8
>> tx queue(s)
>>
>> [ 845.694180] ata4: SATA link down (SStatus 0 SControl 300)
>>
>> [ 845.699730] ata3: SATA link down (SStatus 0 SControl 300)
>>
>> [ 845.705276] ata2: SATA link down (SStatus 0 SControl 300)
>>
>> [ 845.710813] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
>>
>> [ 845.717162] ata5: SATA link down (SStatus 0 SControl 300)
>>
>> [ 845.722738] ata6: SATA link down (SStatus 0 SControl 300)
>>
>> [ 845.728523] ata1.00: ATA-8: ST91000640NS, SN03, max UDMA/133
>>
>> [ 845.734330] ata1.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth
>> 31/32)
>>
>> [ 845.742162] ata1.00: configured for UDMA/133
>>
>> [ 845.746833] scsi 0:0:0:0: Direct-Access ATA ST91000640NS
>> SN03 PQ: 0 ANSI: 5
>>
>> [ 845.791663] sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks:
>> (1.00 TB/931 GiB)
>>
>> [ 845.799600] sd 0:0:0:0: [sda] Write Protect is off
>>
>> [ 845.804545] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>>
>> [ 845.820557] sda: sda1 sda2
>>
>> [ 845.823892] sd 0:0:0:0: [sda] Attached SCSI disk
>>
>> [ 851.888095] mlx4_core 0000:04:00.0: Old device ETS support detected
>>
>> [ 851.894490] mlx4_core 0000:04:00.0: Consider upgrading device FW.
>>
>> [ 852.632012] mlx4_core 0000:04:00.0: PCIe link speed is 8.0GT/s, device
>> supports 8.0GT/s
>>
>> [ 852.640265] mlx4_core 0000:04:00.0: PCIe link width is x8, device
>> supports x8
>>
>> [ 852.787509] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
>>
>> [ 945.585085] igb 0000:81:00.0: changing MTU from 1500 to 2044
>>
>> [ 949.821377] igb 0000:81:00.0 enp129s0f0: igb: enp129s0f0 NIC Link is
>> Up 1000 Mbps Full Duplex, Flow Control: RX
>>
>> [ 950.903525] random: crng init done
>>
>>
>>
>> Notice that the boot process is slow.
>>
>>
>>
>> The machine has the following configuration:
>>
>> 2x Intel Xeon E5-2670
>>
>> 256GB RAM
>>
>> Motherboard Supermicro X9DRG-HF
>>
>> HDD 1TB
>>
>> Mellanox Infiniband ConnectX-3 card (MT27500)
>>
>> GPGPU nVidia Tesla M2075
>>
>>
>>
>> I removed all off-board cards and HDD. The boot process stays stuck in
>> the same stage. I installed CentOS 7 minimal ISO on HDD and the problem did
>> not occur.
>>
>>
>>
>> Regards,
>>
>>
>>
>> --
>>
>> Angelo Cavalcanti
>> br.linkedin.com/in/angelocr
>> _______________________________________________
>> xCAT-user mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>
>>
>>
>>
>>
>> _______________________________________________
>> xCAT-user mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>
>
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
