Re: Cellular modem connection does not come up on boot

[Dan Williams]

On Tue, 2017-04-04 at 21:27 -0400, Liam Monahan wrote:
> I definitely see the conspicuous gap you see with no dial out to #777
> happening. The
> logs that are there are unpruned straight from journalctl.  If I
> hotplug the modem
> back in, then I see additional info about the connection being made
> in the logs
> after “Simple connect started."
> 
> Just to add a little additional color to what appears to be happening
> here…the host
> in question runs docker containers, at least one of which appears to
> be running its
> own systemd process.  I noticed that systemd-udevd runs and
> ModemManager
> looks like it’s picking up on this udev event and re-scanning all
> ports.  When this
> happens, the modem connection terminates.

Interesting.  Could you play with "udevadm --trigger" and see if you
can reproduce the issue?

Dan

> To isolate this, I delayed docker start and let the system become
> quiescent, and
> took all journalctl output since the “Activation successful” message.
> 
> Here’s a log: https://gist.githubusercontent.com/spicybits/f723c9ce51
> df5dad86e83726c44db623/raw/gistfile1.txt
>  6c44db623/raw/gistfile1.txt>
> 
> I first see an "Activation successful.”  Then, at Apr 05 00:57:57 I
> trigger docker to start
> up.  At Apr 05 00:58:00 udev triggers an event and MM reacts.
> 
> Once the modem drops off, it never comes back unless it is replugged.
> 
> Liam
> 
> > On Apr 3, 2017, at 5:15 PM, Dan Williams  wrote:
> > 
> > On Mon, 2017-04-03 at 16:23 -0400, Liam Monahan wrote:
> > > Based off of the better picture I have now of what’s happening
> > > than
> > > when I originally
> > > posted, it looks like there is no condition where MM detection
> > > fails
> > > during boot.  The
> > > problem is that verbose logging seems to suggest that a
> > > connection
> > > does come up
> > > for a short time during boot, but always crashes by the end of
> > > the
> > > boot sequence.
> > > The log posted is very representative of the repeatable behavior
> > > I’ve
> > > been seeing.
> > > 
> > > MM detection and NM connection states both always take place, but
> > > by
> > > the time the
> > > system is quiescent and the boot sequence is “finished,”  the
> > > modem
> > > has disappeared
> > > from “mmcli -L", which in turn no longer presents the device to
> > > NM.  The log I posted is
> > > taken from that scenario taking place.
> > 
> > That's clearer.  A couple things:
> > 
> > 1) are you sure the logs from ModemManager are complete?  I would
> > expect to see a large gap between 12:06:43 and 12:07:13 with  no MM
> > log
> > messages but I'd expect to see some where MM is actually dialing
> > the
> > modem.  For example, the last thing we see is:
> > 
> > Mar 06 12:06:43 e5882e6 ModemManager[676]:   Simple connect
> > started...
> > Mar 06 12:06:43 e5882e6 ModemManager[676]: PIN:
> > unspecified
> > Mar 06 12:06:43 e5882e6 ModemManager[676]: Operator ID:
> > unspecified
> > 
> > and then later MM starts doing signal/registration checks.  But in
> > between there we should certainly see something like "ATDT#777"
> > where
> > MM is actually going through the steps for the "Simple connect",
> > and we
> > should see the resulting "CONNECT" from the modem before PPP
> > happens.
> > 
> > Any idea where those messages might have gone?  If you just grab
> > journal output from ModemManager, do they show up?
> > 
> > > Should I be focused on all the "released by modem” log messages
> > > that
> > > precede a
> > > state change from activated -> unmanaged (reason 'removed’)?  I’m
> > > assuming that removal
> > > is indicated to NM through MM and not directly, right?
> > 
> > Yeah, this is the larger problem.  The modem appears to crash and
> > drop
> > off the USB bus.  This could be due to either a firmware issue that
> > ModemManager is triggering somehow, or it could be due to
> > inadequate
> > power from the USB hub.
> > 
> > Once this happens, does it ever come back?  Or is this what
> > requires
> > the replug?
> > 
> > Dan
> > 
> > > Liam
> > > 
> > > > On Apr 3, 2017, at 3:56 PM, Dan Williams 
> > > > wrote:
> > > > 
> > > > On Mon, 2017-04-03 at 15:08 -0400, Liam Monahan wrote:
> > > > > (a) Full journalctl output is available here: https://gist.gi
> > > > > thub
> > > > > user
> > > > > content.com/spicybits/afca83359e54c1edbc2973388e0821e7/raw/nm
> > > > > -mm-
> > > > > debug.log
> > > > >  > > > > dbc2
> > > > > 9733
> > > > > 88e0821e7/raw/nm-mm-debug.log>  (I wasn't sure about the
> > > > > listserv
> > > > > ettiquete here around posting large logs inline or linking to
> > > > > them…  If this has a resolution, I’ll definitely make sure
> > > > > the
> > > > > relevant parts get preserved for posterity.)
> > > > 
> > > > That log shows a successful USB enumeration, MM detection, and
> > > > NM
> > > > connect.  It doesn't show a failed MM detection 

Re: Cellular modem connection does not come up on boot

[Dan Williams]

On Mon, 2017-04-03 at 11:15 -0400, Liam Monahan wrote:
> Thanks for your guidance.
> 
> When the device doesn’t show up in NM, it can’t be seen through
> mmcli.  I turned up the logging level to see what’s going on.  Maybe
> you have a different reading of this, but the first thing that jumps
> out at me is that something seems to be sending NM/MM a TERM signal.

Ok, if the device cannot be seen in 'mmcli' that means ModemManager
didn't find it either.  MM must see it, because MM presents the modem
to NetworkManager.

If MM doesn't see it, then there are two possibilities:

(a) ModemManager bug with device detection; here we need boot-time logs
of ModemManager with debug log level to see if MM is told about the
device by the kernel/udev, and what it does with that device.  To do
this, edit /usr/lib/systemd/system/ModemManager.service and change the
ExecStart= line to add "--log-level=debug" and reboot the machine until
the problem occurs.  Then grab ModemManager journal output so we can
debug.

(b) kernel driver problems; usually due to your USB Host Controller
driver rather than a modem driver itself; there are sometimes issues
with device enumeration and specific USB chipsets.  'dmesg' output
usually shows this as the device simply not being enumerated on boot-up 
and not showing up in 'lsusb'.  Unfortunately, can take some pretty
heavy kernel debugging.

I see this is a Raspberry Pi 3 using dwc_otg as the USB host controller
driver.  Is the modem behind that USB hub I see being enumerated?  Have
you tried using a different hub to see if that makes a difference?  Is
the hub powered?

Dan

> # journalctl --no-pager --unit=ModemManager --unit=NetworkManager
> -n200
> -- Logs begin at Mon 2017-03-06 12:06:25 UTC, end at Mon 2017-03-06
> 12:08:27 UTC. --
> Mar 06 12:06:29 raspberrypi3 systemd[1]: Starting Modem Manager...
> Mar 06 12:06:29 raspberrypi3 systemd[1]: Starting Network Manager...
> Mar 06 12:06:30 raspberrypi3 ModemManager[637]:   ModemManager
> (version 1.6.4) starting in system bus...
> Mar 06 12:06:30 e5882e6 NetworkManager[723]:
>   [1488801990.5777] NetworkManager (version 1.4.2) is
> starting...
> Mar 06 12:06:30 e5882e6 NetworkManager[723]:
>   [1488801990.5783] Read config:
> /etc/NetworkManager/NetworkManager.conf
> Mar 06 12:06:30 e5882e6 NetworkManager[723]:
>   [1488801990.6102] manager[0x19c9880]: monitoring kernel
> firmware directory '/lib/firmware'.
> Mar 06 12:06:30 e5882e6 NetworkManager[723]:
>   [1488801990.6647] dns-mgr[0x19d4818]: init: dns=default, rc-
> manager=resolvconf
> Mar 06 12:06:30 e5882e6 NetworkManager[723]:
>   [1488801990.6692] rfkill0: found WiFi radio killswitch (at
> /sys/devices/platform/soc/3f30.mmc/mmc_host/mmc1/mmc1:0001/mmc1:0
> 001:1/ieee80211/phy0/rfkill0) (driver brcmfmac_sdio)
> Mar 06 12:06:30 e5882e6 NetworkManager[723]:
>   [1488801990.6702] manager[0x19c9880]: WiFi hardware radio set
> enabled
> Mar 06 12:06:30 e5882e6 NetworkManager[723]:
>   [1488801990.6705] manager[0x19c9880]: WWAN hardware radio set
> enabled
> Mar 06 12:06:30 e5882e6 systemd[1]: Started Network Manager.
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.1467] settings: loaded plugin keyfile: (c) 2007 -
> 2015 Red Hat, Inc.  To report bugs please use the NetworkManager
> mailing list.
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.2602] keyfile: new connection
> /etc/NetworkManager/system-connections/resin-sample (bcb05b81-52cb-
> 3828-8088-c0f55e950375,"resin-sample")
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.3534] keyfile: new connection
> /etc/NetworkManager/system-connections/resin-verizon (9290a7a8-70f7-
> 44b0-94e4-83f0b2635f07,"resin-verizon")
> Mar 06 12:06:31 e5882e6 systemd[1]: Started Modem Manager.
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.8042] settings: hostname: using hostnamed
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.8043] settings: hostname changed from (none) to
> "e5882e6"
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.8055] dhcp-init: Using DHCP client 'dhclient'
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.8056] manager: WiFi enabled by radio killswitch;
> enabled by state file
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.8059] manager: WWAN enabled by radio killswitch;
> enabled by state file
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.8064] manager: Networking is enabled by state
> file
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.8068] Loaded device plugin: NMVxlanFactory
> (internal)
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.8069] Loaded device plugin: NMVlanFactory
> (internal)
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.8071] Loaded device plugin: NMVethFactory
> (internal)
> Mar 06 12:06:31 e5882e6 NetworkManager[723]:
>   [1488801991.8072] Loaded device plugin: NMTunFactory
> (internal)
> Mar 06 12:06:31 e5882e6 NetworkManag

Re: Cellular modem connection does not come up on boot

[Liam Monahan]

Thanks for your guidance.

When the device doesn’t show up in NM, it can’t be seen through mmcli.  I 
turned up the logging level to see what’s going on.  Maybe you have a different 
reading of this, but the first thing that jumps out at me is that something 
seems to be sending NM/MM a TERM signal.

# journalctl --no-pager --unit=ModemManager --unit=NetworkManager -n200
-- Logs begin at Mon 2017-03-06 12:06:25 UTC, end at Mon 2017-03-06 12:08:27 
UTC. --
Mar 06 12:06:29 raspberrypi3 systemd[1]: Starting Modem Manager...
Mar 06 12:06:29 raspberrypi3 systemd[1]: Starting Network Manager...
Mar 06 12:06:30 raspberrypi3 ModemManager[637]:   ModemManager (version 
1.6.4) starting in system bus...
Mar 06 12:06:30 e5882e6 NetworkManager[723]:   [1488801990.5777] 
NetworkManager (version 1.4.2) is starting...
Mar 06 12:06:30 e5882e6 NetworkManager[723]:   [1488801990.5783] Read 
config: /etc/NetworkManager/NetworkManager.conf
Mar 06 12:06:30 e5882e6 NetworkManager[723]:   [1488801990.6102] 
manager[0x19c9880]: monitoring kernel firmware directory '/lib/firmware'.
Mar 06 12:06:30 e5882e6 NetworkManager[723]:   [1488801990.6647] 
dns-mgr[0x19d4818]: init: dns=default, rc-manager=resolvconf
Mar 06 12:06:30 e5882e6 NetworkManager[723]:   [1488801990.6692] rfkill0: 
found WiFi radio killswitch (at 
/sys/devices/platform/soc/3f30.mmc/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/ieee80211/phy0/rfkill0)
 (driver brcmfmac_sdio)
Mar 06 12:06:30 e5882e6 NetworkManager[723]:   [1488801990.6702] 
manager[0x19c9880]: WiFi hardware radio set enabled
Mar 06 12:06:30 e5882e6 NetworkManager[723]:   [1488801990.6705] 
manager[0x19c9880]: WWAN hardware radio set enabled
Mar 06 12:06:30 e5882e6 systemd[1]: Started Network Manager.
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.1467] 
settings: loaded plugin keyfile: (c) 2007 - 2015 Red Hat, Inc.  To report bugs 
please use the NetworkManager mailing list.
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.2602] keyfile: 
new connection /etc/NetworkManager/system-connections/resin-sample 
(bcb05b81-52cb-3828-8088-c0f55e950375,"resin-sample")
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.3534] keyfile: 
new connection /etc/NetworkManager/system-connections/resin-verizon 
(9290a7a8-70f7-44b0-94e4-83f0b2635f07,"resin-verizon")
Mar 06 12:06:31 e5882e6 systemd[1]: Started Modem Manager.
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8042] 
settings: hostname: using hostnamed
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8043] 
settings: hostname changed from (none) to "e5882e6"
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8055] 
dhcp-init: Using DHCP client 'dhclient'
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8056] manager: 
WiFi enabled by radio killswitch; enabled by state file
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8059] manager: 
WWAN enabled by radio killswitch; enabled by state file
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8064] manager: 
Networking is enabled by state file
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8068] Loaded 
device plugin: NMVxlanFactory (internal)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8069] Loaded 
device plugin: NMVlanFactory (internal)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8071] Loaded 
device plugin: NMVethFactory (internal)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8072] Loaded 
device plugin: NMTunFactory (internal)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8074] Loaded 
device plugin: NMMacvlanFactory (internal)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8075] Loaded 
device plugin: NMIPTunnelFactory (internal)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8077] Loaded 
device plugin: NMInfinibandFactory (internal)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8079] Loaded 
device plugin: NMEthernetFactory (internal)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8083] Loaded 
device plugin: NMBridgeFactory (internal)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8085] Loaded 
device plugin: NMBondFactory (internal)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.8525] Loaded 
device plugin: NMWwanFactory 
(/usr/lib/NetworkManager/libnm-device-plugin-wwan.so)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.9084] Loaded 
device plugin: NMBluezManager 
(/usr/lib/NetworkManager/libnm-device-plugin-bluetooth.so)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.9428] Loaded 
device plugin: NMWifiFactory 
(/usr/lib/NetworkManager/libnm-device-plugin-wifi.so)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.9704] device 
(lo): link connected
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.9763] manager: 
(lo): new Generic device (/org/freedesktop/NetworkManager/Devices/0)
Mar 06 12:06:31 e5882e6 NetworkManager[723]:   [1488801991.9883

Re: Cellular modem connection does not come up on boot

[Dan Williams]

On Wed, 2017-03-29 at 10:48 -0400, Liam Monahan wrote:
> I have a USB cellular modem that will not show up in nm on boot.  If
> the system is started up and I unplug and plug it back in, it shows
> up then.  These device are deployed out in the field, so manual
> intervention is not a real option for my use case unfortunately.
> 
> device is ttyUSB3 and does not show up at first:
> 
> # nmcli d
> DEVICE TYPE  STATE CONNECTION 
> eth0   ethernet  connected Wired connection 1 
> wlan0  wifi  disconnected  -- 
> docker0bridgeunmanaged -- 
> lo loopback  unmanaged -- 
> resin-vpn  tun   unmanaged — 

When the device doesn't show up in NM, can you see it with "mmcli -L"
before you replug it?

If you can see it in mmcli, then the problem is likely in
NetworkManager.  Getting debug logs would be necessary, which you can
do by adding this to /etc/NetworkManager/NetworkManager.conf:

[logging]
level=trace

If you can't see it in ModemManager, then the bug is likely in MM.  You
can enable verbose MM debugging by modifying the ModemManager system
unit file and adding "--log-level=debug" to the Exec= line.

One of those should give us more info about the issue.

Dan

> and then after unplugging and plugging back in:
>  
> # nmcli d
> DEVICE TYPE  STATE CONNECTION 
> ttyUSB3cdma  connected resin-verizon  
> eth0   ethernet  connected Wired connection 1 
> wlan0  wifi  disconnected  -- 
> docker0bridgeunmanaged -- 
> lo loopback  unmanaged -- 
> resin-vpn  tun   unmanaged --   
> 
> # mmcli -L
> 
> Found 1 modems:
>   /org/freedesktop/ModemManager1/Modem/0 [Telit] DE910-DUAL
> 
> 
> dmesg makes it seem like the modem connected briefly and then
> disappeared.  Does anyone have any thoughts on what’s going on or how
> I could do a better job of going about debugging this?
> 
> [0.00] Booting Linux on physical CPU 0x0
> [0.00] Initializing cgroup subsys cpuset
> [0.00] Initializing cgroup subsys cpu
> [0.00] Initializing cgroup subsys cpuacct
> [0.00] Linux version 4.4.48 (and...@misc1.dev.resin.io) (gcc
> version 6.2.0 (GCC) ) #2 SMP Sat Mar 11 03:08:12 UTC 2017
> [0.00] CPU: ARMv7 Processor [410fd034] revision 4 (ARMv7),
> cr=10c5383d
> [0.00] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing
> instruction cache
> [0.00] Machine model: Raspberry Pi 3 Model B Rev 1.2
> [0.00] cma: Reserved 8 MiB at 0x3d80
> [0.00] Memory policy: Data cache writealloc
> [0.00] On node 0 totalpages: 253952
> [0.00] free_area_init_node: node 0, pgdat 80ca84c0,
> node_mem_map bcf3a000
> [0.00]   Normal zone: 2232 pages used for memmap
> [0.00]   Normal zone: 0 pages reserved
> [0.00]   Normal zone: 253952 pages, LIFO batch:31
> [0.00] [bcm2709_smp_init_cpus] enter (9540->f3003010)
> [0.00] [bcm2709_smp_init_cpus] ncores=4
> [0.00] PERCPU: Embedded 13 pages/cpu @bcef6000 s23052 r8192
> d22004 u53248
> [0.00] pcpu-alloc: s23052 r8192 d22004 u53248 alloc=13*4096
> [0.00] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
> [0.00] Built 1 zonelists in Zone order, mobility grouping
> on.  Total pages: 251720
> [0.00] Kernel command line: 8250.nr_uarts=1
> dma.dmachans=0x7f35 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416
> bcm2709.boardrev=0xa22082 bcm2709.serial=0xca8e10ef
> smsc95xx.macaddr=B8:27:EB:8E:10:EF bcm2708_fb.fbdepth=16
> bcm2708_fb.fbswap=1 bcm2709.uart_clock=4800
> vc_mem.mem_base=0x3ea0
> vc_mem.mem_size=0x3f00  dwc_otg.lpm_enable=0 console=tty1
> console=ttyS0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait
> [0.00] PID hash table entries: 4096 (order: 2, 16384 bytes)
> [0.00] Dentry cache hash table entries: 131072 (order: 7,
> 524288 bytes)
> [0.00] Inode-cache hash table entries: 65536 (order: 6,
> 262144 bytes)
> [0.00] Memory: 983796K/1015808K available (7029K kernel code,
> 454K rwdata, 1956K rodata, 3524K init, 775K bss, 23820K reserved,
> 8192K cma-reserved)
> [0.00] Virtual kernel memory layout:
>    vector  : 0x - 0x1000   (   4 kB)
>    fixmap  : 0xffc0 - 0xfff0   (3072 kB)
>    vmalloc : 0xbe80 - 0xff80   (1040 MB)
>    lowmem  : 0x8000 - 0xbe00   ( 992 MB)
>    modules : 0x7f00 - 0x8000   (  16 MB)
>  .text : 0x80008000 - 0x808ce6fc   (8986 kB)
>  .init : 0x808cf000 - 0x80c4   (3524 kB)
>  .data : 0x80c4 - 0x80cb1a80   ( 455 kB)
>   .bss : 0x80cb4000 - 0x80d75f84   ( 776 kB)
> [0.00] SLUB: HWalign=64, Order=0-3, MinObjects=0,

Cellular modem connection does not come up on boot

[Liam Monahan]

I have a USB cellular modem that will not show up in nm on boot.  If the system 
is started up and I unplug and plug it back in, it shows up then.  These device 
are deployed out in the field, so manual intervention is not a real option for 
my use case unfortunately.

device is ttyUSB3 and does not show up at first:

# nmcli d
DEVICE TYPE  STATE CONNECTION 
eth0   ethernet  connected Wired connection 1 
wlan0  wifi  disconnected  -- 
docker0bridgeunmanaged -- 
lo loopback  unmanaged -- 
resin-vpn  tun   unmanaged — 

and then after unplugging and plugging back in:
 
# nmcli d
DEVICE TYPE  STATE CONNECTION 
ttyUSB3cdma  connected resin-verizon  
eth0   ethernet  connected Wired connection 1 
wlan0  wifi  disconnected  -- 
docker0bridgeunmanaged -- 
lo loopback  unmanaged -- 
resin-vpn  tun   unmanaged --   

# mmcli -L

Found 1 modems:
/org/freedesktop/ModemManager1/Modem/0 [Telit] DE910-DUAL


dmesg makes it seem like the modem connected briefly and then disappeared.  
Does anyone have any thoughts on what’s going on or how I could do a better job 
of going about debugging this?

[0.00] Booting Linux on physical CPU 0x0
[0.00] Initializing cgroup subsys cpuset
[0.00] Initializing cgroup subsys cpu
[0.00] Initializing cgroup subsys cpuacct
[0.00] Linux version 4.4.48 (and...@misc1.dev.resin.io) (gcc version 
6.2.0 (GCC) ) #2 SMP Sat Mar 11 03:08:12 UTC 2017
[0.00] CPU: ARMv7 Processor [410fd034] revision 4 (ARMv7), cr=10c5383d
[0.00] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing 
instruction cache
[0.00] Machine model: Raspberry Pi 3 Model B Rev 1.2
[0.00] cma: Reserved 8 MiB at 0x3d80
[0.00] Memory policy: Data cache writealloc
[0.00] On node 0 totalpages: 253952
[0.00] free_area_init_node: node 0, pgdat 80ca84c0, node_mem_map 
bcf3a000
[0.00]   Normal zone: 2232 pages used for memmap
[0.00]   Normal zone: 0 pages reserved
[0.00]   Normal zone: 253952 pages, LIFO batch:31
[0.00] [bcm2709_smp_init_cpus] enter (9540->f3003010)
[0.00] [bcm2709_smp_init_cpus] ncores=4
[0.00] PERCPU: Embedded 13 pages/cpu @bcef6000 s23052 r8192 d22004 
u53248
[0.00] pcpu-alloc: s23052 r8192 d22004 u53248 alloc=13*4096
[0.00] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[0.00] Built 1 zonelists in Zone order, mobility grouping on.  Total 
pages: 251720
[0.00] Kernel command line: 8250.nr_uarts=1 dma.dmachans=0x7f35 
bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2709.boardrev=0xa22082 
bcm2709.serial=0xca8e10ef smsc95xx.macaddr=B8:27:EB:8E:10:EF 
bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 bcm2709.uart_clock=4800 
vc_mem.mem_base=0x3ea0 vc_mem.mem_size=0x3f00  dwc_otg.lpm_enable=0 
console=tty1 console=ttyS0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait
[0.00] PID hash table entries: 4096 (order: 2, 16384 bytes)
[0.00] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[0.00] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[0.00] Memory: 983796K/1015808K available (7029K kernel code, 454K 
rwdata, 1956K rodata, 3524K init, 775K bss, 23820K reserved, 8192K cma-reserved)
[0.00] Virtual kernel memory layout:
   vector  : 0x - 0x1000   (   4 kB)
   fixmap  : 0xffc0 - 0xfff0   (3072 kB)
   vmalloc : 0xbe80 - 0xff80   (1040 MB)
   lowmem  : 0x8000 - 0xbe00   ( 992 MB)
   modules : 0x7f00 - 0x8000   (  16 MB)
 .text : 0x80008000 - 0x808ce6fc   (8986 kB)
 .init : 0x808cf000 - 0x80c4   (3524 kB)
 .data : 0x80c4 - 0x80cb1a80   ( 455 kB)
  .bss : 0x80cb4000 - 0x80d75f84   ( 776 kB)
[0.00] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[0.00] Hierarchical RCU implementation.
[0.00]  Build-time adjustment of leaf fanout to 32.
[0.00] NR_IRQS:16 nr_irqs:16 16
[0.00] Architected cp15 timer(s) running at 19.20MHz (phys).
[0.00] clocksource: arch_sys_counter: mask: 0xff 
max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns
[0.08] sched_clock: 56 bits at 19MHz, resolution 52ns, wraps every 
4398046511078ns
[0.24] Switching to timer-based delay loop, resolution 52ns
[0.000269] Console: colour dummy device 80x30
[0.001295] console [tty1] enabled
[0.001349] Calibrating delay loop (skipped), value calculated using timer 
frequency.. 38.40 BogoMIPS (lpj=192000)
[0.001417] pid_max: default: 32768 minimum: 301
[0.0