Re: [LEDE-DEV] Recent issues w/ sysupgrade

[Ryan Mounce]

The culprit is
e505f59 utils/util-linux: Update to 2.30.1

On 6 September 2017 at 12:24, Philip Prindeville
 wrote:
>
> Thanks.
>
> It’s not 100% reproducible which makes it time-consuming to bisect because 
> you have to repeat steps a few times.
>
> But I did conclude decisively that it did NOT present before:
>
> 5eb216e mediatek: update to latest kernel patchset from v4.13-rc
>
> So you don’t have to search any earlier than that.
>
> It’s in the last 3 weeks.
>
> -Philip
>

___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev

Re: [LEDE-DEV] Recent issues w/ sysupgrade

[Ryan Mounce]

On 31 August 2017 at 04:11, Philip Prindeville
 wrote:
> Looking a little further into the console logging, what I’m seeing after a 
> “dd” directly onto the flash is this:
>
> Press the [f] key and hit [enter] to enter failsafe mode
> Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
> [6.299598] mount_root: loading kmods from internal overlay
> [6.307977] kmodloader: loading kernel modules from //etc/modules-boot.d/*
> [6.316014] kmodloader: done loading kernel modules from 
> //etc/modules-boot.d/*
> [6.459868] block: attempting to load /etc/config/fstab
> [6.472689] block: extroot: not configured
> [6.488330] mount_root: rootdisk overlay filesystem has not been formatted 
> yet
> [6.517993] blk_update_request: I/O error, dev loop0, sector 0
> [6.724898] block: attempting to load /etc/config/fstab
> [6.730248] block: extroot: not configured
> [6.734560] mount_root: overlay filesystem has not been fully initialized 
> yet
> [6.741823] mount_root: switching to f2fs overlay
>
>
> if I take the exact same image and try to reinstall it with sysupgrade, I see 
> this:
>
>
> Press the [f] key and hit [enter] to enter failsafe mode
> Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
> [5.319594] mount_root: loading kmods from internal overlay
> [5.328062] kmodloader: loading kernel modules from //etc/modules-boot.d/*
> [5.336078] kmodloader: done loading kernel modules from 
> //etc/modules-boot.d/*
> [5.480712] block: attempting to load /etc/config/fstab
> [5.493487] block: extroot: not configured
> [5.498237] random: procd: uninitialized urandom read (4 bytes read)
> [5.515173] mount_root: rootdisk overlay filesystem has not been formatted 
> yet
>
>
> it’s tough to reproduce, which makes me wonder if it’s either a race 
> condition or an uninitialized variable.

I can consistently reproduce this issue when testing my Turris Omnia
patch set, hopefully I can help to narrow it down.

A couple of times interrupting boot into failsafe and then running
mount_root has fixed things, however more often than not mount_root
hangs in the same way.

This appears to have been introduced in the last couple of months so I
will try to bisect it.

>
> This was after I rebased to:
>
> 2f0a855 lantiq: ACMP252: clean up device modules
>
> What would be significantly different about dd’ing an image onto the CF 
> versus the copy that sysupgrade performs?
>
> -Philip
>
>
> ___
> Lede-dev mailing list
> Lede-dev@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/lede-dev

___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev

Re: [LEDE-DEV] Recent issues w/ sysupgrade

[Philip Prindeville]

> On Sep 5, 2017, at 8:50 PM, Ryan Mounce  wrote:
> 
> On 31 August 2017 at 04:11, Philip Prindeville
>  wrote:
>> Looking a little further into the console logging, what I’m seeing after a 
>> “dd” directly onto the flash is this:
>> 
>> Press the [f] key and hit [enter] to enter failsafe mode
>> Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
>> [6.299598] mount_root: loading kmods from internal overlay
>> [6.307977] kmodloader: loading kernel modules from //etc/modules-boot.d/*
>> [6.316014] kmodloader: done loading kernel modules from 
>> //etc/modules-boot.d/*
>> [6.459868] block: attempting to load /etc/config/fstab
>> [6.472689] block: extroot: not configured
>> [6.488330] mount_root: rootdisk overlay filesystem has not been 
>> formatted yet
>> [6.517993] blk_update_request: I/O error, dev loop0, sector 0
>> [6.724898] block: attempting to load /etc/config/fstab
>> [6.730248] block: extroot: not configured
>> [6.734560] mount_root: overlay filesystem has not been fully initialized 
>> yet
>> [6.741823] mount_root: switching to f2fs overlay
>> 
>> 
>> if I take the exact same image and try to reinstall it with sysupgrade, I 
>> see this:
>> 
>> 
>> Press the [f] key and hit [enter] to enter failsafe mode
>> Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
>> [5.319594] mount_root: loading kmods from internal overlay
>> [5.328062] kmodloader: loading kernel modules from //etc/modules-boot.d/*
>> [5.336078] kmodloader: done loading kernel modules from 
>> //etc/modules-boot.d/*
>> [5.480712] block: attempting to load /etc/config/fstab
>> [5.493487] block: extroot: not configured
>> [5.498237] random: procd: uninitialized urandom read (4 bytes read)
>> [5.515173] mount_root: rootdisk overlay filesystem has not been 
>> formatted yet
>> 
>> 
>> it’s tough to reproduce, which makes me wonder if it’s either a race 
>> condition or an uninitialized variable.
> 
> I can consistently reproduce this issue when testing my Turris Omnia
> patch set, hopefully I can help to narrow it down.
> 
> A couple of times interrupting boot into failsafe and then running
> mount_root has fixed things, however more often than not mount_root
> hangs in the same way.
> 
> This appears to have been introduced in the last couple of months so I
> will try to bisect it.


Thanks.

It’s not 100% reproducible which makes it time-consuming to bisect because you 
have to repeat steps a few times.

But I did conclude decisively that it did NOT present before:

5eb216e mediatek: update to latest kernel patchset from v4.13-rc

So you don’t have to search any earlier than that.

It’s in the last 3 weeks.

-Philip


___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev

Re: [LEDE-DEV] Recent issues w/ sysupgrade

[Philip Prindeville]

Looking a little further into the console logging, what I’m seeing after a “dd” 
directly onto the flash is this:

Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
[6.299598] mount_root: loading kmods from internal overlay
[6.307977] kmodloader: loading kernel modules from //etc/modules-boot.d/*
[6.316014] kmodloader: done loading kernel modules from 
//etc/modules-boot.d/*
[6.459868] block: attempting to load /etc/config/fstab
[6.472689] block: extroot: not configured
[6.488330] mount_root: rootdisk overlay filesystem has not been formatted 
yet
[6.517993] blk_update_request: I/O error, dev loop0, sector 0
[6.724898] block: attempting to load /etc/config/fstab
[6.730248] block: extroot: not configured
[6.734560] mount_root: overlay filesystem has not been fully initialized yet
[6.741823] mount_root: switching to f2fs overlay


if I take the exact same image and try to reinstall it with sysupgrade, I see 
this:


Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
[5.319594] mount_root: loading kmods from internal overlay
[5.328062] kmodloader: loading kernel modules from //etc/modules-boot.d/*
[5.336078] kmodloader: done loading kernel modules from 
//etc/modules-boot.d/*
[5.480712] block: attempting to load /etc/config/fstab
[5.493487] block: extroot: not configured
[5.498237] random: procd: uninitialized urandom read (4 bytes read)
[5.515173] mount_root: rootdisk overlay filesystem has not been formatted 
yet


it’s tough to reproduce, which makes me wonder if it’s either a race condition 
or an uninitialized variable.

This was after I rebased to:

2f0a855 lantiq: ACMP252: clean up device modules

What would be significantly different about dd’ing an image onto the CF versus 
the copy that sysupgrade performs?

-Philip


___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev

Re: [LEDE-DEV] Recent issues w/ sysupgrade

[Philip Prindeville]

> On Aug 29, 2017, at 9:16 AM, Philip Prindeville 
>  wrote:
> 
>> On Aug 29, 2017, at 1:19 AM, Stijn Tintel  wrote:
>> 
>>> On 29-08-17 09:09, Philip Prindeville wrote:
>>> Hi all,
>>> 
>>> I don’t know if sysupgrade is the problem, or if this is where things 
>>> manifest.
>>> 
>>> But I recently (within the last week, but I only rebase once or twice a 
>>> week) started seeing issues with doing sysupgrade on x86_64 hardware.
>>> 
>>> The sysupgrade will appear to go okay, but then when the machine reboots, I 
>>> see:
>>> 
>>> ...
>>> Press the [f] key and hit [enter] to enter failsafe mode
>>> Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
>>> [6.322701] mount_root: loading kmods from internal overlay
>>> [6.331183] kmodloader: loading kernel modules from 
>>> //etc/modules-boot.d/*
>>> [6.339194] kmodloader: done loading kernel modules from 
>>> //etc/modules-boot.d/*
>>> [6.470650] block: attempting to load /etc/config/fstab
>>> [6.483431] block: extroot: not configured
>>> [6.488267] random: procd: uninitialized urandom read (4 bytes read)
>>> [6.506070] mount_root: rootdisk overlay filesystem has not been 
>>> formatted yet
>>> 
>>> 
>>> Is this related to the mk.f2f changes on Thursday?
>> I doubt this will be related. The only real change is
>> https://git.lede-project.org/cdb494fd, and on little endian systems
>> cpu_to_le64 is a noop. Additionaly I tested this on x86/64 and didn't
>> notice any isses. Just look a bit further and you should see this:
>> 
>> Tue Aug 29 07:12:57 2017 user.info kernel: [5.607837] mount_root:
>> overlay filesystem has not been fully initialized yet
>> Tue Aug 29 07:12:57 2017 user.info kernel: [5.608986] mount_root:
>> switching to f2fs overlay
>> 
>> root@LEDE:~# mount  | grep overlay
>> /dev/loop0 on /overlay type f2fs
>> (rw,lazytime,noatime,background_gc=on,user_xattr,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6)
>> overlayfs:/overlay on / type overlay
>> (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
>> 
>> Stijn
> 
> It never gets to that point.
> 
> The message about it not being formatted is the last output then it hangs.
> 


I do not see that (the “switching to f2fs overlay” message) on the console when 
the hang happens at reboot following a sysupgrade (and sometimes even just 
reflashing the CF card).

I have noted, when sysupgrade runs, I do see:


[  379.758026] EXT4-fs (sda1): mounted filesystem without journal. Opts: (null)
Upgrade completed
Rebooting system...
umount: can't unmount /dev: Resource busy
umount: can't unmount /tmp: Resource busy


but I was seeing those before.

So, some progress on the bi-section.

I can rebase to here:

af922c0 mediatek: update to latest kernel patchset from v4.13-rc
3ec259a procd: fix hotplug-preinit.json syntax
1a3b17a procd: fix hotplug.json syntax
a678eb0 ca-certificates: Update to 20170717
8e16e19 imx6: refresh kernel config
c0d5990 imx6: add driver for temp/voltage monitoring
2deed55 uboot-lantiq: Enable TFTP PUT support for backups
47dcfcf gpio-button-hotplug: leave platform_device.dev.platform_data untouched

but then I need to cherry-pick this fix so things will build:

dfee19f mediatek: drop kernel dep on userland module

If I try to rebase any further, to:

e622b30 netifd: update to latest git HEAD

then I see the flakiness at boot time.  John or Felix, are there any changes 
here that would affect booting if the way described above?

-Philip



___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev

Re: [LEDE-DEV] Recent issues w/ sysupgrade

[Philip Prindeville]

> On Aug 29, 2017, at 1:19 AM, Stijn Tintel  wrote:
> 
>> On 29-08-17 09:09, Philip Prindeville wrote:
>> Hi all,
>> 
>> I don’t know if sysupgrade is the problem, or if this is where things 
>> manifest.
>> 
>> But I recently (within the last week, but I only rebase once or twice a 
>> week) started seeing issues with doing sysupgrade on x86_64 hardware.
>> 
>> The sysupgrade will appear to go okay, but then when the machine reboots, I 
>> see:
>> 
>> ...
>> Press the [f] key and hit [enter] to enter failsafe mode
>> Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
>> [6.322701] mount_root: loading kmods from internal overlay
>> [6.331183] kmodloader: loading kernel modules from //etc/modules-boot.d/*
>> [6.339194] kmodloader: done loading kernel modules from 
>> //etc/modules-boot.d/*
>> [6.470650] block: attempting to load /etc/config/fstab
>> [6.483431] block: extroot: not configured
>> [6.488267] random: procd: uninitialized urandom read (4 bytes read)
>> [6.506070] mount_root: rootdisk overlay filesystem has not been 
>> formatted yet
>> 
>> 
>> Is this related to the mk.f2f changes on Thursday?
> I doubt this will be related. The only real change is
> https://git.lede-project.org/cdb494fd, and on little endian systems
> cpu_to_le64 is a noop. Additionaly I tested this on x86/64 and didn't
> notice any isses. Just look a bit further and you should see this:
> 
> Tue Aug 29 07:12:57 2017 user.info kernel: [5.607837] mount_root:
> overlay filesystem has not been fully initialized yet
> Tue Aug 29 07:12:57 2017 user.info kernel: [5.608986] mount_root:
> switching to f2fs overlay
> 
> root@LEDE:~# mount  | grep overlay
> /dev/loop0 on /overlay type f2fs
> (rw,lazytime,noatime,background_gc=on,user_xattr,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6)
> overlayfs:/overlay on / type overlay
> (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
> 
> Stijn

It never gets to that point.

The message about it not being formatted is the last output then it hangs.



___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev

Re: [LEDE-DEV] Recent issues w/ sysupgrade

[Stijn Tintel]

On 29-08-17 09:09, Philip Prindeville wrote:
> Hi all,
>
> I don’t know if sysupgrade is the problem, or if this is where things 
> manifest.
>
> But I recently (within the last week, but I only rebase once or twice a week) 
> started seeing issues with doing sysupgrade on x86_64 hardware.
>
> The sysupgrade will appear to go okay, but then when the machine reboots, I 
> see:
>
> ...
> Press the [f] key and hit [enter] to enter failsafe mode
> Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
> [6.322701] mount_root: loading kmods from internal overlay
> [6.331183] kmodloader: loading kernel modules from //etc/modules-boot.d/*
> [6.339194] kmodloader: done loading kernel modules from 
> //etc/modules-boot.d/*
> [6.470650] block: attempting to load /etc/config/fstab
> [6.483431] block: extroot: not configured
> [6.488267] random: procd: uninitialized urandom read (4 bytes read)
> [6.506070] mount_root: rootdisk overlay filesystem has not been formatted 
> yet
>
>
> Is this related to the mk.f2f changes on Thursday?
I doubt this will be related. The only real change is
https://git.lede-project.org/cdb494fd, and on little endian systems
cpu_to_le64 is a noop. Additionaly I tested this on x86/64 and didn't
notice any isses. Just look a bit further and you should see this:

Tue Aug 29 07:12:57 2017 user.info kernel: [    5.607837] mount_root:
overlay filesystem has not been fully initialized yet
Tue Aug 29 07:12:57 2017 user.info kernel: [    5.608986] mount_root:
switching to f2fs overlay

root@LEDE:~# mount  | grep overlay
/dev/loop0 on /overlay type f2fs
(rw,lazytime,noatime,background_gc=on,user_xattr,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6)
overlayfs:/overlay on / type overlay
(rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)

Stijn

___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev

[LEDE-DEV] Recent issues w/ sysupgrade

[Philip Prindeville]

Hi all,

I don’t know if sysupgrade is the problem, or if this is where things manifest.

But I recently (within the last week, but I only rebase once or twice a week) 
started seeing issues with doing sysupgrade on x86_64 hardware.

The sysupgrade will appear to go okay, but then when the machine reboots, I see:

...
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
[6.322701] mount_root: loading kmods from internal overlay
[6.331183] kmodloader: loading kernel modules from //etc/modules-boot.d/*
[6.339194] kmodloader: done loading kernel modules from 
//etc/modules-boot.d/*
[6.470650] block: attempting to load /etc/config/fstab
[6.483431] block: extroot: not configured
[6.488267] random: procd: uninitialized urandom read (4 bytes read)
[6.506070] mount_root: rootdisk overlay filesystem has not been formatted 
yet


Is this related to the mk.f2f changes on Thursday?

Or maybe the mkimage fix?

I’ll try resetting my cloning further and further back until I can’t reproduce 
the problem…

That might take me a day or so to isolate.

-Philip


___
Lede-dev mailing list
Lede-dev@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/lede-dev