Re: [lxc-users] lxd hang - "panic: runtime error: slice bounds out of range"
On Wed, Feb 01, 2017 at 07:13:25PM +0900, Tomasz Chmielewski wrote: > FYI it was crashing in the following conditions: > > - privileged container (i.e. the one for docker) > - on btrfs filesystem > - btrfs subvolume created inside the container (docker would create such > subvolumes) > - snapshot of the container made Yeah, shouldn't really matter since the whole code was removed and redone not to rely on parsing "btrfs subvolume list" :) > Will your fix eventually go to i.e. 2.0.9? 2.0.9 was released before I wrote that fix (should hit Ubuntu users in the next couple of weeks). The fix will be in 2.0.10 (I just pushed it to the stable-2.0 branch). > > > Tomasz > > > On 2017-02-01 19:02, Stéphane Graber wrote: > > Hey there, > > > > I wrote a fix for that function just yesterday which I think should fix > > your issue. It's been merged in git but isn't in any released version of > > LXD yet. > > > > Since you're using LXD 2.0.8, I made a build of 2.0.8 with that one fix > > applied at: https://dl.stgraber.org/lxd-2.0.8-btrfs > > SHA256: 4d9a7ef7c4635d7dd6c3e41f0eb1a3db12d38a8148b3940aa801c7355510e815 > > > > Stéphane > > > > On Wed, Feb 01, 2017 at 03:19:44PM +0900, Tomasz Chmielewski wrote: > > > Unfortunately it's still crashing, around 1 day after removing the > > > docker > > > container: > > > > > > Feb 1 06:16:20 lxd01 lxd[2593]: error: LXD still not running after > > > 600s > > > timeout. > > > Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Control process exited, > > > code=exited status=1 > > > Feb 1 06:16:20 lxd01 systemd[1]: Failed to start LXD - main daemon. > > > Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Unit entered failed > > > state. > > > Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Failed with result > > > 'exit-code'. > > > Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Service hold-off time > > > over, > > > scheduling restart. > > > Feb 1 06:16:20 lxd01 systemd[1]: Stopped LXD - main daemon. > > > Feb 1 06:16:20 lxd01 systemd[1]: Starting LXD - main daemon... > > > Feb 1 06:16:20 lxd01 lxd[29235]: t=2017-02-01T06:16:20+ lvl=warn > > > msg="CGroup memory swap accounting is disabled, swap limits will be > > > ignored." > > > Feb 1 06:16:20 lxd01 lxd[29235]: panic: runtime error: slice bounds > > > out of > > > range > > > Feb 1 06:16:20 lxd01 lxd[29235]: goroutine 16 [running]: > > > Feb 1 06:16:20 lxd01 lxd[29235]: panic(0xadef00, 0xc82000e050) > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > #011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6 > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > main.(*storageBtrfs).getSubVolumes(0xc8200fa240, 0xc82000b880, 0x3a, > > > 0x0, > > > 0x0, 0x0, 0x0, 0x0) > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812 > > > +0x104f > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > main.(*storageBtrfs).subvolsDelete(0xc8200fa240, 0xc82000b880, 0x3a, > > > 0x0, > > > 0x0) > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574 > > > +0x72 > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > main.(*storageBtrfs).ContainerDelete(0xc8200fa240, 0x7feeacffd0c0, > > > 0xc8204fa480, 0x0, 0x0) > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119 > > > +0xb0 > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200fa240, > > > 0x7feeacffd0c0, > > > 0xc8204fa480, 0x0, 0x0) > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321 > > > +0x5c > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60, > > > 0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0) > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510 > > > +0x22b > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > main.(*containerLXC).Delete(0xc8204fa480, > > > 0x0, 0x0) > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366 > > > +0x30e > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > main.snapshotDelete.func1(0xc8205400b0, > > > 0x0, 0x0) > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248 > > > +0x3e > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > main.(*operation).Run.func1(0xc8205400b0, > > > 0xc820258f60) > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110 > > > +0x3a > > > Feb 1 06:16:20 lxd01 lxd[29235]: created by main.(*operation).Run > > > Feb 1 06:16:20 lxd01 lxd[29235]: > > > #
Re: [lxc-users] lxd hang - "panic: runtime error: slice bounds out of range"
FYI it was crashing in the following conditions: - privileged container (i.e. the one for docker) - on btrfs filesystem - btrfs subvolume created inside the container (docker would create such subvolumes) - snapshot of the container made Will your fix eventually go to i.e. 2.0.9? Tomasz On 2017-02-01 19:02, Stéphane Graber wrote: Hey there, I wrote a fix for that function just yesterday which I think should fix your issue. It's been merged in git but isn't in any released version of LXD yet. Since you're using LXD 2.0.8, I made a build of 2.0.8 with that one fix applied at: https://dl.stgraber.org/lxd-2.0.8-btrfs SHA256: 4d9a7ef7c4635d7dd6c3e41f0eb1a3db12d38a8148b3940aa801c7355510e815 Stéphane On Wed, Feb 01, 2017 at 03:19:44PM +0900, Tomasz Chmielewski wrote: Unfortunately it's still crashing, around 1 day after removing the docker container: Feb 1 06:16:20 lxd01 lxd[2593]: error: LXD still not running after 600s timeout. Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Control process exited, code=exited status=1 Feb 1 06:16:20 lxd01 systemd[1]: Failed to start LXD - main daemon. Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Unit entered failed state. Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Failed with result 'exit-code'. Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Service hold-off time over, scheduling restart. Feb 1 06:16:20 lxd01 systemd[1]: Stopped LXD - main daemon. Feb 1 06:16:20 lxd01 systemd[1]: Starting LXD - main daemon... Feb 1 06:16:20 lxd01 lxd[29235]: t=2017-02-01T06:16:20+ lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." Feb 1 06:16:20 lxd01 lxd[29235]: panic: runtime error: slice bounds out of range Feb 1 06:16:20 lxd01 lxd[29235]: goroutine 16 [running]: Feb 1 06:16:20 lxd01 lxd[29235]: panic(0xadef00, 0xc82000e050) Feb 1 06:16:20 lxd01 lxd[29235]: #011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6 Feb 1 06:16:20 lxd01 lxd[29235]: main.(*storageBtrfs).getSubVolumes(0xc8200fa240, 0xc82000b880, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812 +0x104f Feb 1 06:16:20 lxd01 lxd[29235]: main.(*storageBtrfs).subvolsDelete(0xc8200fa240, 0xc82000b880, 0x3a, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574 +0x72 Feb 1 06:16:20 lxd01 lxd[29235]: main.(*storageBtrfs).ContainerDelete(0xc8200fa240, 0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119 +0xb0 Feb 1 06:16:20 lxd01 lxd[29235]: main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200fa240, 0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321 +0x5c Feb 1 06:16:20 lxd01 lxd[29235]: main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60, 0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510 +0x22b Feb 1 06:16:20 lxd01 lxd[29235]: main.(*containerLXC).Delete(0xc8204fa480, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366 +0x30e Feb 1 06:16:20 lxd01 lxd[29235]: main.snapshotDelete.func1(0xc8205400b0, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248 +0x3e Feb 1 06:16:20 lxd01 lxd[29235]: main.(*operation).Run.func1(0xc8205400b0, 0xc820258f60) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110 +0x3a Feb 1 06:16:20 lxd01 lxd[29235]: created by main.(*operation).Run Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137 +0x127 Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Any clues what's causing this and how to fix? Tomasz On 2017-01-31 16:29, Tomasz Chmielewski wrote: > I think it may be related to > https://www.stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/ > > I have a docker container, with several dockers inside, and with lxd > snapshots. > > Doing this: > > # lxc delete docker > error: Get > http://unix.socket/1.0/operations/7d30bf41-3af6-4b48-b42c-06fdd2bba48b/wait: > EOF > > Results in: > > Jan 31 07:28:36 lxd01 lxd[8363]: panic: runtime error: slice bounds out > of range > Jan 31 07:28:36 lxd01 lxd[8363]: goroutine 867 [running]: > Jan 31 07:28:36 lxd01 lxd[8363]: panic(0xadef00, 0xc82000e050) > Jan 31 07:28:36 lxd01 lxd[8363]: > #
Re: [lxc-users] lxd hang - "panic: runtime error: slice bounds out of range"
Hey there, I wrote a fix for that function just yesterday which I think should fix your issue. It's been merged in git but isn't in any released version of LXD yet. Since you're using LXD 2.0.8, I made a build of 2.0.8 with that one fix applied at: https://dl.stgraber.org/lxd-2.0.8-btrfs SHA256: 4d9a7ef7c4635d7dd6c3e41f0eb1a3db12d38a8148b3940aa801c7355510e815 Stéphane On Wed, Feb 01, 2017 at 03:19:44PM +0900, Tomasz Chmielewski wrote: > Unfortunately it's still crashing, around 1 day after removing the docker > container: > > Feb 1 06:16:20 lxd01 lxd[2593]: error: LXD still not running after 600s > timeout. > Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Control process exited, > code=exited status=1 > Feb 1 06:16:20 lxd01 systemd[1]: Failed to start LXD - main daemon. > Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Unit entered failed state. > Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Failed with result > 'exit-code'. > Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Service hold-off time over, > scheduling restart. > Feb 1 06:16:20 lxd01 systemd[1]: Stopped LXD - main daemon. > Feb 1 06:16:20 lxd01 systemd[1]: Starting LXD - main daemon... > Feb 1 06:16:20 lxd01 lxd[29235]: t=2017-02-01T06:16:20+ lvl=warn > msg="CGroup memory swap accounting is disabled, swap limits will be > ignored." > Feb 1 06:16:20 lxd01 lxd[29235]: panic: runtime error: slice bounds out of > range > Feb 1 06:16:20 lxd01 lxd[29235]: goroutine 16 [running]: > Feb 1 06:16:20 lxd01 lxd[29235]: panic(0xadef00, 0xc82000e050) > Feb 1 06:16:20 lxd01 lxd[29235]: > #011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6 > Feb 1 06:16:20 lxd01 lxd[29235]: > main.(*storageBtrfs).getSubVolumes(0xc8200fa240, 0xc82000b880, 0x3a, 0x0, > 0x0, 0x0, 0x0, 0x0) > Feb 1 06:16:20 lxd01 lxd[29235]: > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812 > +0x104f > Feb 1 06:16:20 lxd01 lxd[29235]: > main.(*storageBtrfs).subvolsDelete(0xc8200fa240, 0xc82000b880, 0x3a, 0x0, > 0x0) > Feb 1 06:16:20 lxd01 lxd[29235]: > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574 > +0x72 > Feb 1 06:16:20 lxd01 lxd[29235]: > main.(*storageBtrfs).ContainerDelete(0xc8200fa240, 0x7feeacffd0c0, > 0xc8204fa480, 0x0, 0x0) > Feb 1 06:16:20 lxd01 lxd[29235]: > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119 > +0xb0 > Feb 1 06:16:20 lxd01 lxd[29235]: > main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200fa240, 0x7feeacffd0c0, > 0xc8204fa480, 0x0, 0x0) > Feb 1 06:16:20 lxd01 lxd[29235]: > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321 > +0x5c > Feb 1 06:16:20 lxd01 lxd[29235]: > main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60, > 0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0) > Feb 1 06:16:20 lxd01 lxd[29235]: > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510 > +0x22b > Feb 1 06:16:20 lxd01 lxd[29235]: main.(*containerLXC).Delete(0xc8204fa480, > 0x0, 0x0) > Feb 1 06:16:20 lxd01 lxd[29235]: > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366 > +0x30e > Feb 1 06:16:20 lxd01 lxd[29235]: main.snapshotDelete.func1(0xc8205400b0, > 0x0, 0x0) > Feb 1 06:16:20 lxd01 lxd[29235]: > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248 > +0x3e > Feb 1 06:16:20 lxd01 lxd[29235]: main.(*operation).Run.func1(0xc8205400b0, > 0xc820258f60) > Feb 1 06:16:20 lxd01 lxd[29235]: > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110 > +0x3a > Feb 1 06:16:20 lxd01 lxd[29235]: created by main.(*operation).Run > Feb 1 06:16:20 lxd01 lxd[29235]: > #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137 > +0x127 > Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Main process exited, > code=exited, status=2/INVALIDARGUMENT > > > Any clues what's causing this and how to fix? > > > Tomasz > > On 2017-01-31 16:29, Tomasz Chmielewski wrote: > > I think it may be related to > > https://www.stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/ > > > > I have a docker container, with several dockers inside, and with lxd > > snapshots. > > > > Doing this: > > > > # lxc delete docker > > error: Get > > http://unix.socket/1.0/operations/7d30bf41-3af6-4b48-b42c-06fdd2bba48b/wait: > > EOF > > > > Results in: > > > > Jan 31 07:28:36 lxd01 lxd[8363]: panic: runtime error: slice bounds out > > of range > > Jan 31 07:28:36 lxd01 lxd[8363]: goroutine 867 [running]: > > Jan 31 07:28:36 lxd01 lxd[8363]: panic(0xadef00, 0xc82000e050) > > Jan 31 07:28:36 lxd01 lxd[8363]: > > #011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6 > > Jan 31 07:28:36 lxd01 lxd[8363]: > > main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc82000bb80, 0x3a, > >
Re: [lxc-users] lxd hang - "panic: runtime error: slice bounds out of range"
Unfortunately it's still crashing, around 1 day after removing the docker container: Feb 1 06:16:20 lxd01 lxd[2593]: error: LXD still not running after 600s timeout. Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Control process exited, code=exited status=1 Feb 1 06:16:20 lxd01 systemd[1]: Failed to start LXD - main daemon. Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Unit entered failed state. Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Failed with result 'exit-code'. Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Service hold-off time over, scheduling restart. Feb 1 06:16:20 lxd01 systemd[1]: Stopped LXD - main daemon. Feb 1 06:16:20 lxd01 systemd[1]: Starting LXD - main daemon... Feb 1 06:16:20 lxd01 lxd[29235]: t=2017-02-01T06:16:20+ lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." Feb 1 06:16:20 lxd01 lxd[29235]: panic: runtime error: slice bounds out of range Feb 1 06:16:20 lxd01 lxd[29235]: goroutine 16 [running]: Feb 1 06:16:20 lxd01 lxd[29235]: panic(0xadef00, 0xc82000e050) Feb 1 06:16:20 lxd01 lxd[29235]: #011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6 Feb 1 06:16:20 lxd01 lxd[29235]: main.(*storageBtrfs).getSubVolumes(0xc8200fa240, 0xc82000b880, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812 +0x104f Feb 1 06:16:20 lxd01 lxd[29235]: main.(*storageBtrfs).subvolsDelete(0xc8200fa240, 0xc82000b880, 0x3a, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574 +0x72 Feb 1 06:16:20 lxd01 lxd[29235]: main.(*storageBtrfs).ContainerDelete(0xc8200fa240, 0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119 +0xb0 Feb 1 06:16:20 lxd01 lxd[29235]: main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200fa240, 0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321 +0x5c Feb 1 06:16:20 lxd01 lxd[29235]: main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60, 0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510 +0x22b Feb 1 06:16:20 lxd01 lxd[29235]: main.(*containerLXC).Delete(0xc8204fa480, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366 +0x30e Feb 1 06:16:20 lxd01 lxd[29235]: main.snapshotDelete.func1(0xc8205400b0, 0x0, 0x0) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248 +0x3e Feb 1 06:16:20 lxd01 lxd[29235]: main.(*operation).Run.func1(0xc8205400b0, 0xc820258f60) Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110 +0x3a Feb 1 06:16:20 lxd01 lxd[29235]: created by main.(*operation).Run Feb 1 06:16:20 lxd01 lxd[29235]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137 +0x127 Feb 1 06:16:20 lxd01 systemd[1]: lxd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Any clues what's causing this and how to fix? Tomasz On 2017-01-31 16:29, Tomasz Chmielewski wrote: I think it may be related to https://www.stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/ I have a docker container, with several dockers inside, and with lxd snapshots. Doing this: # lxc delete docker error: Get http://unix.socket/1.0/operations/7d30bf41-3af6-4b48-b42c-06fdd2bba48b/wait: EOF Results in: Jan 31 07:28:36 lxd01 lxd[8363]: panic: runtime error: slice bounds out of range Jan 31 07:28:36 lxd01 lxd[8363]: goroutine 867 [running]: Jan 31 07:28:36 lxd01 lxd[8363]: panic(0xadef00, 0xc82000e050) Jan 31 07:28:36 lxd01 lxd[8363]: #011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6 Jan 31 07:28:36 lxd01 lxd[8363]: main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc82000bb80, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x0) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812 +0x104f Jan 31 07:28:36 lxd01 lxd[8363]: main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc82000bb80, 0x3a, 0x0, 0x0) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574 +0x72 Jan 31 07:28:36 lxd01 lxd[8363]: main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/stora
Re: [lxc-users] lxd hang - "panic: runtime error: slice bounds out of range"
I think it may be related to https://www.stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/ I have a docker container, with several dockers inside, and with lxd snapshots. Doing this: # lxc delete docker error: Get http://unix.socket/1.0/operations/7d30bf41-3af6-4b48-b42c-06fdd2bba48b/wait: EOF Results in: Jan 31 07:28:36 lxd01 lxd[8363]: panic: runtime error: slice bounds out of range Jan 31 07:28:36 lxd01 lxd[8363]: goroutine 867 [running]: Jan 31 07:28:36 lxd01 lxd[8363]: panic(0xadef00, 0xc82000e050) Jan 31 07:28:36 lxd01 lxd[8363]: #011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6 Jan 31 07:28:36 lxd01 lxd[8363]: main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc82000bb80, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x0) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812 +0x104f Jan 31 07:28:36 lxd01 lxd[8363]: main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc82000bb80, 0x3a, 0x0, 0x0) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574 +0x72 Jan 31 07:28:36 lxd01 lxd[8363]: main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119 +0xb0 Jan 31 07:28:36 lxd01 lxd[8363]: main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240, 0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321 +0x5c Jan 31 07:28:36 lxd01 lxd[8363]: main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60, 0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510 +0x22b Jan 31 07:28:36 lxd01 lxd[8363]: main.(*containerLXC).Delete(0xc8204a0480, 0x0, 0x0) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366 +0x30e Jan 31 07:28:36 lxd01 lxd[8363]: main.containerDeleteSnapshots(0xc820090a00, 0xc82017c017, 0x6, 0x0, 0x0) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/containers.go:208 +0x4ac Jan 31 07:28:36 lxd01 lxd[8363]: main.(*containerLXC).Delete(0xc8204a00c0, 0x0, 0x0) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2371 +0x696 Jan 31 07:28:36 lxd01 lxd[8363]: main.containerDelete.func1(0xc8200c8840, 0x0, 0x0) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_delete.go:22 +0x3e Jan 31 07:28:36 lxd01 lxd[8363]: main.(*operation).Run.func1(0xc8200c8840, 0xc82011f320) Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110 +0x3a Jan 31 07:28:36 lxd01 lxd[8363]: created by main.(*operation).Run Jan 31 07:28:36 lxd01 lxd[8363]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137 +0x127 Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Unit entered failed state. Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Failed with result 'exit-code'. Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Service hold-off time over, scheduling restart. Jan 31 07:28:36 lxd01 systemd[1]: Stopped LXD - main daemon. Jan 31 07:28:36 lxd01 systemd[1]: Starting LXD - main daemon... Jan 31 07:28:36 lxd01 lxd[11969]: t=2017-01-31T07:28:36+ lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." Jan 31 07:28:37 lxd01 systemd[1]: Started LXD - main daemon. Tomasz On 2017-01-31 16:17, Tomasz Chmielewski wrote: lxd process on one of my servers started to hang a few days ago. In syslog, I can see the following being repeated (log below). I see a similar issue here: https://github.com/lxc/lxd/issues/2089 but it's closed. Running: ii lxd 2.0.8-0ubuntu1~ubuntu16.04.2 amd64Container hypervisor based on LXC - daemon ii lxd-client 2.0.8-0ubuntu1~ubuntu16.04.2 amd64Container hypervisor based on LXC - client On Ubuntu 16.04 with 4.9.0 kernel from ubuntu ppa. It seems to recover if I kill this process: root 26853 0.0 0.0 222164 12228 ?Ssl 07:13 0:00 /usr/bin/lxd waitready --timeout=600 Sometimes need to kill it a few times until it recovers. Jan 31 06:26:06 lxd01 lxd[15762]: error: LXD still not running after 600s timeout. Jan 31 06:26:06 lxd01 lxd[4402]: t=2017-01-31T06:26:06+ lvl=warn msg="CGroup memory swap
[lxc-users] lxd hang - "panic: runtime error: slice bounds out of range"
lxd process on one of my servers started to hang a few days ago. In syslog, I can see the following being repeated (log below). I see a similar issue here: https://github.com/lxc/lxd/issues/2089 but it's closed. Running: ii lxd 2.0.8-0ubuntu1~ubuntu16.04.2 amd64Container hypervisor based on LXC - daemon ii lxd-client 2.0.8-0ubuntu1~ubuntu16.04.2 amd64Container hypervisor based on LXC - client On Ubuntu 16.04 with 4.9.0 kernel from ubuntu ppa. It seems to recover if I kill this process: root 26853 0.0 0.0 222164 12228 ?Ssl 07:13 0:00 /usr/bin/lxd waitready --timeout=600 Sometimes need to kill it a few times until it recovers. Jan 31 06:26:06 lxd01 lxd[15762]: error: LXD still not running after 600s timeout. Jan 31 06:26:06 lxd01 lxd[4402]: t=2017-01-31T06:26:06+ lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." Jan 31 06:26:06 lxd01 lxd[4402]: panic: runtime error: slice bounds out of range Jan 31 06:26:06 lxd01 lxd[4402]: goroutine 27 [running]: Jan 31 06:26:06 lxd01 lxd[4402]: panic(0xadef00, 0xc82000e050) Jan 31 06:26:06 lxd01 lxd[4402]: #011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6 Jan 31 06:26:06 lxd01 lxd[4402]: main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc8201b22c0, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x0) Jan 31 06:26:06 lxd01 lxd[4402]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812 +0x104f Jan 31 06:26:06 lxd01 lxd[4402]: main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc8201b22c0, 0x3a, 0x0, 0x0) Jan 31 06:26:06 lxd01 lxd[4402]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574 +0x72 Jan 31 06:26:06 lxd01 lxd[4402]: main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0) Jan 31 06:26:06 lxd01 lxd[4402]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119 +0xb0 Jan 31 06:26:06 lxd01 lxd[4402]: main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240, 0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0) Jan 31 06:26:06 lxd01 lxd[4402]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321 +0x5c Jan 31 06:26:06 lxd01 lxd[4402]: main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60, 0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0) Jan 31 06:26:06 lxd01 lxd[4402]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510 +0x22b Jan 31 06:26:06 lxd01 lxd[4402]: main.(*containerLXC).Delete(0xc8200cc540, 0x0, 0x0) Jan 31 06:26:06 lxd01 lxd[4402]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366 +0x30e Jan 31 06:26:06 lxd01 lxd[4402]: main.snapshotDelete.func1(0xc8200c8000, 0x0, 0x0) Jan 31 06:26:06 lxd01 lxd[4402]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248 +0x3e Jan 31 06:26:06 lxd01 lxd[4402]: main.(*operation).Run.func1(0xc8200c8000, 0xc8200107e0) Jan 31 06:26:06 lxd01 lxd[4402]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110 +0x3a Jan 31 06:26:06 lxd01 lxd[4402]: created by main.(*operation).Run Jan 31 06:26:06 lxd01 lxd[4402]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137 +0x127 Jan 31 06:36:06 lxd01 lxd[4408]: error: LXD still not running after 600s timeout. Jan 31 06:36:06 lxd01 lxd[21952]: t=2017-01-31T06:36:06+ lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." Jan 31 06:36:06 lxd01 lxd[21952]: panic: runtime error: slice bounds out of range Jan 31 06:36:06 lxd01 lxd[21952]: goroutine 16 [running]: Jan 31 06:36:06 lxd01 lxd[21952]: panic(0xadef00, 0xc82000e050) Jan 31 06:36:06 lxd01 lxd[21952]: #011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6 Jan 31 06:36:06 lxd01 lxd[21952]: main.(*storageBtrfs).getSubVolumes(0xc820104240, 0xc8201a0140, 0x3a, 0x0, 0x0, 0x0, 0x0, 0x0) Jan 31 06:36:06 lxd01 lxd[21952]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812 +0x104f Jan 31 06:36:06 lxd01 lxd[21952]: main.(*storageBtrfs).subvolsDelete(0xc820104240, 0xc8201a0140, 0x3a, 0x0, 0x0) Jan 31 06:36:06 lxd01 lxd[21952]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574 +0x72 Jan 31 06:36:06 lxd01 lxd[21952]: main.(*storageBtrfs).ContainerDelete(0xc820104240, 0x7f0acd705938, 0xc8204c8180, 0x0, 0x0) Jan 31 06:36:06 lxd01 lxd[21952]: #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119 +0xb0 Jan 31 06:36:06 lxd01 lxd[21952]: main.(*storageBtrfs).ContainerSnapshotDelete(0xc820104240, 0x7f0acd705938, 0xc8204c8180, 0x0, 0x0) Jan 31 06:36:0
