[Bug 1925261] Re: memory leak on AWS kernels when using docker

2021-08-31 Thread Paul Friel
Tim, We are running Ubuntu 18.04 with the 5.3.0-1030-aws kernel because that is the last Ubuntu provided AMI (ubuntu- bionic-18.04-amd64-server-20200716) that does not contain this kernel bug. I tried installing the latest supported Ubuntu 18.04 kernel again yesterday (5.4.0-1055-aws) and

[Bug 1925261] Re: memory leak on AWS kernels when using docker

2021-08-25 Thread Paul Friel
Tim / Kleber, Thanks for your response on this and I apologize, you are correct that commit 7514c0362ffdd9af953ae94334018e7356b31313 was not the fix for our issue. I had previously just tested the last handful of commits in 5.9.0-rc4 and didn't realize that

[Bug 1925261] Re: memory leak on AWS kernels when using docker

2021-08-06 Thread Paul Friel
Kleber, Sounds good, thank you! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1925261 Title: memory leak on AWS kernels when using docker To manage notifications about this bug go to:

[Bug 1925261] Re: memory leak on AWS kernels when using docker

2021-07-27 Thread Paul Friel
Kleber, I finally had some time to narrow down what commit fixes this issue for us today, below is the commit: commit 7514c0362ffdd9af953ae94334018e7356b31313 Merge: 9322c47b21b9 428fc0aff4e5 Author: Linus Torvalds Date: Sat Sep 5 13:28:40 2020 -0700 Merge branch 'akpm' (patches from

[Bug 1925261] Re: memory leak on AWS kernels when using docker

2021-04-21 Thread Paul Friel
Kleber, I am not sure exactly which commit fixes the issue we are experiencing. I will put some time into bisecting the commits introduced in v5.9-rc4 and building/testing kernels with that code to see if I can narrow down the exact commit that introduced the fix. Thanks, Paul -- You received

[Bug 1925261] [NEW] memory leak on AWS kernels when using docker

2021-04-20 Thread Paul Friel
Public bug reported: Ever since the "ubuntu-bionic-18.04-amd64-server-20200729" EC2 Ubuntu AMI was released which has the "5.3.0-1032-aws" kernel we have been hitting a 100% repro memory leak that causes our app that is running under docker to be OOM killed. The scenario is that we have an app

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-28 Thread Paul Friel
@davidjmemmett I tested the new kernel on one c5.xl yesterday and it worked fine. Deployed the new kernel to all of our environments today and we are seeing intermittent repro of the same behavior we saw in the past (box fails to boot, no SSH available, CPU at 100%). We reverted to the 20180109

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-16 Thread Paul Friel
@kamalmostafa I installed that kernel package and it worked fine with sssd running on a c5.xl -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1746806 Title: sssd appears to crash AWS c5 and m5

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-07 Thread Paul Friel
@kamalmostafa: Do you all have a target date for when the new linux-aws kernel (4.4.0-1051.60) will be released? Thanks everyone for your help in quickly tracking down this issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-06 Thread Paul Friel
@kamalmostafa I installed the rtp0 kernel and verified it boots fine using c5.xl and m5.xl instances with apparmor & sssd enabled. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1746806 Title: sssd

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-06 Thread Paul Friel
@jsalisbury: I tested the kernel you provided above (commit 7de295e2a47849488acec80fc7c9973a4dca204e) and it boots fine on both a c5.xl and a m5.xl. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-05 Thread Paul Friel
@jsalisbury: I installed that kernel and rebooted using a c5.xl, it froze. I booted into a c4.xl and it booted fine, disabled the apparmor service and rebooted into a c5.xl and it booted fine. Re-enabled apparmor and rebooted into the c5.xl again and it froze on boot. -- You received this bug

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-02 Thread Paul Friel
@racb and @jjohansen I installed kernel 4.4.0-1050-aws, disabled sssd and apparmor on boot, and restarted on a c5 and it boots fine.. also boots fine just disabling sssd on boot. If I start sssd without apparmor running everything is fine. If I start apparmor first, then start sssd it freezes

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-01 Thread Paul Friel
I took a few minutes this afternoon and tried several different kernels, here is what I found: works fine: 4.4.0.1047.49 (packaged with AWS Ubuntu Cloud Image from 20180109) BROKEN: 4.4.0.1049.51 (installed with "apt update && apt install linux-aws=4.4.0.1049.51 linux-image-aws=4.4.0.1049.51

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-01 Thread Paul Friel
** Attachment added: "lsb_release_and_apt-cache-policy-sssd.txt" https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1746806/+attachment/5047062/+files/lsb_release_and_apt-cache-policy-sssd.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-01 Thread Paul Friel
** Attachment added: "kern.log" https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1746806/+attachment/5047061/+files/kern.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1746806 Title: sssd

[Bug 1746806] [NEW] sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-01 Thread Paul Friel
Public bug reported: After upgrading to the Ubuntu EC2 AMI from 20180126 (specifically ami-79873901 in us-west-2) we have seen sssd hard locking c5 and m5 EC2 instances after starting the service and CPU goes to 100%. We do not experience this issue with t2 or c4 instance types and we do not see

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-01 Thread Paul Friel
** Attachment added: "sssd_startup_from_shell.log" https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1746806/+attachment/5047064/+files/sssd_startup_from_shell.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-01 Thread Paul Friel
** Attachment added: "apport.sssd-common.apport" https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1746806/+attachment/5047060/+files/apport.sssd-common.apport -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1746806] Re: sssd appears to crash AWS c5 and m5 instances, cause 100% CPU

2018-02-01 Thread Paul Friel
** Attachment added: "syslog" https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1746806/+attachment/5047066/+files/syslog -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1746806 Title: sssd