[JIRA] [winstone-jetty] (JENKINS-33412) Jenkins locks when started in HTTPS mode on a host with 37+ processors
Title: Message Title Elijah Zupancic commented on JENKINS-33412 Re: Jenkins locks when started in HTTPS mode on a host with 37+ processors Thanks for your prompt action. That fix looks like it will solve the problem but it makes me sad. It is doing math to determine the thread count by computing the number of cores available. In the container world, we are seeing things like CPU shares or fair-share scheduling algorithms used to dice up CPU while exposing all of the cores to the OS. This leads to a bunch of weird performance problems especially if CPU to thread affinity is set. This way of configuring applications is not going to be sustainable long-term based on where OS containerization is heading. I'll add a feature request after this issue is fixed to allow for the manual configuration of the number of threads for acceptors, selectors, etc. Once again - thanks for your work. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [winstone-jetty] (JENKINS-33412) Jenkins locks when started in HTTPS mode on a host with 37+ processors
Title: Message Title
Elijah Zupancic edited a comment on JENKINS-33412
Re: Jenkins locks when started in HTTPS mode on a host with 37+ processors
[~danielbeck] You are completely correct. I did the bulk of my testing with winstone-2.9 and I lightly tested with winstone-3.1. I made a mistake with my network setup. I just validated that winstone-3.1 also works correctly and updated the bug to that effect.I would still recommend upgrading the next version of Jenkins to winstone-3.1 in order to fix this bug because this seems like one of those messy things that if you have a fix by upgrading a core library - it is just better doing it that way.That said, read below if we want to go down the root cause analysis route: I work at Joyent and if any developers want environment in which this can be reproduced, please email me your public key (elijah.zupan...@joyent.com) and I will create an instance.[~ydubreuil] The behavior is present across different operating systems including one's that do not use glibc as part of their JVM implementation (e.g. SmartOS). When I was inspecting the application with a debugger, the best that I could tell was that even though a request had come in, EPOLL would not wake and a processing thread would not be dispatched. We see that winstone recieves the request (when FINE logging is enabled):{code:none}FINE: created SCEP@3c108ef8{l(/97.113.3.1:54175)<->r(/165.225.168.215:8443),s=0,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=0}-{SslConnection@56a764e3 SSL NOT_HANDSHAKING i/o/u=-1/-1/-1 ishut=false oshut=false {AsyncHttpConnection@23fe1e75,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0}}{code}However, the process just continues to sleep after this.Here's the output from /proc/PID/status when it is in a hung state:{code:none}Name: javaState: S (sleeping)Tgid: 1673Ngid: 0Pid:1673PPid: 1672TracerPid: 0Uid:1000100010001000Gid:1000100010001000FDSize: 256Groups: 4 24 27 30 46 110 111 1000 NStgid: 1673NSpid: 1673NSpgid: 1672NSsid: 1551VmPeak: 8983212 kBVmSize: 8983208 kBVmLck: 0 kBVmPin: 0 kBVmHWM:199776 kBVmRSS:199276 kBVmData: 8922448 kBVmStk: 136 kBVmExe: 4 kBVmLib: 17216 kBVmPTE: 1080 kBVmPMD:48 kBVmSwap:0 kBThreads:98SigQ: 0/15699SigPnd: ShdPnd: NSsid: 1551VmPeak: 8983212 kBVmSize: 8983208 kBVmLck: 0 kBVmPin: 0 kBVmHWM:199776 kBVmRSS:199276 kBVmData: 8922448 kBVmStk: 136 kBVmExe: 4 kBVmLib: 17216 kBVmPTE: 1080 kBVmPMD:48 kBVmSwap:0 kBThreads:98SigQ: 0/15699SigPnd: ShdPnd: SigBlk: SigIgn: SigCgt: 200181005ccfCapInh: CapPrm: CapEff: CapBnd: 003fSeccomp:0Cpus_allowed: ,Cpus_allowed_list: 0-47Mems_allowed: ,0001Mems_allowed_list: 0voluntary_ctxt_switches:3nonvoluntary_ctxt_switches: 1{code :none }
Add Comment
[JIRA] [winstone-jetty] (JENKINS-33412) Jenkins locks when started in HTTPS mode on a host with 37+ processors
Title: Message Title
Elijah Zupancic commented on JENKINS-33412
Re: Jenkins locks when started in HTTPS mode on a host with 37+ processors
Daniel Beck You are completely correct. I did the bulk of my testing with winstone-2.9 and I lightly tested with winstone-3.1. I made a mistake with my network setup. I just validated that winstone-3.1 also works correctly and updated the bug to that effect.
I would still recommend upgrading the next version of Jenkins to winstone-3.1 in order to fix this bug because this seems like one of those messy things that if you have a fix by upgrading a core library - it is just better doing it that way.
That said, read below if we want to go down the root cause analysis route:
I work at Joyent and if any developers want environment in which this can be reproduced, please email me your public key (elijah.zupan...@joyent.com) and I will create an instance.
Yoann Dubreuil The behavior is present across different operating systems including one's that do not use glibc as part of their JVM implementation (e.g. SmartOS). When I was inspecting the application with a debugger, the best that I could tell was that even though a request had come in, EPOLL would not wake and a processing thread would not be dispatched. We see that winstone recieves the request (when FINE logging is enabled):
FINE: created SCEP@3c108ef8{l(/97.113.3.1:54175)<->r(/165.225.168.215:8443),s=0,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=0}-{SslConnection@56a764e3 SSL NOT_HANDSHAKING i/o/u=-1/-1/-1 ishut=false oshut=false {AsyncHttpConnection@23fe1e75,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0}}
However, the process just continues to sleep after this.
Here's the output from /proc/PID/status when it is in a hung state:
Name: java State: S (sleeping) Tgid: 1673 Ngid: 0 Pid: 1673 PPid: 1672 TracerPid: 0 Uid: 1000 1000 1000 1000 Gid: 1000 1000 1000 1000 FDSize: 256 Groups: 4 24 27 30 46 110 111 1000 NStgid: 1673 NSpid: 1673 NSpgid: 1672 NSsid: 1551 VmPeak: 8983212 kB VmSize: 8983208 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 199776 kB VmRSS: 199276 kB VmData: 8922448 kB VmStk: 136 kB VmExe: 4 kB VmLib: 17216 kB VmPTE: 1080 kB VmPMD: 48 kB VmSwap: 0 kB Threads: 98 SigQ: 0/15699 SigPnd: ShdPnd: NSsid: 1551 VmPeak: 8983212 kB VmSize: 8983208 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 199776 kB VmRSS: 199276 kB VmData: 8922448 kB VmStk: 136 kB VmExe: 4 kB VmLib: 17216 kB VmPTE: 1080 kB VmPMD: 48 kB VmSwap: 0 kB Threads: 98 SigQ: 0/15699 SigPnd: ShdPnd: SigBlk: SigIgn: SigCgt: 200181005ccf CapInh: CapPrm: CapEff: CapBnd: 003f Seccomp: 0 Cpus_allowed: , Cpus_allowed_list: 0-47 Mems_allowed: ,0001 Mems_allowed_list: 0 voluntary_ctxt_switches: 3 nonvoluntary_ctxt_switches: 1
[JIRA] [winstone-jetty] (JENKINS-33412) Jenkins locks when started in HTTPS mode on a host with 37+ processors
Title: Message Title
Elijah Zupancic updated an issue
Jenkins / JENKINS-33412
Jenkins locks when started in HTTPS mode on a host with 37+ processors
Change By:
Elijah Zupancic
*Summary*Using Winstone 3 2 . 1 9 (i.e. the embedded Jetty wrapper) or below it will not run in HTTPS mode on hosts with 37 cores/processors or more. This problem replicates regardless of the JDK or the operating system.*Reproduction*The easiest way to reproduce the error is to use qemu to virtualize a 37 core system. You can do that with the -smp parameter. For example, for testing I run:{code:none}qemu-system-x86_64 -hda ubuntu.img -m 4096 -smp 48{code}Once you have a VM with more than 37 cores setup, install Jenkins 1.652 and configure it to use HTTPS. Attempt to start it and connect to either the HTTP or the HTTPS port. The connection will time out for either port with the server effectively locked until you send it a SIGTERM. Please refer to the attached log file to see its start process.*Why is this important?*You may ask - who is running Jenkins on that big of a server? Well, with containerization technologies (e.g. Docker) taking a center stage, we are seeing more and more deployments where there is no VM involved and hence a container gets a slice of CPU but has visibility to all of the processors on a system. The official Docker image of Jenkins suffers from this defect. Sure a user can set up their own reverse proxy to run TLS through, but it adds unneeded complexity for users looking to containerize their Jenkins environment.*Solution*I've done the work of reducing the surface area of root cause analysis. I removed the entire jenkins war from the jenkins winstone runner (https://github.com/jenkinsci/winstone) and ran a simple hello world war instead. With HTTPS enabled, the issue still reproduced. It took a lot of fiddling to determine that it was exactly at 37 cores in which the hang occurred.Lastly, I tried the exact same reproduction steps with winstone-3. 2-SNAPSHOT 1 . Luckily, with the upgrade to embedded Jetty in the 3. 2 1 version the issue is resolved._Can we upgrade the next Jenkins release to use the winstone-3. 2 1 component?_This would be the easiest and the best fix. I would be happy to contribute to any efforts that would allow for us to get this into a release.
Add Comment
[JIRA] [winstone-jetty] (JENKINS-33412) Jenkins locks when started in HTTPS mode on a host with 37+ processors
Title: Message Title Elijah Zupancic updated an issue Jenkins / JENKINS-33412 Jenkins locks when started in HTTPS mode on a host with 37+ processors Change By: Elijah Zupancic Environment: Jenkins 1.652org.jenkins-ci:winstone 2.9 , 3.0, 3.1 Testing in the Linux JDK 1.7 and 1.8 as well as the Solaris JDK 1.7 1.8 (both OpenJDK and OracleJDK). Reproduces in Ubuntu, Debian, CentOS and SmartOS. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [winstone-jetty] (JENKINS-33412) Jenkins locks when started in HTTPS mode on a host with 37+ processors
Title: Message Title
Elijah Zupancic updated an issue
Jenkins / JENKINS-33412
Jenkins locks when started in HTTPS mode on a host with 37+ processors
Change By:
Elijah Zupancic
*Summary*Using Winstone Jetty 3.1 (i.e. the embedded Jetty wrapper) or below it will not run in HTTPS mode on hosts with 37 cores/processors or more. This problem replicates regardless of the JDK or the operating system.*Reproduction*The easiest way to reproduce the error is to use qemu to virtualize a 37 core system. You can do that with the -smp parameter. For example, for testing I run:{code:none}qemu-system-x86_64 -hda ubuntu.img -m 4096 -smp 48{code}Once you have a VM with more than 37 cores setup, install Jenkins 1.652 and configure it to use HTTPS. Attempt to start it and connect to either the HTTP or the HTTPS port. The connection will time out for either port with the server effectively locked until you send it a SIGTERM. Please refer to the attached log file to see its start process.*Why is this important?*You may ask - who is running Jenkins on that big of a server? Well, with containerization technologies (e.g. Docker) taking a center stage, we are seeing more and more deployments where there is no VM involved and hence a container gets a slice of CPU but has visibility to all of the processors on a system. The official Docker image of Jenkins suffers from this defect. Sure a user can set up their own reverse proxy to run TLS through, but it adds unneeded complexity for users looking to containerize their Jenkins environment.*Solution*I've done the work of reducing the surface area of root cause analysis. I removed the entire jenkins war from the jenkins winstone runner (https://github.com/jenkinsci/winstone) and ran a simple hello world war instead. With HTTPS enabled, the issue still reproduced. It took a lot of fiddling to determine that it was exactly at 37 cores in which the hang occurred.Lastly, I tried the exact same reproduction steps with winstone-3.2-SNAPSHOT. Luckily, with the upgrade to embedded Jetty in the 3.2 version the issue is resolved._Can we upgrade the next Jenkins release to use the winstone-3.2 component?_This would be the easiest and the best fix. I would be happy to contribute to any efforts that would allow for us to get this into a release.
Add Comment
[JIRA] [winstone-jetty] (JENKINS-33412) Jenkins locks when started in HTTPS mode on a host with 37+ processors
Title: Message Title
Elijah Zupancic updated an issue
Jenkins / JENKINS-33412
Jenkins locks when started in HTTPS mode on a host with 37+ processors
Change By:
Elijah Zupancic
*Summary*Using Winstone Jetty 3.1 or below it will not run in HTTPS mode on hosts with 37 cores/processors or more. This problem replicates regardless of the JDK or the operating system.*Reproduction*The easiest way to reproduce the error is to use qemu to virtualize a 37 core system. You can do that with the -smp parameter. For example, for testing I run:{code:none}qemu-system-x86_64 -hda ubuntu.img -m 4096 -smp 48{code}Once you have a VM with more than 37 cores setup, install Jenkins 1.652 and configure it to use HTTPS. Attempt to start it and connect to either the HTTP or the HTTPS port. The connection will time out for either port with the server effectively locked until you send it a SIGTERM. Please refer to the attached log file to see its start process.*Why is this important?*You may ask - who is running Jenkins on that big of a server? Well, with containerization technologies (e.g. Docker) taking a center stage, we are seeing more and more deployments where there is no VM involved and hence a container gets a slice of CPU but has visibility to all of the processors on a system. The official Docker image of Jenkins suffers from this defect. Sure a user can set up their own reverse proxy to run TLS through, but it adds unneeded complexity for users looking to containerize their Jenkins environment.*Solution*I've done the work of reducing the surface area of root cause analysis. I removed the entire jenkins war from the jenkins winstone runner (https://github.com/jenkinsci/winstone) and ran a simple hello world war instead. With HTTPS enabled, the issue still reproduced. It took a lot of fiddling to determine that it was exactly at 37 cores in which the hang occurred.Lastly, I tried the exact same reproduction steps with winstone-3.2-SNAPSHOT. Luckily, with the upgrade to embedded Jetty in the 3.2 version the issue is resolved. _*Can _Can we upgrade the next Jenkins release to use the winstone-3.2 component? * _This would be the easiest and the best fix. I would be happy to contribute to any efforts that would allow for us to get this into a release.
Add Comment
[JIRA] [winstone-jetty] (JENKINS-33412) Jenkins locks when started in HTTPS mode on a host with 37+ processors
Title: Message Title Elijah Zupancic created an issue Jenkins / JENKINS-33412 Jenkins locks when started in HTTPS mode on a host with 37+ processors Issue Type: Bug Assignee: Unassigned Attachments: jenkins_hanging.log Components: winstone-jetty Created: 08/Mar/16 9:28 PM Environment: Jenkins 1.652 org.jenkins-ci:winstone 2.9, 3.0, 3.1 Testing in the Linux JDK 1.7 and 1.8 as well as the Solaris JDK 1.7 1.8 (both OpenJDK and OracleJDK). Reproduces in Ubuntu, Debian, CentOS and SmartOS. Labels: winstone jenkins Priority: Major Reporter: Elijah Zupancic Summary Using Winstone Jetty 3.1 or below it will not run in HTTPS mode on hosts with 37 cores/processors or more. This problem replicates
