as I said your problem is the IO, if you enter en the Instance by ssh and check the iostats you will see more than 5-10% of your operations waiting for IO. NFS, EFS, and in general network filesystems works well with big files but with small files and write concurrence is where the problems start
El vie., 25 sept. 2020 a las 15:02, Mark Schroering (< mark.schroer...@gmail.com>) escribió: > We are using AWS EFS for the Jenkins Home mount. It was configured for > burst throughput, and after reading > https://aws.amazon.com/blogs/storage/best-practices-for-using-amazon-efs-for-container-storage/ > we just changed it to provisioned throughput of 150 MiB/s. The change did > not help with the slow login times. We are still digging through the logs, > but are not sure what is causing the big time gaps. > > > > On Thursday, September 24, 2020 at 7:05:59 PM UTC-4 kuisat...@gmail.com > wrote: > >> Is your Jenkins home in a NFS or other network storage? I think so for >> the mount point, when a user enter a few files are written, because your IO >> is slow the IO operations are blocked waiting to finish that make the login >> slower than expected. You probably has more performance issues, I usually >> recommend to not use NFS file systems for the Jenkins home, take a look to >> this KB >> https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide >> >> El jueves, 24 de septiembre de 2020 a las 15:52:05 UTC+2, >> mark.sc...@gmail.com escribió: >> >>> Here are the logs in a better format. >>> >>> Sep 24, 2020 7:52:17 AM >>> FINE org.pac4j.saml.client.SAML2Client retrieveUserProfileAdding >>> attribute value mark.schroering@*****.com for attribute null >>> Sep 24, 2020 7:52:17 AM >>> FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => >>> key: email / value: [mark.schroering@*****.com] / class >>> java.util.ArrayList >>> Sep 24, 2020 7:52:17 AM >>> FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => >>> key: notBefore / value: 2020-09-24T11:46:38.907Z / class >>> org.joda.time.DateTime >>> Sep 24, 2020 7:52:17 AM >>> FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => >>> key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class >>> org.joda.time.DateTime >>> Sep 24, 2020 7:52:17 AM >>> FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL >>> Sep 24, 2020 7:53:35 AM >>> FINE >>> org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin >>> called. Using consumerServiceUrl >>> https://ci.infra.lifeomic.com/securityRealm/finishLogin >>> Sep 24, 2020 7:53:35 AM >>> FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / >>> Sep 24, 2020 7:53:35 AM >>> FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL >>> Sep 24, 2020 7:53:45 AM >>> FINE >>> org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin >>> called. Using consumerServiceUrl >>> https://ci.infra.lifeomic.com/securityRealm/finishLogin >>> Sep 24, 2020 7:53:45 AM >>> FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL redirection: / >>> Sep 24, 2020 7:53:45 AM >>> FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL >>> Sep 24, 2020 7:54:13 AM >>> INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using >>> SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin >>> Sep 24, 2020 7:54:13 AM >>> INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver >>> resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml >>> Sep 24, 2020 7:54:13 AM >>> INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver >>> resolveAttempting to create directory structure for /mnt/jenkins_home >>> >>> On Thursday, September 24, 2020 at 9:37:54 AM UTC-4 Mark Schroering >>> wrote: >>> >>>> We have noticed it taking a very long time (up to 60s) to complete the >>>> SAML auth flow. Here are some logs showing the bigger time gaps. We are >>>> on version 1.1.7 of the SAML plugin and running Jenkins version 2.257. >>>> >>>> >>>> Sep 24, 2020 7:52:17 AM FINE org.pac4j.saml.client.SAML2Client >>>> retrieveUserProfileAdding attribute value mark.schroering@*****.com >>>> for attribute null Sep 24, 2020 7:52:17 >>>> AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => >>>> key: email / value: [mark.schroering@*****.com] / class >>>> java.util.ArrayList Sep 24, 2020 7:52:17 >>>> AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => >>>> key: notBefore / value: 2020-09-24T11:46:38.907Z / class >>>> org.joda.time.DateTime Sep 24, 2020 7:52:17 >>>> AM FINE org.pac4j.core.profile.UserProfile addAttributeno conversion => >>>> key: notOnOrAfter / value: 2020-09-24T11:56:38.907Z / class >>>> org.joda.time.DateTime Sep 24, 2020 7:52:17 >>>> AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperreset TCCL Sep 24, 2020 >>>> 7:53:35 >>>> AM FINE >>>> org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin >>>> called. Using consumerServiceUrl >>>> https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 >>>> 7:53:35 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL >>>> redirection: / Sep 24, 2020 7:53:35 >>>> AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 >>>> 7:53:45 >>>> AM FINE >>>> org.jenkinsci.plugins.saml.SamlSecurityRealmSamlSecurityRealm.doCommenceLogin >>>> called. Using consumerServiceUrl >>>> https://ci.infra.lifeomic.com/securityRealm/finishLogin Sep 24, 2020 >>>> 7:53:45 AM FINE org.jenkinsci.plugins.saml.SamlSecurityRealmSafe URL >>>> redirection: / Sep 24, 2020 7:53:45 >>>> AM FINEST org.jenkinsci.plugins.saml.OpenSAMLWrapperadapt TCCL Sep 24, 2020 >>>> 7:54:13 >>>> AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver Using >>>> SP entity ID https://ci.infra.lifeomic.com/securityRealm/finishLogin >>>> Sep 24, 2020 7:54:13 >>>> AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver >>>> resolveWriting sp metadata to /mnt/jenkins_home/saml-sp-metadata.xml Sep >>>> 24, 2020 7:54:13 >>>> AM INFO org.pac4j.saml.metadata.SAML2ServiceProviderMetadataResolver >>>> resolveAttempting to create directory structure for /mnt/jenkins_home >>>> >>>> Looking at the browser tools on page load: >>>> >>>> GET /securityRealm/commenceLogin <-- 57s >>>> GET /securityRealm/finishLogin <--- 38s >>>> >>>> the Okta SSO parts in between seem to be quick as expected. >>>> >>>> Any tips on how to further debug or troubleshoot would be appreciated. >>>> >>>> Thanks for the help. >>>> >>>> -- > You received this message because you are subscribed to a topic in the > Google Groups "Jenkins Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/jenkinsci-users/WgAwcT0OGvk/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > jenkinsci-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/1caf58e5-020d-44ab-a682-00a974054fbcn%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-users/1caf58e5-020d-44ab-a682-00a974054fbcn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Un Saludo Iván Fernández Calvo https://www.linkedin.com/in/iv%C3%A1n-fern%C3%A1ndez-calvo-21425033 -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAKo5QrrMdt%2BX22e%3D7YZ5xPL1%3DODEFOg6aFMbQhRBAnY2Gt-y6A%40mail.gmail.com.
