Re: [vpp-dev] FD.io CI Outage
Folks, Jenkins was restarted last night which resolved the outage. RCA is still being investigated. Thanks, -daw- On 6/20/22 4:36 PM, Dave Wallace via lists.fd.io wrote: Folks, Jenkins.fd.io was inadvertently put into maintenance mode on Saturday. It was taken out of maintenance mode about 5 hours ago, but VPP & CSIT jobs are still stuck in the Jenkins Build Queue (currently 18 jobs in the queue). LF-IT tickets [0] have been opened and resolution of the issue is ongoing. Thank you for your patience as today is a US Holiday and the regular LF-IT support team is short staffed. -daw- [0] https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-24179 https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-24181 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21562): https://lists.fd.io/g/vpp-dev/message/21562 Mute This Topic: https://lists.fd.io/mt/91886472/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] FD.io CI Outage
Folks, Jenkins.fd.io was inadvertently put into maintenance mode on Saturday. It was taken out of maintenance mode about 5 hours ago, but VPP & CSIT jobs are still stuck in the Jenkins Build Queue (currently 18 jobs in the queue). LF-IT tickets [0] have been opened and resolution of the issue is ongoing. Thank you for your patience as today is a US Holiday and the regular LF-IT support team is short staffed. -daw- [0] https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-24179 https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-24181 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21557): https://lists.fd.io/g/vpp-dev/message/21557 Mute This Topic: https://lists.fd.io/mt/91886472/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] FD.io CI outage
[0] https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-23405 On 12/15/21 1:13 PM, Dave Wallace via lists.fd.io wrote: Folks, Due to service issues in AWS, gerrit.fd.io (AWS instance) is currently not communicating with jenkins.fd.io (Vexxhost instance). I have opened a ticket with LF-IT [0] and Vanessa has put jenkins.fd.io into shutdown mode in anticipation of a reset when the communications issue is resolved. Thanks, -daw- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20638): https://lists.fd.io/g/vpp-dev/message/20638 Mute This Topic: https://lists.fd.io/mt/87749872/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] FD.io CI outage
Folks, Due to service issues in AWS, gerrit.fd.io (AWS instance) is currently not communicating with jenkins.fd.io (Vexxhost instance). I have opened a ticket with LF-IT [0] and Vanessa has put jenkins.fd.io into shutdown mode in anticipation of a reset when the communications issue is resolved. Thanks, -daw- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20637): https://lists.fd.io/g/vpp-dev/message/20637 Mute This Topic: https://lists.fd.io/mt/87749872/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] FD.io CI Outage has been resolved
Folks, Sorry for the late notice, but I've been meeting bound while troubleshooting today's FD.io CI outage. There was a network configuration change in the Vexxhost datacenter that cause the public IP address of jenkins to become inaccessible from the Nomad cluster causing all jobs to fail. This has been resolved. Another issue still remains, that is why the external address was being used. It was determined that Jenkins was configured this way in the a portion of the cloud configuration even though Jenkins was configured to use the internal IP address in the global configuration variables. Unfortunately updating the cloud configuration failed to work after the network issues were resolved and restoring the cloud configuration to use jenkins.fd.io resolved the issue. Given today's outage, I have recommended that Andrew push back the VPP 21.10 branch pull to Thursday 9/23/2021. Kudo's to Peter Mikus for identifying the root cause of the outage, Vanessa Valderrama for assisting during a PTO day to manage Jenkins configuration changes & restarts, and Anton Baranov & Mohammed Naser for fixing the data center network configuration. Thanks, -daw- -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20174): https://lists.fd.io/g/vpp-dev/message/20174 Mute This Topic: https://lists.fd.io/mt/85775812/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
