Thanks for sharing. Sounds like it could be a network policy thing or something like that? Did you double check the cluster profile config to see if anything is amiss there?
I can't see why you'd get a different error message with 23.1.0 (and the same plugin version) unless there was something different about the cluster profile config (API server url etc), so that might be an avenue for investigation? -Chad On Sat, 28 Oct 2023, 05:16 Kim Pham, <k...@livelyvideo.tv> wrote: > Hi, > > I just wanted to give an update. We're still unsure what's preventing > agents from spinning up on our old cluster. The pods aren't getting > created at all and there's no logs in Kubernetes side indicating activity. > I've tried with a regular Debian agent and the logs are the same. I did > roll back to GoCD 23.1.0 to see what the log output was and it's showing a > SocketTimeoutException error. We're still investigating but it's possible > some configuration changed that we're not aware of with that cluster. > > I was able to spin up a new cluster and it works fine there. That cluster > is running: > > - GoCD Server: 23.3.0 > - GoCD DinD Agent: 23.3.0 > - Elastic Agent Node Pools: 1.25.10-gk3.2700 > > I'll update if there's anything relevant to share. Thanks for all the > help! > > On Thu, Oct 26, 2023 at 10:55 AM Kim Pham <k...@livelyvideo.tv> wrote: > >> Hi Chad, >> >> I was just reading up on the changes for Kubernetes. Looks 1.24 moves to >> containerd runtime images and we are still using DIND for our elastic >> agents. That seems like it could be the culprit. I'll do some testing and >> change our elastic agent images. Thanks for pointing that out. >> >> On Thu, Oct 26, 2023 at 10:46 AM Chad Wilson <ch...@thoughtworks.com> >> wrote: >> >>> Unfortunately the error message is a bit mysterious and useless. Which >>> agent image you are using? Anything special in the elastic agent pod spec >>> that might no longer work as expected on Kubernetes 1.24? (e.g use of >>> docker dind images) >>> >>> Does the pod get created (and fail) if you look at the events on the >>> kubernetes side, or does it never get that far? >>> >>> -Chad >>> >>> On Thu, Oct 26, 2023 at 11:37 PM Chad Wilson <ch...@thoughtworks.com> >>> wrote: >>> >>>> Just curious - were the errors/stack traces on failure essentially >>>> identical before and after you upgraded your gocd and elastic agent plugin >>>> versions? >>>> >>>> On Thu, Oct 26, 2023 at 11:33 PM Kim Pham <k...@livelyvideo.tv> wrote: >>>> >>>>> Hi Ashwanth, >>>>> >>>>> I checked the clusterrole of the service account it's using and it >>>>> basically has full access atm. >>>>> >>>>> PolicyRule: >>>>> Resources Non-Resource URLs Resource Names Verbs >>>>> --------- ----------------- -------------- ----- >>>>> events [] [] [*] >>>>> namespaces [] [] [*] >>>>> nodes [] [] [*] >>>>> pods/log [] [] [*] >>>>> pods [] [] [*] >>>>> >>>>> On Thu, Oct 26, 2023 at 10:21 AM 'Ashwanth Kumar' via go-cd < >>>>> go-cd@googlegroups.com> wrote: >>>>> >>>>>> A wild guess, anything changed on the service account side or a >>>>>> custom role being added as part of the upgrade that is probably not >>>>>> allowing the gocd plugin to create the pod? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> >>>>>> On Thu, 26 Oct 2023 at 20:27, Kim Pham <k...@livelyvideo.tv> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> We recently began to encounter issues where pods were unable to be >>>>>>> created. Nothing has changed in terms of GoCD server, agent, and >>>>>>> Kubernetes elastic agent plugin versions. However, we did notice that >>>>>>> the >>>>>>> cluster went through an automatic upgrade and updated gke version to >>>>>>> 1.24.14. GoCD is able to see the node pools through the 'Status Report' >>>>>>> button. >>>>>>> >>>>>>> When attempting to create an agent on those node pools, I do see a >>>>>>> 500 in the plugin logs and gocd-server logs. Attached are logs. >>>>>>> >>>>>>> I've tried updating GoCD and the plugins to latest release >>>>>>> versions. Our static agents that are running on older gke versions >>>>>>> aren't >>>>>>> having any issues. >>>>>>> >>>>>>> Has anyone encountered this? >>>>>>> >>>>>>> Thanks in advance. >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "go-cd" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to go-cd+unsubscr...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/go-cd/a6b8e99d-f415-4c18-b67d-e86c3df16733n%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/go-cd/a6b8e99d-f415-4c18-b67d-e86c3df16733n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Ashwanth Kumar / ashwanthkumar.in >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "go-cd" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to go-cd+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/go-cd/CAD9m7Cw2cK5yt_r9e5r1sxD%2B%2B%2Bjkd3%3DFTQe51vKqb081MunU%3DQ%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/go-cd/CAD9m7Cw2cK5yt_r9e5r1sxD%2B%2B%2Bjkd3%3DFTQe51vKqb081MunU%3DQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "go-cd" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to go-cd+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/go-cd/CA%2BnJfx4-Zks6-FOr1bYSOroxG2o4e2e4ir0OFUm3TsggWsrYpA%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/go-cd/CA%2BnJfx4-Zks6-FOr1bYSOroxG2o4e2e4ir0OFUm3TsggWsrYpA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to go-cd+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/CAA1RwH-tw%3D0MFCgWRQoT8ZwFYcyJsGT8T5pPxrBzzu46Nn960w%40mail.gmail.com >>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH-tw%3D0MFCgWRQoT8ZwFYcyJsGT8T5pPxrBzzu46Nn960w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to go-cd+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/CA%2BnJfx6jX2e6kPjXr68MizbuBsWFWXdKzJWsBwh-aFDoNNQGpw%40mail.gmail.com > <https://groups.google.com/d/msgid/go-cd/CA%2BnJfx6jX2e6kPjXr68MizbuBsWFWXdKzJWsBwh-aFDoNNQGpw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH-S%3D7f87AR5xPvwuut7YH1cOJTOZk%2B0n-5dB%3DNbzi9aNQ%40mail.gmail.com.
