Did another few checks and it did indeed work OK if there is 1+ pipeline sharing the material so no harm fixing the case you discovered - have also improved the error message and the logging associated to be a bit less misleading. Fixed the underlying problem here <https://github.com/gocd/gocd/pull/13232> - release in 24.4.0 (out now <https://www.gocd.org/download/#docker>) - give it a go?
-Chad On Sun, Nov 3, 2024 at 12:55 AM Chad Wilson <[email protected]> wrote: > The error you see seems to happen because when validating secret lookup > references (to check the *rules *configured on the secret config) the > logic seems to incorrectly assume that any ScmMaterial with secrets will > have 1+ pipeline using that secret. It then incorrectly assumes that "all > pipelines have secret lookup errors" when it compares # pipelines and # > pipelines with errors (0 == 0, oops) and fails. > > That's why you get the useless/empty error message and broken resolution. > If you turn on DEBUG logging for > *com.thoughtworks.go.server.materials.MaterialDatabaseUpdater* you'll see > the below with no error message: > > 2024-11-03 00:14:00,477 DEBUG [147@MessageListener for > MaterialUpdateListener] MaterialDatabaseUpdater:128 - [Material Update] > Modification check failed for material: URL: https://github.com/gocd/aws, > Branch: master > com.thoughtworks.go.server.exceptions.RulesViolationException: > at > com.thoughtworks.go.server.service.RulesService.validateSecretConfigReferences(RulesService.java:82) > at > com.thoughtworks.go.server.service.SecretParamResolver.resolve(SecretParamResolver.java:77) > at > com.thoughtworks.go.server.service.SecretParamResolver.resolve(SecretParamResolver.java:67) > at > com.thoughtworks.go.server.service.MaterialService.resolveSecretParams(MaterialService.java:163) > at > com.thoughtworks.go.server.service.MaterialService.latestModification(MaterialService.java:126) > at > com.thoughtworks.go.server.materials.LegacyMaterialChecker.findLatestModification(LegacyMaterialChecker.java:50) > at > com.thoughtworks.go.server.materials.ScmMaterialUpdater.insertLatestOrNewModifications(ScmMaterialUpdater.java:55) > at > com.thoughtworks.go.server.materials.ScmMaterialUpdater.addNewMaterialWithModifications(ScmMaterialUpdater.java:70) > at > com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.addNewMaterialWithModifications(MaterialDatabaseUpdater.java:179) > at > com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.initializeMaterialWithLatestRevision(MaterialDatabaseUpdater.java:137) > at > com.thoughtworks.go.server.materials.MaterialDatabaseUpdater$1.doInTransaction(MaterialDatabaseUpdater.java:95) > at > com.thoughtworks.go.server.transaction.TransactionCallback.doWithExceptionHandling(TransactionCallback.java:23) > at > com.thoughtworks.go.server.transaction.TransactionTemplate.lambda$executeWithExceptionHandling$1(TransactionTemplate.java:43) > at > org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133) > at > com.thoughtworks.go.server.transaction.TransactionTemplate.executeWithExceptionHandling(TransactionTemplate.java:40) > at > com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterial(MaterialDatabaseUpdater.java:92) > at > com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:64) > at > com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:32) > at > com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83) > at > com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63) > at java.base/java.lang.Thread.run(Thread.java:1583) > > If I fix the error handling so it doesn't incorrectly consider it a rules > violation it seems to work OK. However I'll need to think a little bit more > about whether it *should*. :-) > > I think if you add at least 1 pipeline that refers to the same logical > material (same type, URL, branch, username) you might find it works as > expected? > > *Test Connection* works because it seems to resolve secrets without > validating the references in the same way .... for better or worse. > Arguably it should be validating references here also. > > -Chad > > On Fri, Nov 1, 2024 at 1:10 AM Chad Wilson <[email protected]> wrote: > >> Not really, other than to raise a proper bug/enhancement report at >> https://github.com/gocd/gocd/issues preferably with the easiest way to >> replicate this so it can be tracked. I'd be digging into the GoCD issue >> history to see if this was expected to work (or worked at some point) the >> same as you :-) >> >> -Chad >> >> On Thu, Oct 31, 2024 at 10:32 PM Jason Smyth <[email protected]> wrote: >> >>> Hi Chad, >>> >>> Here is the error as it appears in the GoCD UI: >>> >>> [image: 2024-10-31 10-21 - brave_hDTIkVSRsL.png] >>> >>> I tried the "GoCD file based Secrets Plugin" and replicated the issue >>> with that secret manager as well. I think this suggests that the issue is >>> in the core GoCD configuration repo module. Unless there is something wrong >>> with my test cases. >>> >>> Thoughts? >>> >>> Regards, >>> Jason Smyth >>> >>> >>> On Wednesday 30 October 2024 at 10:24:45 UTC-4 Jason Smyth wrote: >>> >>>> Hi Chad, >>>> >>>> >>>> >>>> This is on the latest (I believe) version: v24.3.0. >>>> >>>> >>>> >>>> Regards, >>>> >>>> *Jason Smyth* >>>> >>>> >>>> >>>> *From:* [email protected] <[email protected]> *On Behalf Of *Chad >>>> Wilson >>>> *Sent:* Tuesday, October 29, 2024 11:00 PM >>>> *To:* [email protected] >>>> *Subject:* Re: [go-cd] Config Repository Not Parsing When Credentials >>>> Supplied via Secret (v24.3.0) >>>> >>>> >>>> >>>> OK, thanks. >>>> >>>> Are you working with a "modern" GoCD version, or is this still some >>>> older 19.x version? >>>> >>>> >>>> >>>> -Chad >>>> >>>> >>>> >>>> On Wed, Oct 30, 2024 at 5:28 AM Jason Smyth <[email protected]> wrote: >>>> >>>> Hi Chad, >>>> >>>> >>>> >>>> My tests were done using the "AWS Secrets Manager plugin for GoCD". >>>> There is nothing in go-server.log aside from what I posted in my initial >>>> message, and the secret plugin logs are empty. I can try turning up logging >>>> verbosity and repeating the tests, but I'm not sure how to do that in my >>>> Docker Compose test environment. >>>> >>>> >>>> >>>> I know that the secret plugin is working. I have pipelines configured >>>> in a working (because I used a password) config repo, and those pipelines >>>> depend on the same secret plugin. Additionally, the test connection button >>>> in the config repo modal returns errors if I intentionally misconfigure the >>>> secret. >>>> >>>> >>>> >>>> I don't think it has anything to do with the repository contents. I >>>> have replicated the behaviour in 2 different test systems, with multiple >>>> source repos, including some that are known to be good because they are >>>> accessed by other GoCD instances (albeit without secrets plugin >>>> references). I have also replicated the issue with both JSON and YAML >>>> repos, so I don't think the issue is with the individual configuration >>>> repository plugins. >>>> >>>> >>>> >>>> The config repos are not used as additional materials. I tested adding >>>> a config repo that pointed to a repo that _is_ used as a material for >>>> pipelines, and it works properly. I suspect that this means that the >>>> existing material configuration takes precedence and GoCD doesn't bother >>>> trying to re-clone, but I could be mistaken there. >>>> >>>> >>>> >>>> If I get a chance, I will try to see if I can replicate the issue with >>>> the "GoCD file based Secrets Plugin". That should provide some indication >>>> of whether the issue is in the secrets plugin or the core GoCD >>>> configuration repo module. >>>> >>>> >>>> >>>> Any other thoughts on things to try? >>>> >>>> >>>> >>>> Regards, >>>> >>>> Jason Smyth >>>> >>>> >>>> >>>> On Tuesday 22 October 2024 at 02:42:08 UTC-4 Chad Wilson wrote: >>>> >>>> This sounds weird/unexpected but haven't had time to try reproducing >>>> this myself. Which secrets plugin are you using? What's in the logs for the >>>> server or the secret plugin itself? >>>> >>>> >>>> >>>> Is the same repo url also used for other materials (whether config >>>> repos or normal materials)? >>>> >>>> On Fri, 18 Oct 2024, 23:29 Jason Smyth, <[email protected]> wrote: >>>> >>>> Hello community, >>>> >>>> >>>> >>>> I encountered a strange issue whereby config repositories don’t seem to >>>> work properly when we try to supply the password via a secret instead of >>>> directly in the config repo config. The connection test succeeds, so the >>>> system is fetching the password from the secret at that point, but once >>>> saved, the config repo fails to parse. >>>> >>>> >>>> >>>> The error message (URL redacted) I see in the UI is: >>>> >>>> >>>> >>>> There was an error parsing this configuration repository: >>>> >>>> MODIFICATION CHECK FAILED FOR MATERIAL: URL: >>>> HTTPS://DEV.AZURE.COM/ORGANIZATION/TEAMPROJECT/_GIT/SANDBOX-JASONS, >>>> BRANCH: GOCD-PIPELINE-TEST >>>> >>>> NO PIPELINES ARE AFFECTED BY THIS MATERIAL, PERHAPS THIS MATERIAL IS >>>> UNUSED. >>>> >>>> >>>> >>>> Failed to load pipelines defined in this repository: There was an >>>> unknown error performing the operation. Possible reason (Not Found) >>>> >>>> >>>> >>>> The GoCD server log shows the following warning: >>>> >>>> >>>> >>>> 2024-10-18 10:47:01 jvm 1 | 2024-10-18 14:47:01,263 WARN >>>> [143@MessageListener for ConfigMaterialUpdateListener] >>>> ConfigMaterialUpdateListener:65 - [Config Material Update] Cannot update >>>> configuration part because material update has failed. Reason: >>>> >>>> >>>> >>>> When I switched from using a secret to directly supplying the password >>>> via the UI, the configuration repository started working as intended. >>>> >>>> >>>> >>>> I’m reasonably certain that this is a bug, but wanted to check with the >>>> community to confirm that using secrets in this way is supposed to be a >>>> supported use-case. >>>> >>>> >>>> >>>> Any thoughts or guidance would be appreciated. >>>> >>>> >>>> >>>> Regards, >>>> >>>> *Jason Smyth* >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "go-cd" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/DM6PR16MB36715673328A38D68EA7AE8ECF402%40DM6PR16MB3671.namprd16.prod.outlook.com >>>> <https://groups.google.com/d/msgid/go-cd/DM6PR16MB36715673328A38D68EA7AE8ECF402%40DM6PR16MB3671.namprd16.prod.outlook.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "go-cd" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion visit >>>> https://groups.google.com/d/msgid/go-cd/5e1c44b3-be46-4764-8a39-fb45703f9193n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/go-cd/5e1c44b3-be46-4764-8a39-fb45703f9193n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "go-cd" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/go-cd/JlzHTa-Vy_0/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion visit >>>> https://groups.google.com/d/msgid/go-cd/CAA1RwH-qHq8U1qXRiZuupuh-T0ojU8j%3DTSbAt4VveEahAP8DMg%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH-qHq8U1qXRiZuupuh-T0ojU8j%3DTSbAt4VveEahAP8DMg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion visit >>> https://groups.google.com/d/msgid/go-cd/8b6f8c9a-78fd-45c2-badb-3a46befff0dcn%40googlegroups.com >>> <https://groups.google.com/d/msgid/go-cd/8b6f8c9a-78fd-45c2-badb-3a46befff0dcn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAA1RwH_DH9jRRoRTe7Kqdxkg8dQJpY86x6Gvs4%2BCm_mxrBJk8g%40mail.gmail.com.
