On Mon, Dec 30, 2024 at 5:36 AM Jason Smyth <[email protected]> wrote:
> > In terms of cross-repository code, what I mean is that if I am working > with an app that lives at ./Apps/App1, the project for this code is > structured in such a way that it assumes someLib exists at > ../../Libs/someLib/bin (relative to my App1 directory). This relative path > is accurate within the TFVC monorepo, but with the move to Git, someLib has > been moved into a separate Git repo. In order to ensure the builds continue > to work without having too much impact on the developer workflow, we need > to recreate that same relative pathing at checkout time. > Given the situation you are in, and assuming splitting out this part to another repo is desirable for other reasons - you could consider - whether git submodules are a reasonable enough "bridge" to your earlier TFVC monorepo (keeping in mind the complexity) - modelling as an artifact dependency instead, where every time those libs are changed, they are uploaded/artifacted by GoCD and then the pipelines that need them fetch the artifact and put it at the expected ../../Libs/someLib/bin location prior to the rest of the tasks running. If I recall correctly, artifact destinations in GoCD are pretty flexible about where those artifacts go, since the fetch process happens post clone/fetch Which solution works the best probably depends a bit on the local developer workflow the devs teams want to have, since presumably when working locally they also need a mechanism to get those libs/binaries into the right location; or perhaps to work concurrently on changing code that contributes to those libs as well as the code that uses them. -Chad > > Realistically, we probably should work with the development team to come > up with some new workflows that are more aligned with Git and code > separation principles, but the objective of our current initiative is to > modernize our tech stack as cheaply as possible. In practical terms, that > means we are primarily looking to recreate our existing workflows using the > current version of GoCD (instead of 19.8.0), backed by Git on Azure DevOps > rather than TFVC on TFS Server 2013. > > I think you are probably right and the only sane way of tackling this for > now is to use no more than one git-path material for any given Git repo on > a pipeline. I will have to think a bit on how to sanely name the materials > to ensure that: > > - When 2 pipelines should share a material, it is easy to give it the > same name in both YAML files, and > - There are no naming conflicts between multiple pipelines that need > different slices of any given repo. > > Thank you again for taking the time to respond. As always, your insights > are appreciated. > > Cheers, > Jason > > > On Sunday, 29 December 2024 at 01:38:32 UTC-5 Chad Wilson wrote: > >> Sorry, I should have mentioned that I am broadly aware of git >> sparse-checkout >> <https://github.blog/open-source/git/bring-your-monorepo-down-to-size-with-sparse-checkout/> >> (most >> similar to a git native approach for this) but have not gone into this in >> detail or evaluated whether it could make sense in the context of something >> like GoCD - or is really only something that could effectively be used by >> an end-user. >> >> This git feature was added to git subsequent to most of the rework I did >> on the GoCD git-path plugin and I haven't looked at whether other build >> automation tools have support for use of this server side. >> >> -Chad >> >> On Sun, Dec 29, 2024 at 2:32 PM Chad Wilson <[email protected]> >> wrote: >> >>> Hiya Jason >>> >>> On Sun, Dec 29, 2024 at 8:46 AM Jason Smyth <[email protected]> wrote: >>> >>>> Hi everyone, >>>> >>>> I am starting with the git-path plugin and I am having trouble >>>> understanding how it should be configured to ensure the files end up where >>>> I expect them when the agent fetches them. >>>> >>>> I am working with a pipeline that uses the following materials >>>> (whittled down to what I understand to be the relevant bits): >>>> >>>> materials: >>>> App.Trunk: >>>> plugin_configuration: >>>> id: "git-path" >>>> options: >>>> url: https://dev.azure.com/Org/Project/_git/App >>>> path: Trunk >>>> destination: source/Project/App/Trunk >>>> App.Documents.Spec: >>>> plugin_configuration: >>>> id: "git-path" >>>> options: >>>> url: https://dev.azure.com/Org/Project/_git/App >>>> path: Documents/Spec >>>> destination: source/Project/App/Documents/Spec >>>> >>>> The intention was that the contents of App$/Trunk should be placed in >>>> source/Project/App/Trunk and the contents of App$/Documents/Spec should be >>>> placed in source/Project/App/Documents/Spec. Instead, the plugin seems to >>>> be fetching the entire repo into each of the destinations. Is this the >>>> expected behaviour? >>>> >>> >>> Yes, it is the expected behaviour. It clones the entire repo and leaves >>> everything else behind in other paths, at the versions they are current to >>> as of the specific `path` (aka git ref spec). To my knowledge there is no >>> native git way to get part of a file system tree like you'd suggest (as a >>> git ref represents the state of the *entire repo* at a given commit >>> independent of file system knowledge), so the only other alternative for >>> the plugin's implementation would likely be to do some file system level >>> hijinx to remove paths not fetched, which from a practical perspective >>> would likely mean removing ability to use all possible git ref specs >>> (documented >>> here >>> <https://github.com/TWChennai/gocd-git-path-material-plugin?tab=readme-ov-file#constructing-path-expressions>) >>> and instead allowing only simple path prefixes (which would be difficult to >>> validate in its own right without diving into a git ref spec parser. >>> >>> Basically the git path plugin allows you to mitigate excessive >>> triggering and reinterpret up-to-dateness for a subset of a repo (as >>> opposed to the allowlist/denylist approach which have other problems) - but >>> doesn't introduce some fuller concept of only fetching part of the git >>> repo. The remaining clone is still a fully functional git working directory >>> and repository off a given commit, which it would cease to be if doing some >>> non-git-native hijinx afterwards. This is somewhat discussed at >>> https://github.com/TWChennai/gocd-git-path-material-plugin?tab=readme-ov-file#stale-data-in-agent-repository-clones >>> and is why the language/examples focus on how it monitors for changes. >>> >>> The other problem with this approach here is that if a commit was made >>> that changes contents of both "Trunk" and "Documents/Spec", the independent >>> materials could detect this single commit at different times due to the way >>> material polling works. A triggered build may kick off with only the >>> changes for "Trunk" and the previous ref for "Documents/Spec" (or vice >>> versa). If these paths are not sufficiently independent, modelling as >>> separate materials is likely to hurt rather than help. >>> >>> >>>> >>>> If so, are there any guidelines for how to deal with multiple git-path >>>> materials that need to poll different paths in a single repo, while >>>> ensuring that the relative paths remain intact on the agent at job run >>>> time? >>>> >>> >>> >>>> >>>> Things that I need to consider: >>>> >>>> - App.Trunk and App.Documents.Spec are likely to be reused across >>>> various pipelines, though not necessarily always together. >>>> - We probably do not want to configure a custom git-path material >>>> for every existing combination of paths. >>>> - There is a significant amount of cross-repository code, so >>>> relative paths both inside and across repositories can be relevant. >>>> (I.E., >>>> for any given file, the right version of that file needs to be >>>> downloaded >>>> to "./Project/Repo/path/to/file".) >>>> >>>> >>> Only sensible option IMHO is to use a single material off the same >>> "wider" repo for both paths (violating your second requirement) >>> >>> path: "Trunk, Documents/Spec" >>> >>> If you used something non-yaml to generate your config repo contents you >>> could conceptually programmatically generate this. >>> >>> I'm not sure I understand what "cross repository code" means, but you >>> could perhaps consider shifting some of that responsibility to git >>> submodules - although I don't really like them personally due to complexity >>> and the way they change developer flow. I also don't know how effectively >>> submodules work with the git-path plugin specifically. >>> >>> >>>> >>>> I'm thinking I will need to pull the git-path materials into a separate >>>> location, then copy the relevant files to the expected location in the >>>> first (few) task(s) of the job. (E.G., fetch them into >>>> ./git-path/<materialName>, then copy "./git-path/App.Trunk/Trunk" to >>>> "./source/Project/App/Trunk" ) This feels incredibly hacky, though. Are >>>> there any cleaner options? >>>> >>> >>> Personally I don't think what you are trying to do is compatible with >>> the conceptual design goals of either git or GoCD. If these various paths >>> are really independent of one another and cannot be looked at within the >>> scope of the wider repo or organised into a "simple" mono repo structure, >>> should they possibly be independent repositories? >>> >>> Otherwise you are losing all the guarantees of GoCD that the materials >>> are at consistent ref versions with one another, etc, and that a git repo >>> at a given ref is a complete representation of the repo at that ref/sha. >>> The git-path plugin already slightly moves away from the GoCD integrity >>> guarantee to "allow" for different subsets of a repo to be considered as >>> independent materials and push more "risk" into the hands of the user - but >>> there's probably a limit to how far you should consider pushing that >>> compromise. >>> >>> But to answer your question, no there are no cleaner options if trying >>> to slice-and-dice a repository at various repository versions/refs and >>> assemble it back together. Personally I would (and have historically) >>> combined paths together if I still felt the plugin was useful enough to use >>> in its current form. >>> >>> -Chad >>> >>> >> -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/d/msgid/go-cd/c23f9c20-a7f3-4bd4-b58c-ba7348c1fa65n%40googlegroups.com > <https://groups.google.com/d/msgid/go-cd/c23f9c20-a7f3-4bd4-b58c-ba7348c1fa65n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAA1RwH8SXFL2gwMCn%2BHtZYk7qDm%2B%2BKAD6A8Ct7UaZ5P3tHsZSw%40mail.gmail.com.
