dduvall added a comment.
In T265512#6642612 <https://phabricator.wikimedia.org/T265512#6642612>, @Mstyles wrote: > The part that is really important to our team is getting our docker image into the WMF docker registry so that we can use it to deploy our Flink cluster to the WMF Kubernetes cluster. Including the Java development process is a nice to have, but there are many Java projects in the same RDF repository that will still be using our established Java build and deploy process. Changing our Java process just for this particular Java project (the streaming-updater-producer) could lead to more complications down the line. That's important to us, too. We want a smooth process for you all and a setup that's maintainable. In our experience getting other projects into the pipeline, having everything in band is an important requirement for a reliable delivery pipeline. I understand the hesitancy though, and we definitely don't want to complicate the Java processes themselves, just figure out how to containerize them in a way that fits with our pipeline model. Is losing the publishing of jars one of the concerns? We should be able to containerize the build and publish processes and move it into the pipeline as well. In other words, you can easily define a Blubber variant for the publishing step and we can configure Jenkins to expose the credentials for Archive to your project's pipeline job. You'd no longer have to manage that process manually, and the build/publish jar/publish image processes would all be managed in one place, under your team's control/discretion, and contiguous. In T265512#6648792 <https://phabricator.wikimedia.org/T265512#6648792>, @akosiaris wrote: > In T265512#6648623 <https://phabricator.wikimedia.org/T265512#6648623>, @Gehel wrote: > >> > > > >> My (limited) understanding of the usual pipelines is that we expect a 1-to-1 mapping between project source code, packaged artifact and deployment. > > That is true for usual pipelines, but having met this already for the deployment pipeline, there has been work already to not have that 1-to-1 mapping. Examples of the fact we don't have a 1-to-1 mapping between packaged artifact and deployment is eventgate, which has 4 different installations[1]. Also, depending on the support from the framework used, there isn't necessarily a 1-1 mapping between repo and packaged artifact. It is true that 1 repo needs to be the umbrella one for which configuration is setup, but otherwise that repo can fetch in dependencies. Finally there has been work on the possibility to create more than 1 docker container (the artifact part) from a single repo. However, our testbed's (ORES) team does not have the capacity to test that out and the ORES migration to kubernetes has been declined (for reasons mostly unrelated to the pipeline). As @akosiaris mentions, the `.pipeline/config.yaml` allows for handling multiple projects in the same repo. The actual config for this would look something like: pipelines: foo-test: directory: foo # stages for testing/building/publishing foo bar-test: directory: bar # stages for testing/building/publishing bar We should probably add a guide for this. Also, since the directory is actually traversed per stage in the current implementation, we could easily modify pipelinelib's configuration to allow for that `directory:` directive to be given for different stages in the same pipeline, if that would be helpful in this case. It might be more generally useful. Something like: pipelines: test: stages: - name: test-foo directory: foo build: test run: true - name: test-bar directory: bar build: test run: true # ... Anyway, that's just to say we're up for making changes that would make a pipelinelib migration easier, smoother, and more maintainable. :) TASK DETAIL https://phabricator.wikimedia.org/T265512 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles, dduvall Cc: dduvall, jeena, akosiaris, Aklapper, Gehel, Mstyles, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
