akosiaris added a comment.
In T273097#6801429 <https://phabricator.wikimedia.org/T273097#6801429>, @RKemper wrote: > @akosiaris Is your concern with the idea of using a`flink` base image solution mainly just centered around the inefficiency/inconvenience of needing SRE to merge any flink version upgrades? I wouldn't call it merely an inconvenience. I would call it a source of potential friction between teams, see below for an elaboration. > Since we have an embedded SRE on search (me) and to a lesser extent Guillaume, I think it wouldn't be too much of a problem. So, say a bus factor of 1.5 or so? Not ideal, but workable. That being said, it all depends on the urgency, see below. > In general having our dependencies managed by a docker image will make it easier for us to be explicit about what version we're using, and it seems like the default docker-y way of doing things. Is there a technical reason why a base image might not be a good idea? Unfortunately, even being explicit on the level of docker images won't make it easer. And the reason for that is the source of the content (the flink tar.gz file) will still not be under Search's control (correct me if I am wrong, I might have misunderstood something) but instead on the flink project's servers. What this will mean is that the creation of the base image will succeed only for a short period of time after bumping the version of flink, since the flink project, per my undestanding, removes old versions from their servers. However that is not the only time we build images. We semi-regularly have to rebuild the entire tree of docker images for some reason, the most usual ones being security updates in the base images, some misconfiguration or some newer features being added to our image building toolkit. What that means is that on the next shellshock/heartbleed/younameit the rebuild breaks. Then, SRE comes knocking on Search's door, asking for a bump the version of flink in that container cause we need to rebuild it. Now, you will obviously point out that Search will probably be relying on some version of the container they can still rely on the old one and upgrade on their own timeline. However, during that timeframe before the upgrade happens is: - SRE has an unbuildable image which it does not control and knows little about. So it does what comes naturally, which is complain. - Search is running an image that does not have the security upgrades. - SRE is pushing for having images without known security vulnerabilities. - Search is forced to alter plans and reprioritize upgrading the flink version cause SRE is complaining. So, a source of friction between teams. All of this can be solved by just moving the source of the problem (the flink .tar.gz) somewhere that Search controls so they can guarantee that a) it's the version Search wants b) fetching it during the building phase will not fail. But if that happens, most of this discussion becomes moot and the layer that docker image would be is just a potential (depending on whether the layer is present or not on the build host) speedup during building (which we can do, but for the right reasons). Interestingly, the debian package approach discussed in T266495 <https://phabricator.wikimedia.org/T266495> does have the 2 attributes outlined above which is why it's preferable to me. It still relies on an SRE of course to upgrade it which is a minus, but it shares this minus with this proposal. Ideally we wouldn't even need that and any member of Search would be able to update it. TASK DETAIL https://phabricator.wikimedia.org/T273097 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: akosiaris Cc: RKemper, dcausse, Aklapper, Gehel, akosiaris, Mstyles, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
