I think it is probably reasonable to automate this when a GPU resource hint is used. I think we still need to expose this as a config option for the ML containers (and it is the same with distroless) since it is pretty difficult to say with confidence that those images are/aren't needed (even if you're using a transform like RunInference, maybe you're using Spacy which isn't a default dependency included in the ML images) and there is a cost to using them (longer startup times).
> This being the messy world of ML, would these images be mahine/accelerator agnostic? That is the goal (at least to be agnostic within GPU types), and the images will be as simple as possible to accommodate this. I think building from an Nvidia base should accomplish this for most cases. For anything beyond that, I think it is reasonable to ask users to build their own container. On Tue, Jul 1, 2025 at 1:36 PM Robert Bradshaw <rober...@waymo.com> wrote: > On Tue, Jul 1, 2025 at 10:32 AM Kenneth Knowles <k...@apache.org> wrote: > > > > Obligatory question: can we automate this? Specifically: can we publish > the ML-specific containers and then use them as appropriate without making > it a user-facing knob? > > +1 > > Transforms can declare their own environments. The only problem with > this is that distinct environments prohibit fusion--we need a way to > say that a given environment is a superset of another. (We can do this > with dependencies, but not with arbitrary docker images.) (One could > possibly get away with the "AnyOf" environment as the base environment > as well, if we define (and enforce) a preference order.) > > This being the messy world of ML, would these images be > mahine/accelerator agnostic? > > > Kenn > > > > On Mon, Jun 30, 2025 at 12:07 PM Danny McCormick via dev < > dev@beam.apache.org> wrote: > >> > >> Hey everyone, I'd like to propose publishing some ML-specific Beam > containers alongside our normal base containers. The end result would be > allowing users to specify `--sdk_container_image=ml` or > `--sdk_container_image=gpu` so that their jobs run in containers which work > well with ML/GPU jobs. > >> > >> I put together a tiny design, please take a look and let me know what > you think. > >> > >> > https://docs.google.com/document/d/1JcVFJsPbVvtvaYdGi-DzWy9PIIYJhL7LwWGEXt2NZMk/edit?usp=sharing > >> > >> Thanks, > >> Danny >
