On Tue, Jul 1, 2025 at 2:37 PM Danny McCormick via dev <dev@beam.apache.org> wrote:
> I think it is probably reasonable to automate this when a GPU resource > hint is used. I think we still need to expose this as a config option for > the ML containers (and it is the same with distroless) since it is pretty > difficult to say with confidence that those images are/aren't needed (even > if you're using a transform like RunInference, maybe you're using Spacy > which isn't a default dependency included in the ML images) and there is a > cost to using them (longer startup times). > > > This being the messy world of ML, would these images be > mahine/accelerator agnostic? > > That is the goal (at least to be agnostic within GPU types), and the > images will be as simple as possible to accommodate this. I think building > from an Nvidia base should accomplish this for most cases. For anything > beyond that, I think it is reasonable to ask users to build their own > container. > > On Tue, Jul 1, 2025 at 1:36 PM Robert Bradshaw <rober...@waymo.com> wrote: > >> On Tue, Jul 1, 2025 at 10:32 AM Kenneth Knowles <k...@apache.org> wrote: >> > >> > Obligatory question: can we automate this? Specifically: can we publish >> the ML-specific containers and then use them as appropriate without making >> it a user-facing knob? >> >> +1 >> >> Transforms can declare their own environments. The only problem with >> this is that distinct environments prohibit fusion--we need a way to >> say that a given environment is a superset of another. (We can do this >> with dependencies, but not with arbitrary docker images.) (One could >> possibly get away with the "AnyOf" environment as the base environment >> as well, if we define (and enforce) a preference order.) >> >> This comes up a lot for us (Schrodinger). e.g. our runner allows for transforms to specify what licenses they require, but the current rules for environment compatibility make it difficult to allow transforms that have no license requirements to fuse with environments that do have requirements (as a workaround, we just implement this through transform annotations). It'd also be really convenient for us since we don't ship our software with GCP libraries so we need a separate environment for GCP-transforms. Allowing fusion of GCP-transforms with non-GCP-transforms will be a bit difficult with the current system. This being the messy world of ML, would these images be >> mahine/accelerator agnostic? >> >> > Kenn >> > >> > On Mon, Jun 30, 2025 at 12:07 PM Danny McCormick via dev < >> dev@beam.apache.org> wrote: >> >> >> >> Hey everyone, I'd like to propose publishing some ML-specific Beam >> containers alongside our normal base containers. The end result would be >> allowing users to specify `--sdk_container_image=ml` or >> `--sdk_container_image=gpu` so that their jobs run in containers which work >> well with ML/GPU jobs. >> >> >> >> I put together a tiny design, please take a look and let me know what >> you think. >> >> >> >> >> https://docs.google.com/document/d/1JcVFJsPbVvtvaYdGi-DzWy9PIIYJhL7LwWGEXt2NZMk/edit?usp=sharing >> >> >> >> Thanks, >> >> Danny >> >
