Looks like this is Docker's expected behavior [1]. For Python, this isn't a problem because the default tag is `2.17.0.dev`, not `latest`. Not sure why that's different for Java? Anyway, according to this comment:
'Note that docker pull will not *overwrite* your local image, it will "untag" it (i.e. remove the name and :latest tag from the old image and put it on the new one)' I imagine you should be able to get around this without changing source code by setting pipeline option `--environment_config=$IMAGE_ID`. Where $IMAGE_ID is the hash of your local image. [1] https://github.com/moby/moby/issues/10291 Kyle Weaver | Software Engineer | github.com/ibzib | [email protected] On Tue, Sep 24, 2019 at 11:56 AM Brian Hulette <[email protected]> wrote: > I'm working on a demo cross-language pipeline on a local flink cluster > that relies on my python row coder PR [1]. The PR includes some changes to > the Java worker code, so I need to build a Java SDK container locally and > use that in the pipeline. > > Unfortunately, whenever I run the pipeline, the apachebeam/java_sdk:latest > tag is moved off of my locally built image to a newly downloaded image with > a creation date 2 weeks ago, and that image is used instead. It looks like > the reason is we run `docker pull` before running the container [2]. As the > comment says this should be a no-op if the image already exists, but that > doesn't seem to be the case. If I just run `docker pull > apachebeam/java_sdk:latest` on my local machine it downloads the 2 week old > image and happily informs me: > > Status: Downloaded newer image for apachebeam/java_sdk:latest > > Does anyone know how I can prevent `docker pull` from doing this? I can > unblock myself for now just by commenting out the docker pull command, but > I'd like to understand what is going on here. > > Thanks, > Brian > > [1] https://github.com/apache/beam/pull/9188 > [2] > https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerCommand.java#L80 >
