Kyle, Benjamin Thank you for the quick reply. And yes, adding "environment_config" to Pipeline options did the trick.
=========================================================== options = PipelineOptions([ "--runner=FlinkRunner", "--flink_version=1.8", "--flink_master_url=localhost:8081", "--environment_config= asia.gcr.io/PROJECTNAME/beam/python3", "--experiments=beam_fn_api" ]) p = beam.Pipeline(options=options) =========================================================== Thanks, Yu Watanabe On Tue, Sep 17, 2019 at 11:21 AM Benjamin Tan <benjamintanwei...@gmail.com> wrote: > Something like this: > > options = PipelineOptions(["--runner=PortableRunner", > "--environment_config=apachebeam/python3.7_sdk > ", # <--- > "--job_endpoint=dnnserver2:8099"]) > > On 2019/09/17 02:14:00, Kyle Weaver <kcwea...@google.com> wrote: > > I think environment_config is the option you are looking for. > > > > On Mon, Sep 16, 2019 at 7:06 PM Yu Watanabe <yu.w.ten...@gmail.com> > wrote: > > > > > Hello. > > > > > > For testing I would like to use image uploaded to google container > > > registry. > > > How can I use images pulled from other than bintray.io ? > > > > > > > > > > ============================================================================ > > > admin@ip-172-31-9-89:~$ gcloud container images list --repository > > > asia.gcr.io/[PROJECTNAME] <http://asia.gcr.io/%5BPROJECTNAME%5D> < > http://asia.gcr.io/%5BPROJECTNAME%5D> > > > NAME > > > asia.gcr.io/[PROJECTNAME]/beam > <http://asia.gcr.io/%5BPROJECTNAME%5D/beam> < > http://asia.gcr.io/%5BPROJECTNAME%5D/beam> > > > > > > > ============================================================================ > > > > > > Looks like FlinkRunner (2.15.0) uses bintray repository as a default > > > behavior. As a result I am not able to use > > > > > > Caused by: java.lang.Exception: The user defined 'open()' method > caused an > > > exception: java.io.IOException: Received exit code 125 for command > 'docker > > > run -d --network=host --env=DOCKER_MAC_CONTAINER=null --rm > > > ywatanabe-docker-apache.bintray.io/beam/python3:latest --id=1 > > > --logging_endpoint=localhost:33787 --artifact_endpoint=localhost:41681 > > > --provision_endpoint=localhost:45089 > --control_endpoint=localhost:37045'. > > > stderr: Unable to find image ' > > > ywatanabe-docker-apache.bintray.io/beam/python3:latest' locallydocker: > > > Error response from daemon: manifest for > > > ywatanabe-docker-apache.bintray.io/beam/python3:latest not found: > > > manifest unknown: The named manifest is not known to the registry.See > > > 'docker run -- > > > > > > According to online doc, there is not parameter which controls which > image > > > to use . > > > > > > https://beam.apache.org/documentation/runners/flink/ > > > > > > Pipeline options I am using is below. > > > > > > > > > > ============================================================================ > > > options = PipelineOptions([ > > > "--runner=FlinkRunner", > > > "--flink_version=1.8", > > > "--flink_master_url=localhost:8081", > > > "--worker_harness_container_image= > > > asia.gcr.io/PROJECTNAME/beam/python3", > > > "--experiments=beam_fn_api" > > > ]) > > > > > > > ============================================================================ > > > > > > Perhaps is there any environment variable to specify which image to > use ? > > > > > > Best Regards, > > > Yu Watanabe > > > > > > -- > > > Yu Watanabe > > > Weekend Freelancer who loves to challenge building data platform > > > yu.w.ten...@gmail.com > > > [image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> > [image: > > > Twitter icon] <https://twitter.com/yuwtennis> > > > > > -- > > Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com > > > -- Yu Watanabe Weekend Freelancer who loves to challenge building data platform yu.w.ten...@gmail.com [image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> [image: Twitter icon] <https://twitter.com/yuwtennis>