Kyle, Benjamin
Thank you for the quick reply.
And yes, adding "environment_config" to Pipeline options did the trick.
===========================================================
options = PipelineOptions([
"--runner=FlinkRunner",
"--flink_version=1.8",
"--flink_master_url=localhost:8081",
"--environment_config=
asia.gcr.io/PROJECTNAME/beam/python3",
"--experiments=beam_fn_api"
])
p = beam.Pipeline(options=options)
===========================================================
Thanks,
Yu Watanabe
On Tue, Sep 17, 2019 at 11:21 AM Benjamin Tan <[email protected]>
wrote:
> Something like this:
>
> options = PipelineOptions(["--runner=PortableRunner",
> "--environment_config=apachebeam/python3.7_sdk
> ", # <---
> "--job_endpoint=dnnserver2:8099"])
>
> On 2019/09/17 02:14:00, Kyle Weaver <[email protected]> wrote:
> > I think environment_config is the option you are looking for.
> >
> > On Mon, Sep 16, 2019 at 7:06 PM Yu Watanabe <[email protected]>
> wrote:
> >
> > > Hello.
> > >
> > > For testing I would like to use image uploaded to google container
> > > registry.
> > > How can I use images pulled from other than bintray.io ?
> > >
> > >
> > >
> ============================================================================
> > > admin@ip-172-31-9-89:~$ gcloud container images list --repository
> > > asia.gcr.io/[PROJECTNAME] <http://asia.gcr.io/%5BPROJECTNAME%5D> <
> http://asia.gcr.io/%5BPROJECTNAME%5D>
> > > NAME
> > > asia.gcr.io/[PROJECTNAME]/beam
> <http://asia.gcr.io/%5BPROJECTNAME%5D/beam> <
> http://asia.gcr.io/%5BPROJECTNAME%5D/beam>
> > >
> > >
> ============================================================================
> > >
> > > Looks like FlinkRunner (2.15.0) uses bintray repository as a default
> > > behavior. As a result I am not able to use
> > >
> > > Caused by: java.lang.Exception: The user defined 'open()' method
> caused an
> > > exception: java.io.IOException: Received exit code 125 for command
> 'docker
> > > run -d --network=host --env=DOCKER_MAC_CONTAINER=null --rm
> > > ywatanabe-docker-apache.bintray.io/beam/python3:latest --id=1
> > > --logging_endpoint=localhost:33787 --artifact_endpoint=localhost:41681
> > > --provision_endpoint=localhost:45089
> --control_endpoint=localhost:37045'.
> > > stderr: Unable to find image '
> > > ywatanabe-docker-apache.bintray.io/beam/python3:latest' locallydocker:
> > > Error response from daemon: manifest for
> > > ywatanabe-docker-apache.bintray.io/beam/python3:latest not found:
> > > manifest unknown: The named manifest is not known to the registry.See
> > > 'docker run --
> > >
> > > According to online doc, there is not parameter which controls which
> image
> > > to use .
> > >
> > > https://beam.apache.org/documentation/runners/flink/
> > >
> > > Pipeline options I am using is below.
> > >
> > >
> > >
> ============================================================================
> > > options = PipelineOptions([
> > > "--runner=FlinkRunner",
> > > "--flink_version=1.8",
> > > "--flink_master_url=localhost:8081",
> > > "--worker_harness_container_image=
> > > asia.gcr.io/PROJECTNAME/beam/python3",
> > > "--experiments=beam_fn_api"
> > > ])
> > >
> > >
> ============================================================================
> > >
> > > Perhaps is there any environment variable to specify which image to
> use ?
> > >
> > > Best Regards,
> > > Yu Watanabe
> > >
> > > --
> > > Yu Watanabe
> > > Weekend Freelancer who loves to challenge building data platform
> > > [email protected]
> > > [image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1>
> [image:
> > > Twitter icon] <https://twitter.com/yuwtennis>
> > >
> > --
> > Kyle Weaver | Software Engineer | github.com/ibzib | [email protected]
> >
>
--
Yu Watanabe
Weekend Freelancer who loves to challenge building data platform
[email protected]
[image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> [image:
Twitter icon] <https://twitter.com/yuwtennis>
