Sorry, yes, you are correct, though Google does not document this. 1. Formerly I can import pyscog and requests because they are in the worker image you linked to. 2. secretmanager cannot be imported because it is not in the worker image. 3. passing the parameter --parameters sdk_container_image=$IMAGE_URL_WORKER causes the worker to use the pre-built image 4. I cannot use the same Docker image for both launcher and worker because of the ENTRYPOINT
On Sun, Nov 3, 2024 at 1:53 PM XQ Hu via user <[email protected]> wrote: > I think the problem is you do not specify sdk_container_image when running > your template. > > > https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies#run-the-template > has more details. > > Basically, you do not need to > https://github.com/paulhtremblay/data-engineering/blob/main/dataflow_/flex_proj_with_secret_manager/Dockerfile > for your template launcher. > > You can use the same image for both launcher and dataflow workers. You > only need to copy python_template_launcher to your image like > https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/dataflow/flex-templates/pipeline_with_dependencies/Dockerfile#L38. > Then you can use this image for both launcher and dataflow workers. When > running the template job, you need to add --parameters > sdk_container_image=$SDK_CONTAINER_IMAGE. > > > On Sun, Nov 3, 2024 at 4:18 PM Henry Tremblay <[email protected]> > wrote: > >> A few weeks ago I had posted a problem I had with importing the Google >> Cloud Secret Manager library in Python. >> >> Here is the problem and solution: >> >> >> https://github.com/paulhtremblay/data-engineering/tree/main/dataflow_/flex_proj_with_secret_manager >> >> -- >> Henry Tremblay >> Data Engineer, Best Buy >> > -- Henry Tremblay Data Engineer, Best Buy
