alxmrs commented on issue #22349: URL: https://github.com/apache/beam/issues/22349#issuecomment-1190736973
> it invokes system Python, whereas for this use case we want Python processes to run from conda Python Interesting – I haven't hit this problem. I use the docker image in the issue description (or [see here](https://github.com/google/weather-tools/blob/main/weather_mv/Dockerfile)) along with [build step in a setup.py file to install conda dependencies](https://github.com/google/weather-tools/blob/0ca1a39eb1057acf9b9a663bd122b5fb7d408228/weather_mv/setup.py#L93). I just took a look at the pangeo Dockerfile that compiles sources with go. I think this step is unnecessary; rather, I believe that there are other strategies to set up conda python on the image. This works by installing deps into the base conda environment instead of a named conda env – maybe that's the source of the problem? On @yuvipanda's suggestions: I think the status quo offers enough hooks to accomplish a lot of what Yuvi is asking for. For example, the boot sequence for Beam will eventually call `setup.py install` (and `setup.py build`). From there, you can install required dependencies, or use distutils to perform sophisticated build actions, including running [raw linux commands](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/juliaset/setup.py#L75). Having said all this, I actually don't really know what problem Yuvi / the `forge` container is trying to solve in the first place. Thus, I am probably totally off base. @cisaacstern or @yuvipanda can you help me understand the infrastructure? To add a high level note on my proposal: I think of this Beam/conda docker image (with the boot entrypoint) as the runtime environment, from which we can add further dependencies at startup time. This is unlike the typical docker approach, where we include all dependencies at the image build time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
