Does the exception `ModuleNotFoundError: No module named 'dotenv' ` occur during job submission or during job running?
The argument `--pyRequirements` only works during job running, that's it will download the dependencies specified via `--pyRequirements` and add them to PYTHONPATH of the Python worker. For your problem, I guess you could try one of the following ways: - If the exception occurs during job submission (job compiling), then I guess you could try to move the `import` of `dotenv` from the header of the file to where it's used. That's turning it into a local import. Then it will avoid importing this library during job compiling. - Use option `-pyfs` [1]. - Use option `-pyarch`[2] together with option `-pyexec`[3] and ` -pyclientexec`[4]. Regards, Dian [1] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/dependency_management/#python-libraries [2] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/dependency_management/#archives [3] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/dependency_management/#python-interpreter [4] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/dependency_management/#python-interpreter-of-client On Tue, Nov 23, 2021 at 3:17 AM John Lafer <[email protected]> wrote: > Hi, > > I'm running a PyFlink 1.13.1 cluster in Docker. The pyflink-walkthrough > example from the Apache Flink Playgrounds repo (flink-playgrounds - Git > at Google <https://apache.googlesource.com/flink-playgrounds/>) works > fine. > > However, I'm unable to add a Python package dependency, like > "python-dotenv" to the Python job. I've tried the suggested techniques for > specifying dependencies in a requirements file as described in Dependency > Management > <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/dependency_management/> > (i.e., > --pyRequirements or using t_env.set_python_requirements) but they seem to > have no effect. > > For example, in my Dockerfile I copy my requirements.txt file into the > jobmanager host at /opt/flink/requirements.txt like so: > RUN echo "python-dotenv==0.19.2" >> /opt/flink/requirements.txt > I've verified its contents inside the jobmanager container. > > And then from my client machine I submit the job like this into my > docker-based cluster: > docker-compose exec jobmanager ./bin/flink run -py > /opt/pyflink-walkthrough/payment_msg_proccessing.py -d --pyRequirements > /opt/flink/requirements.txt > > However, all attempts result in the job producing the error: > ModuleNotFoundError: > No module named 'dotenv' > > I'm wondering if the technique I'm using works for Python jobs or only for > Python UDFs? > > Help would be appreciated. > >
