Apache Bean on GCP / Forcing to use py 3.11
Hello sorry for the partially off topic question I am running a pipeline in which one of hte dependencies need to run on py 3.11 But i dont see any options that allow me to force the python version to be used Could anyone help? Kind regards Marco
Re: Apache Bean on GCP / Forcing to use py 3.11
If you could use py 3.11 locally, you will get python 3.11 in your cloud environment as well. Is that not happening? When you run Apache Beam on GCP, the python version you are using in your local virtual environment will be used in the cloud environment as well. I believe this is true for non-GCP environments as well. On Mon, Jun 10, 2024 at 11:08 AM Sofia’s World wrote: > Hello > sorry for the partially off topic question > I am running a pipeline in which one of hte dependencies need to run on py > 3.11 > But i dont see any options that allow me to force the python version to be > used > > Could anyone help? > Kind regards > Marco >
Re: Apache Bean on GCP / Forcing to use py 3.11
Hello, Are you running your pipeline from the python 3.11 environment? If you are running from a python 3.11 environment and don't use a custom docker container image, DataflowRunner(Assuming Apache Beam on GCP means Apache Beam on DataflowRunner), will use Python 3.11. Thanks, Anand
Re: Apache Bean on GCP / Forcing to use py 3.11
Hello no i am running my pipelien on GCP directly via a flex template, configured using a Docker file Any chances to do something in the Dockerfile to force the version at runtime? Thanks On Mon, Jun 10, 2024 at 7:24 PM Anand Inguva via user wrote: > Hello, > > Are you running your pipeline from the python 3.11 environment? If you > are running from a python 3.11 environment and don't use a custom docker > container image, DataflowRunner(Assuming Apache Beam on GCP means Apache > Beam on DataflowRunner), will use Python 3.11. > > Thanks, > Anand >
Re: Apache Bean on GCP / Forcing to use py 3.11
In this case the Python version will be defined by the Python version installed in the docker image of your flex template. So, you'd have to build your flex template from a base image with Python 3.11. On Mon, Jun 10, 2024 at 12:50 PM Sofia’s World wrote: > Hello > no i am running my pipelien on GCP directly via a flex template, > configured using a Docker file > Any chances to do something in the Dockerfile to force the version at > runtime? > Thanks > > On Mon, Jun 10, 2024 at 7:24 PM Anand Inguva via user < > user@beam.apache.org> wrote: > >> Hello, >> >> Are you running your pipeline from the python 3.11 environment? If you >> are running from a python 3.11 environment and don't use a custom docker >> container image, DataflowRunner(Assuming Apache Beam on GCP means Apache >> Beam on DataflowRunner), will use Python 3.11. >> >> Thanks, >> Anand >> >
Re: Apache Bean on GCP / Forcing to use py 3.11
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies is a great example. On Mon, Jun 10, 2024 at 4:28 PM Valentyn Tymofieiev via user < user@beam.apache.org> wrote: > In this case the Python version will be defined by the Python version > installed in the docker image of your flex template. So, you'd have to > build your flex template from a base image with Python 3.11. > > On Mon, Jun 10, 2024 at 12:50 PM Sofia’s World > wrote: > >> Hello >> no i am running my pipelien on GCP directly via a flex template, >> configured using a Docker file >> Any chances to do something in the Dockerfile to force the version at >> runtime? >> Thanks >> >> On Mon, Jun 10, 2024 at 7:24 PM Anand Inguva via user < >> user@beam.apache.org> wrote: >> >>> Hello, >>> >>> Are you running your pipeline from the python 3.11 environment? If you >>> are running from a python 3.11 environment and don't use a custom docker >>> container image, DataflowRunner(Assuming Apache Beam on GCP means Apache >>> Beam on DataflowRunner), will use Python 3.11. >>> >>> Thanks, >>> Anand >>> >>
Re: Apache Bean on GCP / Forcing to use py 3.11
thanks all, it seemed to work but now i am getting a different problem, having issues in building pyarrow... Step #0 - "build-shareloader-template": Step #4 - "dftester-image": :36: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html Step #0 - "build-shareloader-template": Step #4 - "dftester-image": WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section' Step #0 - "build-shareloader-template": Step #4 - "dftester-image": Traceback (most recent call last): Step #0 - "build-shareloader-template": Step #4 - "dftester-image": File "/tmp/pip-build-env-meihcxsp/overlay/lib/python3.11/site-packages/setuptools_scm/_integration/pyproject_reading.py", line 36, in read_pyproject Step #0 - "build-shareloader-template": Step #4 - "dftester-image": section = defn.get("tool", {})[tool_name] Step #0 - "build-shareloader-template": Step #4 - "dftester-image": ^^^ Step #0 - "build-shareloader-template": Step #4 - "dftester-image": KeyError: 'setuptools_scm' Step #0 - "build-shareloader-template": Step #4 - "dftester-image": running bdist_wheel It is somehow getting messed up with a toml ? Could anyone advise? thanks Marco On Tue, Jun 11, 2024 at 1:00 AM XQ Hu via user wrote: > > https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies > is a great example. > > On Mon, Jun 10, 2024 at 4:28 PM Valentyn Tymofieiev via user < > user@beam.apache.org> wrote: > >> In this case the Python version will be defined by the Python version >> installed in the docker image of your flex template. So, you'd have to >> build your flex template from a base image with Python 3.11. >> >> On Mon, Jun 10, 2024 at 12:50 PM Sofia’s World >> wrote: >> >>> Hello >>> no i am running my pipelien on GCP directly via a flex template, >>> configured using a Docker file >>> Any chances to do something in the Dockerfile to force the version at >>> runtime? >>> Thanks >>> >>> On Mon, Jun 10, 2024 at 7:24 PM Anand Inguva via user < >>> user@beam.apache.org> wrote: >>> Hello, Are you running your pipeline from the python 3.11 environment? If you are running from a python 3.11 environment and don't use a custom docker container image, DataflowRunner(Assuming Apache Beam on GCP means Apache Beam on DataflowRunner), will use Python 3.11. Thanks, Anand >>>
Re: Apache Bean on GCP / Forcing to use py 3.11
Can you share your Dockerfile? On Tue, Jun 11, 2024 at 4:43 PM Sofia’s World wrote: > thanks all, it seemed to work but now i am getting a different problem, > having issues in building pyarrow... > > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > :36: DeprecationWarning: pkg_resources is deprecated as an API. See > https://setuptools.pypa.io/en/latest/pkg_resources.html > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml > does not contain a tool.setuptools_scm section' > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > Traceback (most recent call last): > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > File > "/tmp/pip-build-env-meihcxsp/overlay/lib/python3.11/site-packages/setuptools_scm/_integration/pyproject_reading.py", > line 36, in read_pyproject > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > section = defn.get("tool", {})[tool_name] > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > ^^^ > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > KeyError: 'setuptools_scm' > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > running bdist_wheel > > > > > It is somehow getting messed up with a toml ? > > > Could anyone advise? > > thanks > > Marco > > > > > > On Tue, Jun 11, 2024 at 1:00 AM XQ Hu via user > wrote: > >> >> https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies >> is a great example. >> >> On Mon, Jun 10, 2024 at 4:28 PM Valentyn Tymofieiev via user < >> user@beam.apache.org> wrote: >> >>> In this case the Python version will be defined by the Python version >>> installed in the docker image of your flex template. So, you'd have to >>> build your flex template from a base image with Python 3.11. >>> >>> On Mon, Jun 10, 2024 at 12:50 PM Sofia’s World >>> wrote: >>> Hello no i am running my pipelien on GCP directly via a flex template, configured using a Docker file Any chances to do something in the Dockerfile to force the version at runtime? Thanks On Mon, Jun 10, 2024 at 7:24 PM Anand Inguva via user < user@beam.apache.org> wrote: > Hello, > > Are you running your pipeline from the python 3.11 environment? If > you are running from a python 3.11 environment and don't use a custom > docker container image, DataflowRunner(Assuming Apache Beam on GCP means > Apache Beam on DataflowRunner), will use Python 3.11. > > Thanks, > Anand >
Re: Apache Bean on GCP / Forcing to use py 3.11
Any reason to use this? RUN pip install avro-python3 pyarrow==0.15.1 apache-beam[gcp]==2.30.0 pandas-datareader==0.9.0 It is typically recommended to use the latest Beam and build the docker image using the requirements released for each Beam, for example, https://github.com/apache/beam/blob/release-2.56.0/sdks/python/container/py311/base_image_requirements.txt On Wed, Jun 12, 2024 at 1:31 AM Sofia’s World wrote: > Sure, apologies, it crossed my mind it would have been useful to refert to > it > > so this is the docker file > > > https://github.com/mmistroni/GCP_Experiments/edit/master/dataflow/shareloader/Dockerfile_tester > > I was using a setup.py as well, but then i commented out the usage in the > dockerfile after checking some flex templates which said it is not needed > > > https://github.com/mmistroni/GCP_Experiments/blob/master/dataflow/shareloader/setup_dftester.py > > thanks in advance > Marco > > > > > > > > On Tue, Jun 11, 2024 at 10:54 PM XQ Hu wrote: > >> Can you share your Dockerfile? >> >> On Tue, Jun 11, 2024 at 4:43 PM Sofia’s World >> wrote: >> >>> thanks all, it seemed to work but now i am getting a different problem, >>> having issues in building pyarrow... >>> >>> Step #0 - "build-shareloader-template": Step #4 - "dftester-image": >>> :36: DeprecationWarning: pkg_resources is deprecated as an API. See >>> https://setuptools.pypa.io/en/latest/pkg_resources.html >>> Step #0 - "build-shareloader-template": Step #4 - "dftester-image": >>> WARNING setuptools_scm.pyproject_reading toml section missing >>> 'pyproject.toml does not contain a tool.setuptools_scm section' >>> Step #0 - "build-shareloader-template": Step #4 - "dftester-image": >>> Traceback (most recent call last): >>> Step #0 - "build-shareloader-template": Step #4 - "dftester-image": >>> File >>> "/tmp/pip-build-env-meihcxsp/overlay/lib/python3.11/site-packages/setuptools_scm/_integration/pyproject_reading.py", >>> line 36, in read_pyproject >>> Step #0 - "build-shareloader-template": Step #4 - "dftester-image": >>> section = defn.get("tool", {})[tool_name] >>> Step #0 - "build-shareloader-template": Step #4 - "dftester-image": >>> ^^^ >>> Step #0 - "build-shareloader-template": Step #4 - "dftester-image": >>> KeyError: 'setuptools_scm' >>> Step #0 - "build-shareloader-template": Step #4 - "dftester-image": >>> running bdist_wheel >>> >>> >>> >>> >>> It is somehow getting messed up with a toml ? >>> >>> >>> Could anyone advise? >>> >>> thanks >>> >>> Marco >>> >>> >>> >>> >>> >>> On Tue, Jun 11, 2024 at 1:00 AM XQ Hu via user >>> wrote: >>> https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies is a great example. On Mon, Jun 10, 2024 at 4:28 PM Valentyn Tymofieiev via user < user@beam.apache.org> wrote: > In this case the Python version will be defined by the Python version > installed in the docker image of your flex template. So, you'd have to > build your flex template from a base image with Python 3.11. > > On Mon, Jun 10, 2024 at 12:50 PM Sofia’s World > wrote: > >> Hello >> no i am running my pipelien on GCP directly via a flex template, >> configured using a Docker file >> Any chances to do something in the Dockerfile to force the version at >> runtime? >> Thanks >> >> On Mon, Jun 10, 2024 at 7:24 PM Anand Inguva via user < >> user@beam.apache.org> wrote: >> >>> Hello, >>> >>> Are you running your pipeline from the python 3.11 environment? If >>> you are running from a python 3.11 environment and don't use a custom >>> docker container image, DataflowRunner(Assuming Apache Beam on GCP means >>> Apache Beam on DataflowRunner), will use Python 3.11. >>> >>> Thanks, >>> Anand >>> >>
Re: Apache Bean on GCP / Forcing to use py 3.11
Many thanks Hu, worked like a charm few qq so in my reqs.txt i should put all beam requirements PLUS my own? and in the setup.py, shall i just declare "apache-beam[gcp]==2.54.0", # Must match the version in `Dockerfile``. thanks and kind regards Marco On Wed, Jun 12, 2024 at 1:48 PM XQ Hu wrote: > Any reason to use this? > > RUN pip install avro-python3 pyarrow==0.15.1 apache-beam[gcp]==2.30.0 > pandas-datareader==0.9.0 > > It is typically recommended to use the latest Beam and build the docker > image using the requirements released for each Beam, for example, > https://github.com/apache/beam/blob/release-2.56.0/sdks/python/container/py311/base_image_requirements.txt > > On Wed, Jun 12, 2024 at 1:31 AM Sofia’s World wrote: > >> Sure, apologies, it crossed my mind it would have been useful to refert >> to it >> >> so this is the docker file >> >> >> https://github.com/mmistroni/GCP_Experiments/edit/master/dataflow/shareloader/Dockerfile_tester >> >> I was using a setup.py as well, but then i commented out the usage in the >> dockerfile after checking some flex templates which said it is not needed >> >> >> https://github.com/mmistroni/GCP_Experiments/blob/master/dataflow/shareloader/setup_dftester.py >> >> thanks in advance >> Marco >> >> >> >> >> >> >> >> On Tue, Jun 11, 2024 at 10:54 PM XQ Hu wrote: >> >>> Can you share your Dockerfile? >>> >>> On Tue, Jun 11, 2024 at 4:43 PM Sofia’s World >>> wrote: >>> thanks all, it seemed to work but now i am getting a different problem, having issues in building pyarrow... Step #0 - "build-shareloader-template": Step #4 - "dftester-image": :36: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html Step #0 - "build-shareloader-template": Step #4 - "dftester-image": WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section' Step #0 - "build-shareloader-template": Step #4 - "dftester-image": Traceback (most recent call last): Step #0 - "build-shareloader-template": Step #4 - "dftester-image": File "/tmp/pip-build-env-meihcxsp/overlay/lib/python3.11/site-packages/setuptools_scm/_integration/pyproject_reading.py", line 36, in read_pyproject Step #0 - "build-shareloader-template": Step #4 - "dftester-image": section = defn.get("tool", {})[tool_name] Step #0 - "build-shareloader-template": Step #4 - "dftester-image": ^^^ Step #0 - "build-shareloader-template": Step #4 - "dftester-image": KeyError: 'setuptools_scm' Step #0 - "build-shareloader-template": Step #4 - "dftester-image": running bdist_wheel It is somehow getting messed up with a toml ? Could anyone advise? thanks Marco On Tue, Jun 11, 2024 at 1:00 AM XQ Hu via user wrote: > > https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies > is a great example. > > On Mon, Jun 10, 2024 at 4:28 PM Valentyn Tymofieiev via user < > user@beam.apache.org> wrote: > >> In this case the Python version will be defined by the Python version >> installed in the docker image of your flex template. So, you'd have to >> build your flex template from a base image with Python 3.11. >> >> On Mon, Jun 10, 2024 at 12:50 PM Sofia’s World >> wrote: >> >>> Hello >>> no i am running my pipelien on GCP directly via a flex template, >>> configured using a Docker file >>> Any chances to do something in the Dockerfile to force the version >>> at runtime? >>> Thanks >>> >>> On Mon, Jun 10, 2024 at 7:24 PM Anand Inguva via user < >>> user@beam.apache.org> wrote: >>> Hello, Are you running your pipeline from the python 3.11 environment? If you are running from a python 3.11 environment and don't use a custom docker container image, DataflowRunner(Assuming Apache Beam on GCP means Apache Beam on DataflowRunner), will use Python 3.11. Thanks, Anand >>>
Re: Apache Bean on GCP / Forcing to use py 3.11
I recommend to put all top-level dependencies for your pipeline in setup.py install_requires section, and autogenerate the requirements.txt, which would then include all transitive dependencies and ensure reproducible builds. For approaches to generate the requirements.txt file from top level requirements specified in the setup.py file, see: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies#optional-update-the-dependencies-in-the-requirements-file-and-rebuild-the-docker-images . Valentyn On Thu, Jun 13, 2024 at 9:52 PM Sofia’s World wrote: > Many thanks Hu, worked like a charm > > few qq > so in my reqs.txt i should put all beam requirements PLUS my own? > > and in the setup.py, shall i just declare > > "apache-beam[gcp]==2.54.0", # Must match the version in `Dockerfile``. > > thanks and kind regards > Marco > > > > > > > On Wed, Jun 12, 2024 at 1:48 PM XQ Hu wrote: > >> Any reason to use this? >> >> RUN pip install avro-python3 pyarrow==0.15.1 apache-beam[gcp]==2.30.0 >> pandas-datareader==0.9.0 >> >> It is typically recommended to use the latest Beam and build the docker >> image using the requirements released for each Beam, for example, >> https://github.com/apache/beam/blob/release-2.56.0/sdks/python/container/py311/base_image_requirements.txt >> >> On Wed, Jun 12, 2024 at 1:31 AM Sofia’s World >> wrote: >> >>> Sure, apologies, it crossed my mind it would have been useful to refert >>> to it >>> >>> so this is the docker file >>> >>> >>> https://github.com/mmistroni/GCP_Experiments/edit/master/dataflow/shareloader/Dockerfile_tester >>> >>> I was using a setup.py as well, but then i commented out the usage in >>> the dockerfile after checking some flex templates which said it is not >>> needed >>> >>> >>> https://github.com/mmistroni/GCP_Experiments/blob/master/dataflow/shareloader/setup_dftester.py >>> >>> thanks in advance >>> Marco >>> >>> >>> >>> >>> >>> >>> >>> On Tue, Jun 11, 2024 at 10:54 PM XQ Hu wrote: >>> Can you share your Dockerfile? On Tue, Jun 11, 2024 at 4:43 PM Sofia’s World wrote: > thanks all, it seemed to work but now i am getting a different > problem, having issues in building pyarrow... > > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > :36: DeprecationWarning: pkg_resources is deprecated as an API. > See https://setuptools.pypa.io/en/latest/pkg_resources.html > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > WARNING setuptools_scm.pyproject_reading toml section missing > 'pyproject.toml does not contain a tool.setuptools_scm section' > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > Traceback (most recent call last): > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > File > "/tmp/pip-build-env-meihcxsp/overlay/lib/python3.11/site-packages/setuptools_scm/_integration/pyproject_reading.py", > line 36, in read_pyproject > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > section = defn.get("tool", {})[tool_name] > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > ^^^ > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > KeyError: 'setuptools_scm' > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > running bdist_wheel > > > > > It is somehow getting messed up with a toml ? > > > Could anyone advise? > > thanks > > Marco > > > > > > On Tue, Jun 11, 2024 at 1:00 AM XQ Hu via user > wrote: > >> >> https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies >> is a great example. >> >> On Mon, Jun 10, 2024 at 4:28 PM Valentyn Tymofieiev via user < >> user@beam.apache.org> wrote: >> >>> In this case the Python version will be defined by the Python >>> version installed in the docker image of your flex template. So, you'd >>> have to build your flex template from a base image with Python 3.11. >>> >>> On Mon, Jun 10, 2024 at 12:50 PM Sofia’s World >>> wrote: >>> Hello no i am running my pipelien on GCP directly via a flex template, configured using a Docker file Any chances to do something in the Dockerfile to force the version at runtime? Thanks On Mon, Jun 10, 2024 at 7:24 PM Anand Inguva via user < user@beam.apache.org> wrote: > Hello, > > Are you running your pipeline from the python 3.11 environment? > If you are running from a python 3.11 environment and don't use a > custom > docker c
Re: Apache Bean on GCP / Forcing to use py 3.11
Sorry, i cheered up too early i can successfully build the image however, at runtime the code fails always with this exception and i cannot figure out why i mimicked the sample directory structure mypackage --- __init__,py dftester.py obb_utils.py dataflow_tester_main.py this is the content of my dataflow_tester_main.py from mypackage import dftester import logging if __name__ == '__main__': logging.getLogger().setLevel(logging.INFO) dftester.run() and this is my dockerfile https://github.com/mmistroni/GCP_Experiments/blob/master/dataflow/shareloader/Dockerfile_tester and at the bottom if this email my exception I am puzzled on where the error is coming from as i have almost copied this sample https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/dataflow/flex-templates/pipeline_with_dependencies/main.py thanks and regards Marco Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 115, in create_harness _load_main_session(semi_persistent_directory) File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 354, in _load_main_session pickler.load_session(session_file) File "/usr/local/lib/python3.11/site-packages/apache_beam/internal/pickler.py", line 65, in load_session return desired_pickle_lib.load_session(file_path) ^^ File "/usr/local/lib/python3.11/site-packages/apache_beam/internal/dill_pickler.py", line 446, in load_session return dill.load_session(file_path) File "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 368, in load_session module = unpickler.load() File "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 472, in load obj = StockUnpickler.load(self) ^ File "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 462, in find_class return StockUnpickler.find_class(self, module, name) ^ ModuleNotFoundError: No module named 'modules' On Fri, Jun 14, 2024 at 5:52 AM Sofia’s World wrote: > Many thanks Hu, worked like a charm > > few qq > so in my reqs.txt i should put all beam requirements PLUS my own? > > and in the setup.py, shall i just declare > > "apache-beam[gcp]==2.54.0", # Must match the version in `Dockerfile``. > > thanks and kind regards > Marco > > > > > > > On Wed, Jun 12, 2024 at 1:48 PM XQ Hu wrote: > >> Any reason to use this? >> >> RUN pip install avro-python3 pyarrow==0.15.1 apache-beam[gcp]==2.30.0 >> pandas-datareader==0.9.0 >> >> It is typically recommended to use the latest Beam and build the docker >> image using the requirements released for each Beam, for example, >> https://github.com/apache/beam/blob/release-2.56.0/sdks/python/container/py311/base_image_requirements.txt >> >> On Wed, Jun 12, 2024 at 1:31 AM Sofia’s World >> wrote: >> >>> Sure, apologies, it crossed my mind it would have been useful to refert >>> to it >>> >>> so this is the docker file >>> >>> >>> https://github.com/mmistroni/GCP_Experiments/edit/master/dataflow/shareloader/Dockerfile_tester >>> >>> I was using a setup.py as well, but then i commented out the usage in >>> the dockerfile after checking some flex templates which said it is not >>> needed >>> >>> >>> https://github.com/mmistroni/GCP_Experiments/blob/master/dataflow/shareloader/setup_dftester.py >>> >>> thanks in advance >>> Marco >>> >>> >>> >>> >>> >>> >>> >>> On Tue, Jun 11, 2024 at 10:54 PM XQ Hu wrote: >>> Can you share your Dockerfile? On Tue, Jun 11, 2024 at 4:43 PM Sofia’s World wrote: > thanks all, it seemed to work but now i am getting a different > problem, having issues in building pyarrow... > > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > :36: DeprecationWarning: pkg_resources is deprecated as an API. > See https://setuptools.pypa.io/en/latest/pkg_resources.html > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > WARNING setuptools_scm.pyproject_reading toml section missing > 'pyproject.toml does not contain a tool.setuptools_scm section' > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > Traceback (most recent call last): > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > File > "/tmp/pip-build-env-meihcxsp/overlay/lib/python3.11/site-packages/setuptools_scm/_integration/pyproject_reading.py", > line 36, in read_pyproject > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > section = defn.get("tool", {})[tool_name] > Step #0 - "build-shareloader-template": Step #4 - "dftester-image": > ^^^ > Step #0 - "build-shareloader-template": Step #4 - "dfteste
Re: Apache Bean on GCP / Forcing to use py 3.11
Your pipeline launcher refers to a package named 'modules', but this package is not available in the runtime environment. On Sat, Jun 15, 2024 at 11:17 AM Sofia’s World wrote: > Sorry, i cheered up too early > i can successfully build the image however, at runtime the code fails > always with this exception and i cannot figure out why > > i mimicked the sample directory structure > > > mypackage >--- __init__,py >dftester.py >obb_utils.py > > dataflow_tester_main.py > > this is the content of my dataflow_tester_main.py > > from mypackage import dftester > import logging > if __name__ == '__main__': > logging.getLogger().setLevel(logging.INFO) > dftester.run() > > > and this is my dockerfile > > > https://github.com/mmistroni/GCP_Experiments/blob/master/dataflow/shareloader/Dockerfile_tester > > and at the bottom if this email my exception > I am puzzled on where the error is coming from as i have almost copied > this sample > https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/dataflow/flex-templates/pipeline_with_dependencies/main.py > > thanks and regards > Marco > > > > > > > > > > > > Traceback (most recent call last): File > "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker_main.py", > line 115, in create_harness _load_main_session(semi_persistent_directory) > File > "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker_main.py", > line 354, in _load_main_session pickler.load_session(session_file) File > "/usr/local/lib/python3.11/site-packages/apache_beam/internal/pickler.py", > line 65, in load_session return desired_pickle_lib.load_session(file_path) > ^^ File > "/usr/local/lib/python3.11/site-packages/apache_beam/internal/dill_pickler.py", > line 446, in load_session return dill.load_session(file_path) > File > "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 368, in > load_session module = unpickler.load() File > "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 472, in load > obj = StockUnpickler.load(self) ^ File > "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 462, in > find_class return StockUnpickler.find_class(self, module, name) > ^ ModuleNotFoundError: No > module named 'modules' > > > > > > > > On Fri, Jun 14, 2024 at 5:52 AM Sofia’s World wrote: > >> Many thanks Hu, worked like a charm >> >> few qq >> so in my reqs.txt i should put all beam requirements PLUS my own? >> >> and in the setup.py, shall i just declare >> >> "apache-beam[gcp]==2.54.0", # Must match the version in `Dockerfile``. >> >> thanks and kind regards >> Marco >> >> >> >> >> >> >> On Wed, Jun 12, 2024 at 1:48 PM XQ Hu wrote: >> >>> Any reason to use this? >>> >>> RUN pip install avro-python3 pyarrow==0.15.1 apache-beam[gcp]==2.30.0 >>> pandas-datareader==0.9.0 >>> >>> It is typically recommended to use the latest Beam and build the docker >>> image using the requirements released for each Beam, for example, >>> https://github.com/apache/beam/blob/release-2.56.0/sdks/python/container/py311/base_image_requirements.txt >>> >>> On Wed, Jun 12, 2024 at 1:31 AM Sofia’s World >>> wrote: >>> Sure, apologies, it crossed my mind it would have been useful to refert to it so this is the docker file https://github.com/mmistroni/GCP_Experiments/edit/master/dataflow/shareloader/Dockerfile_tester I was using a setup.py as well, but then i commented out the usage in the dockerfile after checking some flex templates which said it is not needed https://github.com/mmistroni/GCP_Experiments/blob/master/dataflow/shareloader/setup_dftester.py thanks in advance Marco On Tue, Jun 11, 2024 at 10:54 PM XQ Hu wrote: > Can you share your Dockerfile? > > On Tue, Jun 11, 2024 at 4:43 PM Sofia’s World > wrote: > >> thanks all, it seemed to work but now i am getting a different >> problem, having issues in building pyarrow... >> >> Step #0 - "build-shareloader-template": Step #4 - "dftester-image": >> :36: DeprecationWarning: pkg_resources is deprecated as an API. >> See https://setuptools.pypa.io/en/latest/pkg_resources.html >> Step #0 - "build-shareloader-template": Step #4 - "dftester-image": >> WARNING setuptools_scm.pyproject_reading toml section missing >> 'pyproject.toml does not contain a tool.setuptools_scm section' >> Step #0 - "build-shareloader-template": Step #4 - "dftester-image": >> Traceback (most recent call last): >> Step #0 - "build-shareloader-template": Step #4 - "dftester-image": >>File >> "/tmp/pip-build-env-meihcxsp/overlay/lib/python3.11/site-packages/setuptools_scm/_integration/pyproject_re
Re: Apache Bean on GCP / Forcing to use py 3.11
Valentin, many thanks... i actually spotted the reference in teh setup file However , after correcting it, i am still at square 1 where somehow my runtime environment does not see it.. so i added some debugging to my Dockerfile to check if i forgot to copy something, and here's the output, where i can see the mypackage has been copied here's my directory structure mypackage __init__.py obbutils.py launcher.py __init__.py dataflow_tester.py setup_dftester.py (copied to setup.py) i can see the directory structure has been maintained when i copy my files to docker as i added some debug to my dockerfile Step #0 - "dftester-image": Removing intermediate container 4c4e763289d2 Step #0 - "dftester-image": ---> cda378f70a9e Step #0 - "dftester-image": Step 6/23 : COPY requirements.txt . Step #0 - "dftester-image": ---> 9a43da08b013 Step #0 - "dftester-image": Step 7/23 : COPY setup_dftester.py setup.py Step #0 - "dftester-image": ---> 5a6bf71df052 Step #0 - "dftester-image": Step 8/23 : COPY dataflow_tester.py . Step #0 - "dftester-image": ---> 82cfe1f1f9ed Step #0 - "dftester-image": Step 9/23 : COPY mypackage mypackage Step #0 - "dftester-image": ---> d86497b791d0 Step #0 - "dftester-image": Step 10/23 : COPY __init__.py ${WORKDIR}/__init__.py Step #0 - "dftester-image": ---> 337d149d64c7 Step #0 - "dftester-image": Step 11/23 : RUN echo '- listing workdir' Step #0 - "dftester-image": ---> Running in 9d97d8a64319 Step #0 - "dftester-image": - listing workdir Step #0 - "dftester-image": Removing intermediate container 9d97d8a64319 Step #0 - "dftester-image": ---> bc9a6a2aa462 Step #0 - "dftester-image": Step 12/23 : RUN ls -la ${WORKDIR} Step #0 - "dftester-image": ---> Running in cf164108f9d6 Step #0 - "dftester-image": total 24 Step #0 - "dftester-image": drwxr-xr-x 1 root root 4096 Jun 16 08:59 . Step #0 - "dftester-image": drwxr-xr-x 1 root root 4096 Jun 16 08:59 .. Step #0 - "dftester-image": -rw-r--r-- 1 root root0 Jun 16 08:57 __init__.py Step #0 - "dftester-image": -rw-r--r-- 1 root root 135 Jun 16 08:57 dataflow_tester.py Step #0 - "dftester-image": drwxr-xr-x 2 root root 4096 Jun 16 08:59 mypackage Step #0 - "dftester-image": -rw-r--r-- 1 root root 64 Jun 16 08:57 requirements.txt Step #0 - "dftester-image": -rw-r--r-- 1 root root 736 Jun 16 08:57 setup.py Step #0 - "dftester-image": Removing intermediate container cf164108f9d6 Step #0 - "dftester-image": ---> eb1a080b7948 Step #0 - "dftester-image": Step 13/23 : RUN echo '--- listing modules -' Step #0 - "dftester-image": ---> Running in 884f03dd81d6 Step #0 - "dftester-image": --- listing modules - Step #0 - "dftester-image": Removing intermediate container 884f03dd81d6 Step #0 - "dftester-image": ---> 9f6f7e27bd2f Step #0 - "dftester-image": Step 14/23 : RUN ls -la ${WORKDIR}/mypackage Step #0 - "dftester-image": ---> Running in bd74ade37010 Step #0 - "dftester-image": total 16 Step #0 - "dftester-image": drwxr-xr-x 2 root root 4096 Jun 16 08:59 . Step #0 - "dftester-image": drwxr-xr-x 1 root root 4096 Jun 16 08:59 .. Step #0 - "dftester-image": -rw-r--r-- 1 root root0 Jun 16 08:57 __init__.py Step #0 - "dftester-image": -rw-r--r-- 1 root root 1442 Jun 16 08:57 launcher.py Step #0 - "dftester-image": -rw-r--r-- 1 root root 607 Jun 16 08:57 obb_utils.py Step #0 - "dftester-image": Removing intermediate container bd74ade37010 i have this in my setup.py REQUIRED_PACKAGES = [ 'openbb', "apache-beam[gcp]", # Must match the version in `Dockerfile``. 'sendgrid', 'pandas_datareader', 'vaderSentiment', 'numpy', 'bs4', 'lxml', 'pandas_datareader', 'beautifulsoup4', 'xlrd', 'openpyxl' ] setuptools.setup( name='mypackage', version='0.0.1', description='Shres Runner Package.', install_requires=REQUIRED_PACKAGES, packages=setuptools.find_packages() ) and this is my dataflow_tester.py from mypackage import launcher import logging if __name__ == '__main__': logging.getLogger().setLevel(logging.INFO) launcher.run() have compared my setup vs https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies and all looks the same (apart from my copying the __init__.,py fromo the directory where the main file(dataflow_tester.py) resides Would you know how else can i debug what is going on and why my mypackages subdirectory is not being seen? Kind regars Marco On Sat, Jun 15, 2024 at 7:27 PM Valentyn Tymofieiev via user < user@beam.apache.org> wrote: > Your pipeline launcher refers to a package named 'modules', but this > package is not available in the runtime environment. > > On Sat, Jun 15, 2024 at 11:17 AM Sofia’s World > wrote: > >> Sorry, i cheered up too early >> i can successfully build the image however, at runtime the code fails >> always with this exception and i cannot figure out why >> >> i mimicked the sample directory structure >> >> >> mypackage
Re: Apache Bean on GCP / Forcing to use py 3.11
What is the error message now? You can easily ssh to your docker container and check everything is installed correctly by: docker run --rm -it --entrypoint=/bin/bash $CUSTOM_CONTAINER_IMAGE On Sun, Jun 16, 2024 at 5:18 AM Sofia’s World wrote: > Valentin, many thanks... i actually spotted the reference in teh setup file > However , after correcting it, i am still at square 1 where somehow my > runtime environment does not see it.. so i added some debugging to my > Dockerfile to check if i forgot to copy something, > and here's the output, where i can see the mypackage has been copied > > here's my directory structure > > mypackage > __init__.py > obbutils.py > launcher.py > __init__.py > dataflow_tester.py > setup_dftester.py (copied to setup.py) > > i can see the directory structure has been maintained when i copy my files > to docker as i added some debug to my dockerfile > > Step #0 - "dftester-image": Removing intermediate container 4c4e763289d2 > Step #0 - "dftester-image": ---> cda378f70a9e > Step #0 - "dftester-image": Step 6/23 : COPY requirements.txt . > Step #0 - "dftester-image": ---> 9a43da08b013 > Step #0 - "dftester-image": Step 7/23 : COPY setup_dftester.py setup.py > Step #0 - "dftester-image": ---> 5a6bf71df052 > Step #0 - "dftester-image": Step 8/23 : COPY dataflow_tester.py . > Step #0 - "dftester-image": ---> 82cfe1f1f9ed > Step #0 - "dftester-image": Step 9/23 : COPY mypackage mypackage > Step #0 - "dftester-image": ---> d86497b791d0 > Step #0 - "dftester-image": Step 10/23 : COPY __init__.py > ${WORKDIR}/__init__.py > Step #0 - "dftester-image": ---> 337d149d64c7 > Step #0 - "dftester-image": Step 11/23 : RUN echo '- listing workdir' > Step #0 - "dftester-image": ---> Running in 9d97d8a64319 > Step #0 - "dftester-image": - listing workdir > Step #0 - "dftester-image": Removing intermediate container 9d97d8a64319 > Step #0 - "dftester-image": ---> bc9a6a2aa462 > Step #0 - "dftester-image": Step 12/23 : RUN ls -la ${WORKDIR} > Step #0 - "dftester-image": ---> Running in cf164108f9d6 > Step #0 - "dftester-image": total 24 > Step #0 - "dftester-image": drwxr-xr-x 1 root root 4096 Jun 16 08:59 . > Step #0 - "dftester-image": drwxr-xr-x 1 root root 4096 Jun 16 08:59 .. > Step #0 - "dftester-image": -rw-r--r-- 1 root root0 Jun 16 08:57 > __init__.py > Step #0 - "dftester-image": -rw-r--r-- 1 root root 135 Jun 16 08:57 > dataflow_tester.py > Step #0 - "dftester-image": drwxr-xr-x 2 root root 4096 Jun 16 08:59 > mypackage > Step #0 - "dftester-image": -rw-r--r-- 1 root root 64 Jun 16 08:57 > requirements.txt > Step #0 - "dftester-image": -rw-r--r-- 1 root root 736 Jun 16 08:57 > setup.py > Step #0 - "dftester-image": Removing intermediate container cf164108f9d6 > Step #0 - "dftester-image": ---> eb1a080b7948 > Step #0 - "dftester-image": Step 13/23 : RUN echo '--- listing modules > -' > Step #0 - "dftester-image": ---> Running in 884f03dd81d6 > Step #0 - "dftester-image": --- listing modules - > Step #0 - "dftester-image": Removing intermediate container 884f03dd81d6 > Step #0 - "dftester-image": ---> 9f6f7e27bd2f > Step #0 - "dftester-image": Step 14/23 : RUN ls -la ${WORKDIR}/mypackage > Step #0 - "dftester-image": ---> Running in bd74ade37010 > Step #0 - "dftester-image": total 16 > Step #0 - "dftester-image": drwxr-xr-x 2 root root 4096 Jun 16 08:59 . > Step #0 - "dftester-image": drwxr-xr-x 1 root root 4096 Jun 16 08:59 .. > Step #0 - "dftester-image": -rw-r--r-- 1 root root0 Jun 16 08:57 > __init__.py > Step #0 - "dftester-image": -rw-r--r-- 1 root root 1442 Jun 16 08:57 > launcher.py > Step #0 - "dftester-image": -rw-r--r-- 1 root root 607 Jun 16 08:57 > obb_utils.py > Step #0 - "dftester-image": Removing intermediate container bd74ade37010 > > > i have this in my setup.py > > REQUIRED_PACKAGES = [ > 'openbb', > "apache-beam[gcp]", # Must match the version in `Dockerfile``. > 'sendgrid', > 'pandas_datareader', > 'vaderSentiment', > 'numpy', > 'bs4', > 'lxml', > 'pandas_datareader', > 'beautifulsoup4', > 'xlrd', > 'openpyxl' > ] > > > setuptools.setup( > name='mypackage', > version='0.0.1', > description='Shres Runner Package.', > install_requires=REQUIRED_PACKAGES, > packages=setuptools.find_packages() > ) > > > and this is my dataflow_tester.py > > from mypackage import launcher > import logging > if __name__ == '__main__': > logging.getLogger().setLevel(logging.INFO) > launcher.run() > > > > have compared my setup vs > https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies > and all looks the same (apart from my copying the __init__.,py fromo the > directory where the main file(dataflow_tester.py) resides > > Would you know how else can i debug what is going on and why my > mypackages subdirectory is not being seen? > > Kind regars > Marco > > > > > On Sat, Jun 15, 2024 at 7:27 PM Valentyn
Re: Apache Bean on GCP / Forcing to use py 3.11
Error is same...- see bottom - i have tried to ssh in the container and the directory is setup as expected.. so not quite sure where the issue is i will try to start from the pipeline with dependencies sample and work out from there w.o bothering the list thanks again for following up Marco Could not load main session. Inspect which external dependencies are used in the main module of your pipeline. Verify that corresponding packages are installed in the pipeline runtime environment and their installed versions match the versions used in pipeline submission environment. For more information, see: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/ Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 115, in create_harness _load_main_session(semi_persistent_directory) File "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 354, in _load_main_session pickler.load_session(session_file) File "/usr/local/lib/python3.11/site-packages/apache_beam/internal/pickler.py", line 65, in load_session return desired_pickle_lib.load_session(file_path) ^^ File "/usr/local/lib/python3.11/site-packages/apache_beam/internal/dill_pickler.py", line 446, in load_session return dill.load_session(file_path) File "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 368, in load_session module = unpickler.load() File "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 472, in load obj = StockUnpickler.load(self) ^ File "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 827, in _import_module return getattr(__import__(module, None, None, [obj]), obj) ^ ModuleNotFoundError: No module named 'mypackage' On Sun, 16 Jun 2024, 14:50 XQ Hu via user, wrote: > What is the error message now? > You can easily ssh to your docker container and check everything is > installed correctly by: > docker run --rm -it --entrypoint=/bin/bash $CUSTOM_CONTAINER_IMAGE > > > On Sun, Jun 16, 2024 at 5:18 AM Sofia’s World wrote: > >> Valentin, many thanks... i actually spotted the reference in teh setup >> file >> However , after correcting it, i am still at square 1 where somehow my >> runtime environment does not see it.. so i added some debugging to my >> Dockerfile to check if i forgot to copy something, >> and here's the output, where i can see the mypackage has been copied >> >> here's my directory structure >> >> mypackage >> __init__.py >> obbutils.py >> launcher.py >> __init__.py >> dataflow_tester.py >> setup_dftester.py (copied to setup.py) >> >> i can see the directory structure has been maintained when i copy my >> files to docker as i added some debug to my dockerfile >> >> Step #0 - "dftester-image": Removing intermediate container 4c4e763289d2 >> Step #0 - "dftester-image": ---> cda378f70a9e >> Step #0 - "dftester-image": Step 6/23 : COPY requirements.txt . >> Step #0 - "dftester-image": ---> 9a43da08b013 >> Step #0 - "dftester-image": Step 7/23 : COPY setup_dftester.py setup.py >> Step #0 - "dftester-image": ---> 5a6bf71df052 >> Step #0 - "dftester-image": Step 8/23 : COPY dataflow_tester.py . >> Step #0 - "dftester-image": ---> 82cfe1f1f9ed >> Step #0 - "dftester-image": Step 9/23 : COPY mypackage mypackage >> Step #0 - "dftester-image": ---> d86497b791d0 >> Step #0 - "dftester-image": Step 10/23 : COPY __init__.py >> ${WORKDIR}/__init__.py >> Step #0 - "dftester-image": ---> 337d149d64c7 >> Step #0 - "dftester-image": Step 11/23 : RUN echo '- listing workdir' >> Step #0 - "dftester-image": ---> Running in 9d97d8a64319 >> Step #0 - "dftester-image": - listing workdir >> Step #0 - "dftester-image": Removing intermediate container 9d97d8a64319 >> Step #0 - "dftester-image": ---> bc9a6a2aa462 >> Step #0 - "dftester-image": Step 12/23 : RUN ls -la ${WORKDIR} >> Step #0 - "dftester-image": ---> Running in cf164108f9d6 >> Step #0 - "dftester-image": total 24 >> Step #0 - "dftester-image": drwxr-xr-x 1 root root 4096 Jun 16 08:59 . >> Step #0 - "dftester-image": drwxr-xr-x 1 root root 4096 Jun 16 08:59 .. >> Step #0 - "dftester-image": -rw-r--r-- 1 root root0 Jun 16 08:57 >> __init__.py >> Step #0 - "dftester-image": -rw-r--r-- 1 root root 135 Jun 16 08:57 >> dataflow_tester.py >> Step #0 - "dftester-image": drwxr-xr-x 2 root root 4096 Jun 16 08:59 >> mypackage >> Step #0 - "dftester-image": -rw-r--r-- 1 root root 64 Jun 16 08:57 >> requirements.txt >> Step #0 - "dftester-image": -rw-r--r-- 1 root root 736 Jun 16 08:57 >> setup.py >> Step #0 - "dftester-image": Removing intermediate container cf164108f9d6 >> Step #0 - "dftester-image": ---> eb1a080b7948 >> Step #0 - "dftester-image": Step 13/23 : RUN echo '--- listing modules >> -' >> Step #0 - "dftester-image": ---> Running in 884f03dd8
Re: Apache Bean on GCP / Forcing to use py 3.11
You have “mypackage” incorrectly built. Please check and confirm that. Utkarsh On Sun, Jun 16, 2024 at 12:48 PM Sofia’s World wrote: > Error is same...- see bottom - > i have tried to ssh in the container and the directory is setup as > expected.. so not quite sure where the issue is > i will try to start from the pipeline with dependencies sample and work > out from there w.o bothering the list > > thanks again for following up > Marco > > Could not load main session. Inspect which external dependencies are used > in the main module of your pipeline. Verify that corresponding packages are > installed in the pipeline runtime environment and their installed versions > match the versions used in pipeline submission environment. For more > information, see: > https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/ > Traceback (most recent call last): File > "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker_main.py", > line 115, in create_harness _load_main_session(semi_persistent_directory) > File > "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker_main.py", > line 354, in _load_main_session pickler.load_session(session_file) File > "/usr/local/lib/python3.11/site-packages/apache_beam/internal/pickler.py", > line 65, in load_session return desired_pickle_lib.load_session(file_path) > ^^ File > "/usr/local/lib/python3.11/site-packages/apache_beam/internal/dill_pickler.py", > line 446, in load_session return dill.load_session(file_path) > File > "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 368, in > load_session module = unpickler.load() File > "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 472, in load > obj = StockUnpickler.load(self) ^ File > "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 827, in > _import_module return getattr(__import__(module, None, None, [obj]), obj) > ^ ModuleNotFoundError: No module named > 'mypackage' > > > > On Sun, 16 Jun 2024, 14:50 XQ Hu via user, wrote: > >> What is the error message now? >> You can easily ssh to your docker container and check everything is >> installed correctly by: >> docker run --rm -it --entrypoint=/bin/bash $CUSTOM_CONTAINER_IMAGE >> >> >> On Sun, Jun 16, 2024 at 5:18 AM Sofia’s World >> wrote: >> >>> Valentin, many thanks... i actually spotted the reference in teh setup >>> file >>> However , after correcting it, i am still at square 1 where somehow my >>> runtime environment does not see it.. so i added some debugging to my >>> Dockerfile to check if i forgot to copy something, >>> and here's the output, where i can see the mypackage has been copied >>> >>> here's my directory structure >>> >>> mypackage >>> __init__.py >>> obbutils.py >>> launcher.py >>> __init__.py >>> dataflow_tester.py >>> setup_dftester.py (copied to setup.py) >>> >>> i can see the directory structure has been maintained when i copy my >>> files to docker as i added some debug to my dockerfile >>> >>> Step #0 - "dftester-image": Removing intermediate container 4c4e763289d2 >>> Step #0 - "dftester-image": ---> cda378f70a9e >>> Step #0 - "dftester-image": Step 6/23 : COPY requirements.txt . >>> Step #0 - "dftester-image": ---> 9a43da08b013 >>> Step #0 - "dftester-image": Step 7/23 : COPY setup_dftester.py setup.py >>> Step #0 - "dftester-image": ---> 5a6bf71df052 >>> Step #0 - "dftester-image": Step 8/23 : COPY dataflow_tester.py . >>> Step #0 - "dftester-image": ---> 82cfe1f1f9ed >>> Step #0 - "dftester-image": Step 9/23 : COPY mypackage mypackage >>> Step #0 - "dftester-image": ---> d86497b791d0 >>> Step #0 - "dftester-image": Step 10/23 : COPY __init__.py >>> ${WORKDIR}/__init__.py >>> Step #0 - "dftester-image": ---> 337d149d64c7 >>> Step #0 - "dftester-image": Step 11/23 : RUN echo '- listing workdir' >>> Step #0 - "dftester-image": ---> Running in 9d97d8a64319 >>> Step #0 - "dftester-image": - listing workdir >>> Step #0 - "dftester-image": Removing intermediate container 9d97d8a64319 >>> Step #0 - "dftester-image": ---> bc9a6a2aa462 >>> Step #0 - "dftester-image": Step 12/23 : RUN ls -la ${WORKDIR} >>> Step #0 - "dftester-image": ---> Running in cf164108f9d6 >>> Step #0 - "dftester-image": total 24 >>> Step #0 - "dftester-image": drwxr-xr-x 1 root root 4096 Jun 16 08:59 . >>> Step #0 - "dftester-image": drwxr-xr-x 1 root root 4096 Jun 16 08:59 .. >>> Step #0 - "dftester-image": -rw-r--r-- 1 root root0 Jun 16 08:57 >>> __init__.py >>> Step #0 - "dftester-image": -rw-r--r-- 1 root root 135 Jun 16 08:57 >>> dataflow_tester.py >>> Step #0 - "dftester-image": drwxr-xr-x 2 root root 4096 Jun 16 08:59 >>> mypackage >>> Step #0 - "dftester-image": -rw-r--r-- 1 root root 64 Jun 16 08:57 >>> requirements.txt >>> Step #0 - "dftester-image": -rw-r--r-- 1 root root 736 Jun 16 08:57 >>> s
Re: Apache Bean on GCP / Forcing to use py 3.11
Thanks. It appears that i did not read fully the documentation and i missed this in my dataflow flex-template run , '--parameters' , 'sdk_container_image=$_SDK_CONTAINER_IMAGE' All my other jobs use a dodgy docker file which does not require the parameter above... I should be fine for the time being, at least my pipeline is not plagued anymore by import errors thanks all for help ing out kind regards Marco On Sun, Jun 16, 2024 at 6:27 PM Utkarsh Parekh wrote: > You have “mypackage” incorrectly built. Please check and confirm that. > > Utkarsh > > On Sun, Jun 16, 2024 at 12:48 PM Sofia’s World > wrote: > >> Error is same...- see bottom - >> i have tried to ssh in the container and the directory is setup as >> expected.. so not quite sure where the issue is >> i will try to start from the pipeline with dependencies sample and work >> out from there w.o bothering the list >> >> thanks again for following up >> Marco >> >> Could not load main session. Inspect which external dependencies are used >> in the main module of your pipeline. Verify that corresponding packages are >> installed in the pipeline runtime environment and their installed versions >> match the versions used in pipeline submission environment. For more >> information, see: >> https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/ >> Traceback (most recent call last): File >> "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker_main.py", >> line 115, in create_harness _load_main_session(semi_persistent_directory) >> File >> "/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker_main.py", >> line 354, in _load_main_session pickler.load_session(session_file) File >> "/usr/local/lib/python3.11/site-packages/apache_beam/internal/pickler.py", >> line 65, in load_session return desired_pickle_lib.load_session(file_path) >> ^^ File >> "/usr/local/lib/python3.11/site-packages/apache_beam/internal/dill_pickler.py", >> line 446, in load_session return dill.load_session(file_path) >> File >> "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 368, in >> load_session module = unpickler.load() File >> "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 472, in load >> obj = StockUnpickler.load(self) ^ File >> "/usr/local/lib/python3.11/site-packages/dill/_dill.py", line 827, in >> _import_module return getattr(__import__(module, None, None, [obj]), obj) >> ^ ModuleNotFoundError: No module named >> 'mypackage' >> >> >> >> On Sun, 16 Jun 2024, 14:50 XQ Hu via user, wrote: >> >>> What is the error message now? >>> You can easily ssh to your docker container and check everything is >>> installed correctly by: >>> docker run --rm -it --entrypoint=/bin/bash $CUSTOM_CONTAINER_IMAGE >>> >>> >>> On Sun, Jun 16, 2024 at 5:18 AM Sofia’s World >>> wrote: >>> Valentin, many thanks... i actually spotted the reference in teh setup file However , after correcting it, i am still at square 1 where somehow my runtime environment does not see it.. so i added some debugging to my Dockerfile to check if i forgot to copy something, and here's the output, where i can see the mypackage has been copied here's my directory structure mypackage __init__.py obbutils.py launcher.py __init__.py dataflow_tester.py setup_dftester.py (copied to setup.py) i can see the directory structure has been maintained when i copy my files to docker as i added some debug to my dockerfile Step #0 - "dftester-image": Removing intermediate container 4c4e763289d2 Step #0 - "dftester-image": ---> cda378f70a9e Step #0 - "dftester-image": Step 6/23 : COPY requirements.txt . Step #0 - "dftester-image": ---> 9a43da08b013 Step #0 - "dftester-image": Step 7/23 : COPY setup_dftester.py setup.py Step #0 - "dftester-image": ---> 5a6bf71df052 Step #0 - "dftester-image": Step 8/23 : COPY dataflow_tester.py . Step #0 - "dftester-image": ---> 82cfe1f1f9ed Step #0 - "dftester-image": Step 9/23 : COPY mypackage mypackage Step #0 - "dftester-image": ---> d86497b791d0 Step #0 - "dftester-image": Step 10/23 : COPY __init__.py ${WORKDIR}/__init__.py Step #0 - "dftester-image": ---> 337d149d64c7 Step #0 - "dftester-image": Step 11/23 : RUN echo '- listing workdir' Step #0 - "dftester-image": ---> Running in 9d97d8a64319 Step #0 - "dftester-image": - listing workdir Step #0 - "dftester-image": Removing intermediate container 9d97d8a64319 Step #0 - "dftester-image": ---> bc9a6a2aa462 Step #0 - "dftester-image": Step 12/23 : RUN ls -la ${WORKDIR} Step #0 - "dftester-image": ---> Running in cf164108f9d6 Step #0 - "dftester-image": total 24 >>>