The gen_protos.py will be called while building a sdist, wheel or an editable installation. We use pytest through the tox package and during the tox build process, gen_protos.py is called during either wheel or sdist creation.
For building sdist, the process now changed from `python setup.py sdist` to `python -m build --sdist` and editable installation still happens with `pip install -e .` and rest of the development process follows more or less the same old behavior. On Thu, Oct 12, 2023 at 5:06 PM Robert Bradshaw <rober...@google.com> wrote: > On Thu, Oct 12, 2023 at 2:04 PM Anand Inguva <ananding...@google.com> > wrote: > >> I am in the process of updating the documentation at >> https://cwiki.apache.org/confluence/display/BEAM/Python+Tips related to >> setup.py/pyproject.toml changes, but yes you can't call setup.py >> directly because it might fail due to the lack of presence of beam python's >> build time dependencies. >> >> With regards to other files(eg:protos), we will follow the similar >> behavior as before(generating proros using `gen_protos.py`). >> > > Meaning this will be called automatically when needed (e.g. from pytest)? > > >> On Thu, Oct 12, 2023 at 4:01 PM Robert Bradshaw <rober...@google.com> >> wrote: >> >>> Does this change any development practices? E.g. if I clone the repo, >>> I'm assuming I couldn't run "setup.py test" anymore. What about the >>> generated files (like protos, or the yaml definitions copied from other >>> parts of the repo)? >>> >>> On Thu, Oct 12, 2023 at 12:27 PM Anand Inguva via dev < >>> dev@beam.apache.org> wrote: >>> >>>> The PR https://github.com/apache/beam/pull/28385 is merged today. If >>>> there are any observed failures, please comment on the PR and I will follow >>>> up with a forward fix. Thanks. >>>> >>>> On Fri, Sep 1, 2023 at 2:30 PM Anand Inguva <ananding...@google.com> >>>> wrote: >>>> >>>>> Since there is positive feedback from the dev community, I am going >>>>> ahead and implementing this proposal for Python SDK. >>>>> >>>>> @aus...@apache.org <aus...@apache.org> Initially let's move forward >>>>> with the setuptools as backend for building package and as part of the >>>>> future work, we can find a better backend than setuptools. >>>>> >>>>> Thanks for the feedback. >>>>> Anand >>>>> >>>>> On Mon, Aug 28, 2023 at 12:00 PM Austin Bennett <aus...@apache.org> >>>>> wrote: >>>>> >>>>>> I've thought about this a ton, but haven't been in a position to >>>>>> undertake the work. Thanks for bringing this up, @Anand Inguva >>>>>> <ananding...@google.com> ! >>>>>> >>>>>> I'd point us to https://python-poetry.org/ ... [ which is where I'd >>>>>> look take us, but I'm also not able to do all the work, so my >>>>>> suggestion/preference doensn't matter that much ] >>>>>> >>>>>> https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <- >>>>>> for info on pyproject.toml file. >>>>>> >>>>>> Notice the use of a 'lock' file is very valuable, ex: >>>>>> https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control >>>>>> >>>>>> I haven't come across `build`, that might be great too. I'd >>>>>> highlight that Poetry is pretty common across industry these days, >>>>>> rock-solid, ecosystem of interoperability, users, etc... If not >>>>>> familiar, >>>>>> PLEASE have a look at that. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev < >>>>>> dev@beam.apache.org> wrote: >>>>>> >>>>>>> +1 >>>>>>> Hi Anand, >>>>>>> I appreciate this effort. Managing python dependencies has been a >>>>>>> major pain point for me, and I think this approach would help. >>>>>>> Kerry >>>>>>> >>>>>>> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev < >>>>>>> dev@beam.apache.org> wrote: >>>>>>> >>>>>>>> Hello Beam Dev Team, >>>>>>>> >>>>>>>> I've compiled a design document >>>>>>>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1] >>>>>>>> proposing the integration of pyproject.toml into Apache Beam's Python >>>>>>>> build >>>>>>>> process. Your insights and feedback would be invaluable. >>>>>>>> >>>>>>>> What is pyproject.toml? >>>>>>>> pyproject.toml is a configuration file that specifies a project's >>>>>>>> build dependencies and other project-related metadata in a standardized >>>>>>>> format. Before pyproject.toml, Python projects often had multiple >>>>>>>> configuration files (like setup.py, setup.cfg, and requirements.txt). >>>>>>>> pyproject.toml aims to centralize these configurations into one place, >>>>>>>> making project setups more organized and straightforward. One of the >>>>>>>> significant features enabled by pyproject.toml is the ability to >>>>>>>> perform >>>>>>>> isolated builds. This ensures that build dependencies are separated >>>>>>>> from >>>>>>>> the project's runtime dependencies, leading to more consistent and >>>>>>>> reproducible builds. >>>>>>>> >>>>>>>> [1] >>>>>>>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Anand >>>>>>>> >>>>>>>
