The requirements.txt and Pex dependency issue will get resolved after we move to the native bazel rules. I was in the process of creating the ticket and it got me thinking. If we have infra go through the trouble of installing python3-venv there still may be an issue that we are unaware of and could cause a bit of back and forth between us.
Nick, Can you just build the client executables (java and python) for us and then we can fix the build steps for the job you referenced using a docker container in the next release? Saad made a good point about time since the last release. Even if we file a ticket with infra it could take some time before they get to it. By some time, I mean a few weeks to a month. On Mon, Jan 3, 2022 at 4:28 PM Nicholas Nezis <[email protected]> wrote: > Dave, > > Could we get help? I'm not sure how to ask the Infrastructure team about > the python3-venv package. Is there an alias? > > And with regards to the requirements.txt. I agree. It seems the project > uses it in some places, but also has these dependencies listed here that > are built. They seem to be used in the "tools/rules/pex/BUILD" script. > Perhaps there is a reason they need to be referenced like this? Seems like > it's maybe something originally setup that can now be improved. > > https://github.com/apache/incubator-heron/blob/fab089ce2c9cdbdd2d6a7ef7cda6cf22b201469c/WORKSPACE#L178-L240 > > On Mon, Jan 3, 2022 at 4:57 PM Dave Fisher <[email protected]> wrote: > > > > > > > > On Jan 3, 2022, at 1:25 PM, Nicholas Nezis <[email protected]> > > wrote: > > > > > > I don't think this should be a blocker to cutting the release. Removing > > PEX > > > might have various consequences. > > > > > > We should still attempt to remove it if we think this is the issue, > but I > > > suspect we were running into other issues. > > > 1. Building on non AMD64 architecture > > > a. I'm happy to help build locally > > > 2. Building on Jenkins which doesn't have the python3-venv package. > > > a. Can we request the package be installed? I'm not sure how we ask > > the > > > build infrastructure team to add it. > > > > I’m surprised that venv is missing from the python setup. That’s somewhat > > core to Infra’s process. > > > > - Send an email to [email protected] > > - Create an INFRA Jira at issues.apache.org/jira login with your apache > > id. > > > > > b. We can convert our build script to use Docker for the build. This > > > might take a bit more effort compared to adding python3-venv > > > > > > Often we have had issues with Python compatibility. And some of those > > > issues were related to the move from Python 2 to Python 3. Another > > > complicating factor is that we often have trouble updating some of the > > > dependencies because instead of using `pip3 install` to download a > > > dependency, we instead download the source and compile the packages. > This > > > may provide more flexibility to run on different architectures, but > also > > > can add an extra layer of complexity depending on each package's needs. > > > > Why wouldn’t we just use a requirements.txt with pip3? > > > > > > > > So while I think the move from Pex to the more native Bazel Python > binary > > > rules might be better, I don't know if it directly addresses the > > perceived > > > issues. > > > > > > On Sun, Jan 2, 2022 at 1:12 AM Ning Wang <[email protected]> wrote: > > > > > >> it seems like pex has been painful to maintain. It would be great to > try > > >> the official rule. Thanks! > > >> > > >> On Sat, Jan 1, 2022 at 11:30 AM Saad Ur Rahman < > [email protected] > > > > > >> wrote: > > >> > > >>> I will assist as much as I can with the migration. > > >>> > > >>> On Sat, Jan 1, 2022 at 12:47 PM Josh Fischer <[email protected]> > > >> wrote: > > >>> > > >>>> Hey All, > > >>>> > > >>>> After doing work on the next RC we've found some issues related to > > >> Python > > >>>> and Pex. I'm thinking we need to remove the custom Pex rules from > > >> repo > > >>>> and depend on native bazel rules. I'll create a branch within the > > >> Heron > > >>>> repo to start work on it. I'll most likely have some questions for > > >>>> everyone, and I'd like feedback as we go along. > > >>>> > > >>>> Thanks > > >>>> > > >>>> On Fri, Dec 24, 2021 at 6:53 PM Saad Ur Rahman < > > >> [email protected]> > > >>>> wrote: > > >>>> > > >>>>> Thank you, Josh, that would be very interesting indeed. > > >>>>> > > >>>>> On Fri., Dec. 24, 2021, 5:48 p.m. Josh Fischer, < > [email protected] > > >>> > > >>>>> wrote: > > >>>>> > > >>>>>> Saad, > > >>>>>> > > >>>>>> I welcome any help. If you are interested I can walk you through > > >> the > > >>>>>> process. > > >>>>>> > > >>>>>> On Fri, Dec 24, 2021 at 3:42 PM Saad Ur Rahman < > > >>>> [email protected]> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> Thank you, Josh. Please let me know if there is anything I can do > > >>> to > > >>>>>> help. > > >>>>>>> > > >>>>>>> On Fri., Dec. 24, 2021, 2:52 p.m. Josh Fischer, < > > >>> [email protected] > > >>>>> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>>> Nick, > > >>>>>>>> I would appreciate the help if you are willing. > > >>>>>>>> > > >>>>>>>> Saad, > > >>>>>>>> > > >>>>>>>> Ok, I’ll start the process Sunday or Monday. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Thu, Dec 23, 2021 at 7:19 PM Saad <[email protected]> > > >>>>> wrote: > > >>>>>>>> > > >>>>>>>>> I feel we are in a good place to cut a release as well. > > >>>>>>>>> > > >>>>>>>>> On Thu., Dec. 23, 2021, 6:44 p.m. Nicholas Nezis, < > > >>>>>>>>> [email protected]> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> Yes I think we should cut a release now. The remaining PRs > > >>> can > > >>>>> wait > > >>>>>>> and > > >>>>>>>>> may > > >>>>>>>>>> change APIs. > > >>>>>>>>>> > > >>>>>>>>>> I'd like to help with publishing any release artifacts that > > >>> we > > >>>>>>> haven't > > >>>>>>>>>> published previously. Specifically thinking of HeronPy and > > >>> Helm > > >>>>>>>>> artifacts. > > >>>>>>>>>> > > >>>>>>>>>> On Wed, Dec 22, 2021, 9:59 AM Josh Fischer < > > >>>> [email protected]> > > >>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> Hi, > > >>>>>>>>>>> > > >>>>>>>>>>> The kubernetes scheduler PRs are now completed and > > >>> merged. I > > >>>>>> know > > >>>>>>> we > > >>>>>>>>>> have > > >>>>>>>>>>> a couple of PRs that were recently opened, do we need to > > >>> wait > > >>>>> for > > >>>>>>> any > > >>>>>>>>> of > > >>>>>>>>>>> those before we start the voting process? > > >>>>>>>>>>> > > >>>>>>>>>>> - Josh > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> -- > > >>>>>>>> Sent from A Mobile Device > > >>>>>>>> > > >>>>>>> > > >>>>>> -- > > >>>>>> Sent from A Mobile Device > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > > >
