And oddly enough, I just checked the build we've been talking about, made some changes and was able to get it to build correctly.
See the status at the link. :shrug: https://ci-builds.apache.org/job/Heron/job/build_heron-api.jar/17/ On Mon, Jan 3, 2022 at 8:31 PM Josh Fischer <[email protected]> wrote: > The requirements.txt and Pex dependency issue will get resolved after we > move to the native bazel rules. I was in the process of creating the ticket > and it got me thinking. If we have infra go through the trouble of > installing python3-venv there still may be an issue that we are unaware of > and could cause a bit of back and forth between us. > > Nick, > Can you just build the client executables (java and python) for us and > then we can fix the build steps for the job you referenced using a docker > container in the next release? > > Saad made a good point about time since the last release. Even if we file > a ticket with infra it could take some time before they get to it. By some > time, I mean a few weeks to a month. > > On Mon, Jan 3, 2022 at 4:28 PM Nicholas Nezis <[email protected]> > wrote: > >> Dave, >> >> Could we get help? I'm not sure how to ask the Infrastructure team about >> the python3-venv package. Is there an alias? >> >> And with regards to the requirements.txt. I agree. It seems the project >> uses it in some places, but also has these dependencies listed here that >> are built. They seem to be used in the "tools/rules/pex/BUILD" script. >> Perhaps there is a reason they need to be referenced like this? Seems like >> it's maybe something originally setup that can now be improved. >> >> https://github.com/apache/incubator-heron/blob/fab089ce2c9cdbdd2d6a7ef7cda6cf22b201469c/WORKSPACE#L178-L240 >> >> On Mon, Jan 3, 2022 at 4:57 PM Dave Fisher <[email protected]> wrote: >> >> > >> > >> > > On Jan 3, 2022, at 1:25 PM, Nicholas Nezis <[email protected]> >> > wrote: >> > > >> > > I don't think this should be a blocker to cutting the release. >> Removing >> > PEX >> > > might have various consequences. >> > > >> > > We should still attempt to remove it if we think this is the issue, >> but I >> > > suspect we were running into other issues. >> > > 1. Building on non AMD64 architecture >> > > a. I'm happy to help build locally >> > > 2. Building on Jenkins which doesn't have the python3-venv package. >> > > a. Can we request the package be installed? I'm not sure how we ask >> > the >> > > build infrastructure team to add it. >> > >> > I’m surprised that venv is missing from the python setup. That’s >> somewhat >> > core to Infra’s process. >> > >> > - Send an email to [email protected] >> > - Create an INFRA Jira at issues.apache.org/jira login with your apache >> > id. >> > >> > > b. We can convert our build script to use Docker for the build. >> This >> > > might take a bit more effort compared to adding python3-venv >> > > >> > > Often we have had issues with Python compatibility. And some of those >> > > issues were related to the move from Python 2 to Python 3. Another >> > > complicating factor is that we often have trouble updating some of the >> > > dependencies because instead of using `pip3 install` to download a >> > > dependency, we instead download the source and compile the packages. >> This >> > > may provide more flexibility to run on different architectures, but >> also >> > > can add an extra layer of complexity depending on each package's >> needs. >> > >> > Why wouldn’t we just use a requirements.txt with pip3? >> > >> > > >> > > So while I think the move from Pex to the more native Bazel Python >> binary >> > > rules might be better, I don't know if it directly addresses the >> > perceived >> > > issues. >> > > >> > > On Sun, Jan 2, 2022 at 1:12 AM Ning Wang <[email protected]> >> wrote: >> > > >> > >> it seems like pex has been painful to maintain. It would be great to >> try >> > >> the official rule. Thanks! >> > >> >> > >> On Sat, Jan 1, 2022 at 11:30 AM Saad Ur Rahman < >> [email protected] >> > > >> > >> wrote: >> > >> >> > >>> I will assist as much as I can with the migration. >> > >>> >> > >>> On Sat, Jan 1, 2022 at 12:47 PM Josh Fischer <[email protected]> >> > >> wrote: >> > >>> >> > >>>> Hey All, >> > >>>> >> > >>>> After doing work on the next RC we've found some issues related to >> > >> Python >> > >>>> and Pex. I'm thinking we need to remove the custom Pex rules from >> > >> repo >> > >>>> and depend on native bazel rules. I'll create a branch within the >> > >> Heron >> > >>>> repo to start work on it. I'll most likely have some questions for >> > >>>> everyone, and I'd like feedback as we go along. >> > >>>> >> > >>>> Thanks >> > >>>> >> > >>>> On Fri, Dec 24, 2021 at 6:53 PM Saad Ur Rahman < >> > >> [email protected]> >> > >>>> wrote: >> > >>>> >> > >>>>> Thank you, Josh, that would be very interesting indeed. >> > >>>>> >> > >>>>> On Fri., Dec. 24, 2021, 5:48 p.m. Josh Fischer, < >> [email protected] >> > >>> >> > >>>>> wrote: >> > >>>>> >> > >>>>>> Saad, >> > >>>>>> >> > >>>>>> I welcome any help. If you are interested I can walk you through >> > >> the >> > >>>>>> process. >> > >>>>>> >> > >>>>>> On Fri, Dec 24, 2021 at 3:42 PM Saad Ur Rahman < >> > >>>> [email protected]> >> > >>>>>> wrote: >> > >>>>>> >> > >>>>>>> Thank you, Josh. Please let me know if there is anything I can >> do >> > >>> to >> > >>>>>> help. >> > >>>>>>> >> > >>>>>>> On Fri., Dec. 24, 2021, 2:52 p.m. Josh Fischer, < >> > >>> [email protected] >> > >>>>> >> > >>>>>>> wrote: >> > >>>>>>> >> > >>>>>>>> Nick, >> > >>>>>>>> I would appreciate the help if you are willing. >> > >>>>>>>> >> > >>>>>>>> Saad, >> > >>>>>>>> >> > >>>>>>>> Ok, I’ll start the process Sunday or Monday. >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> On Thu, Dec 23, 2021 at 7:19 PM Saad <[email protected]> >> > >>>>> wrote: >> > >>>>>>>> >> > >>>>>>>>> I feel we are in a good place to cut a release as well. >> > >>>>>>>>> >> > >>>>>>>>> On Thu., Dec. 23, 2021, 6:44 p.m. Nicholas Nezis, < >> > >>>>>>>>> [email protected]> >> > >>>>>>>>> wrote: >> > >>>>>>>>> >> > >>>>>>>>>> Yes I think we should cut a release now. The remaining PRs >> > >>> can >> > >>>>> wait >> > >>>>>>> and >> > >>>>>>>>> may >> > >>>>>>>>>> change APIs. >> > >>>>>>>>>> >> > >>>>>>>>>> I'd like to help with publishing any release artifacts that >> > >>> we >> > >>>>>>> haven't >> > >>>>>>>>>> published previously. Specifically thinking of HeronPy and >> > >>> Helm >> > >>>>>>>>> artifacts. >> > >>>>>>>>>> >> > >>>>>>>>>> On Wed, Dec 22, 2021, 9:59 AM Josh Fischer < >> > >>>> [email protected]> >> > >>>>>>>> wrote: >> > >>>>>>>>>> >> > >>>>>>>>>>> Hi, >> > >>>>>>>>>>> >> > >>>>>>>>>>> The kubernetes scheduler PRs are now completed and >> > >>> merged. I >> > >>>>>> know >> > >>>>>>> we >> > >>>>>>>>>> have >> > >>>>>>>>>>> a couple of PRs that were recently opened, do we need to >> > >>> wait >> > >>>>> for >> > >>>>>>> any >> > >>>>>>>>> of >> > >>>>>>>>>>> those before we start the voting process? >> > >>>>>>>>>>> >> > >>>>>>>>>>> - Josh >> > >>>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>> -- >> > >>>>>>>> Sent from A Mobile Device >> > >>>>>>>> >> > >>>>>>> >> > >>>>>> -- >> > >>>>>> Sent from A Mobile Device >> > >>>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> >> > >> > >> >
