I can clean this up tonight and file a ticket. Nick, I’ll cc you in the ticket so you see what’s going on.
On Mon, Jan 3, 2022 at 5:21 PM Dave Fisher <[email protected]> wrote: > The script on ci-builds.a.o is: > > #!/bin/bash > > set -e > set -o pipefail > > # Install Python > #sudo apt-get install python3-venv > > # Install bazel (linux build) because CI hosts may not have it installed > #bash scripts/ci/setup_bazel.sh linux > > HERON_BUILD_VERSION=$(echo $HERON_GIT_TAG | rev | cut -c5- | rev) > > #mkdir -p artifacts/$HERON_BUILD_VERSION/heron-api > #mkdir -p artifacts/$HERON_BUILD_VERSION/heron-spi > #mkdir -p artifacts/$HERON_BUILD_VERSION/heron-simulator > #mkdir -p artifacts/$HERON_BUILD_VERSION/heron-storm > > bazelVersion=4.1.0 > > # Build v0.20.1-incubating packages for centos7 and put in artifacts folder > HERON_BUILD_USER=release-agent > #bash scripts/ci/build_maven_artifacts.sh $HERON_BUILD_VERSION artifacts > docker run --rm -w /incubator-heron -v ${PWD}:/incubator-heron > openjdk:11.0.6-jdk-buster \ > bash apt-get update && apt-get install -y \ > wget \ > && wget -O /tmp/bazel.sh > https://github.com/bazelbuild/bazel/releases/download/$bazelVersion/bazel-$bazelVersion-installer-linux-x86_64.sh > \ > && /incubator-heron/scripts/ci/build_maven_artifacts.sh > $HERON_BUILD_VERSION artifacts > > There are several lines commented out. Please correct this to how it needs > to run and then I can work with Infra to adjust it properly. > > Regards, > Dave > > > On Jan 3, 2022, at 2:28 PM, Nicholas Nezis <[email protected]> > wrote: > > > > Dave, > > > > Could we get help? I'm not sure how to ask the Infrastructure team about > > the python3-venv package. Is there an alias? > > > > And with regards to the requirements.txt. I agree. It seems the project > > uses it in some places, but also has these dependencies listed here that > > are built. They seem to be used in the "tools/rules/pex/BUILD" script. > > Perhaps there is a reason they need to be referenced like this? Seems > like > > it's maybe something originally setup that can now be improved. > > > https://github.com/apache/incubator-heron/blob/fab089ce2c9cdbdd2d6a7ef7cda6cf22b201469c/WORKSPACE#L178-L240 > > > > On Mon, Jan 3, 2022 at 4:57 PM Dave Fisher <[email protected]> wrote: > > > >> > >> > >>> On Jan 3, 2022, at 1:25 PM, Nicholas Nezis <[email protected]> > >> wrote: > >>> > >>> I don't think this should be a blocker to cutting the release. Removing > >> PEX > >>> might have various consequences. > >>> > >>> We should still attempt to remove it if we think this is the issue, > but I > >>> suspect we were running into other issues. > >>> 1. Building on non AMD64 architecture > >>> a. I'm happy to help build locally > >>> 2. Building on Jenkins which doesn't have the python3-venv package. > >>> a. Can we request the package be installed? I'm not sure how we ask > >> the > >>> build infrastructure team to add it. > >> > >> I’m surprised that venv is missing from the python setup. That’s > somewhat > >> core to Infra’s process. > >> > >> - Send an email to [email protected] > >> - Create an INFRA Jira at issues.apache.org/jira login with your apache > >> id. > >> > >>> b. We can convert our build script to use Docker for the build. This > >>> might take a bit more effort compared to adding python3-venv > >>> > >>> Often we have had issues with Python compatibility. And some of those > >>> issues were related to the move from Python 2 to Python 3. Another > >>> complicating factor is that we often have trouble updating some of the > >>> dependencies because instead of using `pip3 install` to download a > >>> dependency, we instead download the source and compile the packages. > This > >>> may provide more flexibility to run on different architectures, but > also > >>> can add an extra layer of complexity depending on each package's needs. > >> > >> Why wouldn’t we just use a requirements.txt with pip3? > >> > >>> > >>> So while I think the move from Pex to the more native Bazel Python > binary > >>> rules might be better, I don't know if it directly addresses the > >> perceived > >>> issues. > >>> > >>> On Sun, Jan 2, 2022 at 1:12 AM Ning Wang <[email protected]> wrote: > >>> > >>>> it seems like pex has been painful to maintain. It would be great to > try > >>>> the official rule. Thanks! > >>>> > >>>> On Sat, Jan 1, 2022 at 11:30 AM Saad Ur Rahman < > [email protected] > >>> > >>>> wrote: > >>>> > >>>>> I will assist as much as I can with the migration. > >>>>> > >>>>> On Sat, Jan 1, 2022 at 12:47 PM Josh Fischer <[email protected]> > >>>> wrote: > >>>>> > >>>>>> Hey All, > >>>>>> > >>>>>> After doing work on the next RC we've found some issues related to > >>>> Python > >>>>>> and Pex. I'm thinking we need to remove the custom Pex rules from > >>>> repo > >>>>>> and depend on native bazel rules. I'll create a branch within the > >>>> Heron > >>>>>> repo to start work on it. I'll most likely have some questions for > >>>>>> everyone, and I'd like feedback as we go along. > >>>>>> > >>>>>> Thanks > >>>>>> > >>>>>> On Fri, Dec 24, 2021 at 6:53 PM Saad Ur Rahman < > >>>> [email protected]> > >>>>>> wrote: > >>>>>> > >>>>>>> Thank you, Josh, that would be very interesting indeed. > >>>>>>> > >>>>>>> On Fri., Dec. 24, 2021, 5:48 p.m. Josh Fischer, < > [email protected] > >>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Saad, > >>>>>>>> > >>>>>>>> I welcome any help. If you are interested I can walk you through > >>>> the > >>>>>>>> process. > >>>>>>>> > >>>>>>>> On Fri, Dec 24, 2021 at 3:42 PM Saad Ur Rahman < > >>>>>> [email protected]> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Thank you, Josh. Please let me know if there is anything I can do > >>>>> to > >>>>>>>> help. > >>>>>>>>> > >>>>>>>>> On Fri., Dec. 24, 2021, 2:52 p.m. Josh Fischer, < > >>>>> [email protected] > >>>>>>> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Nick, > >>>>>>>>>> I would appreciate the help if you are willing. > >>>>>>>>>> > >>>>>>>>>> Saad, > >>>>>>>>>> > >>>>>>>>>> Ok, I’ll start the process Sunday or Monday. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Thu, Dec 23, 2021 at 7:19 PM Saad <[email protected]> > >>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> I feel we are in a good place to cut a release as well. > >>>>>>>>>>> > >>>>>>>>>>> On Thu., Dec. 23, 2021, 6:44 p.m. Nicholas Nezis, < > >>>>>>>>>>> [email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Yes I think we should cut a release now. The remaining PRs > >>>>> can > >>>>>>> wait > >>>>>>>>> and > >>>>>>>>>>> may > >>>>>>>>>>>> change APIs. > >>>>>>>>>>>> > >>>>>>>>>>>> I'd like to help with publishing any release artifacts that > >>>>> we > >>>>>>>>> haven't > >>>>>>>>>>>> published previously. Specifically thinking of HeronPy and > >>>>> Helm > >>>>>>>>>>> artifacts. > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, Dec 22, 2021, 9:59 AM Josh Fischer < > >>>>>> [email protected]> > >>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi, > >>>>>>>>>>>>> > >>>>>>>>>>>>> The kubernetes scheduler PRs are now completed and > >>>>> merged. I > >>>>>>>> know > >>>>>>>>> we > >>>>>>>>>>>> have > >>>>>>>>>>>>> a couple of PRs that were recently opened, do we need to > >>>>> wait > >>>>>>> for > >>>>>>>>> any > >>>>>>>>>>> of > >>>>>>>>>>>>> those before we start the voting process? > >>>>>>>>>>>>> > >>>>>>>>>>>>> - Josh > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Sent from A Mobile Device > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> -- > >>>>>>>> Sent from A Mobile Device > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >> > >> > > -- Sent from A Mobile Device
