The script on ci-builds.a.o is:
#!/bin/bash
set -e
set -o pipefail
# Install Python
#sudo apt-get install python3-venv
# Install bazel (linux build) because CI hosts may not have it installed
#bash scripts/ci/setup_bazel.sh linux
HERON_BUILD_VERSION=$(echo $HERON_GIT_TAG | rev | cut -c5- | rev)
#mkdir -p artifacts/$HERON_BUILD_VERSION/heron-api
#mkdir -p artifacts/$HERON_BUILD_VERSION/heron-spi
#mkdir -p artifacts/$HERON_BUILD_VERSION/heron-simulator
#mkdir -p artifacts/$HERON_BUILD_VERSION/heron-storm
bazelVersion=4.1.0
# Build v0.20.1-incubating packages for centos7 and put in artifacts folder
HERON_BUILD_USER=release-agent
#bash scripts/ci/build_maven_artifacts.sh $HERON_BUILD_VERSION artifacts
docker run --rm -w /incubator-heron -v ${PWD}:/incubator-heron
openjdk:11.0.6-jdk-buster \
bash apt-get update && apt-get install -y \
wget \
&& wget -O /tmp/bazel.sh
https://github.com/bazelbuild/bazel/releases/download/$bazelVersion/bazel-$bazelVersion-installer-linux-x86_64.sh
\
&& /incubator-heron/scripts/ci/build_maven_artifacts.sh $HERON_BUILD_VERSION
artifacts
There are several lines commented out. Please correct this to how it needs to
run and then I can work with Infra to adjust it properly.
Regards,
Dave
> On Jan 3, 2022, at 2:28 PM, Nicholas Nezis <[email protected]> wrote:
>
> Dave,
>
> Could we get help? I'm not sure how to ask the Infrastructure team about
> the python3-venv package. Is there an alias?
>
> And with regards to the requirements.txt. I agree. It seems the project
> uses it in some places, but also has these dependencies listed here that
> are built. They seem to be used in the "tools/rules/pex/BUILD" script.
> Perhaps there is a reason they need to be referenced like this? Seems like
> it's maybe something originally setup that can now be improved.
> https://github.com/apache/incubator-heron/blob/fab089ce2c9cdbdd2d6a7ef7cda6cf22b201469c/WORKSPACE#L178-L240
>
> On Mon, Jan 3, 2022 at 4:57 PM Dave Fisher <[email protected]> wrote:
>
>>
>>
>>> On Jan 3, 2022, at 1:25 PM, Nicholas Nezis <[email protected]>
>> wrote:
>>>
>>> I don't think this should be a blocker to cutting the release. Removing
>> PEX
>>> might have various consequences.
>>>
>>> We should still attempt to remove it if we think this is the issue, but I
>>> suspect we were running into other issues.
>>> 1. Building on non AMD64 architecture
>>> a. I'm happy to help build locally
>>> 2. Building on Jenkins which doesn't have the python3-venv package.
>>> a. Can we request the package be installed? I'm not sure how we ask
>> the
>>> build infrastructure team to add it.
>>
>> I’m surprised that venv is missing from the python setup. That’s somewhat
>> core to Infra’s process.
>>
>> - Send an email to [email protected]
>> - Create an INFRA Jira at issues.apache.org/jira login with your apache
>> id.
>>
>>> b. We can convert our build script to use Docker for the build. This
>>> might take a bit more effort compared to adding python3-venv
>>>
>>> Often we have had issues with Python compatibility. And some of those
>>> issues were related to the move from Python 2 to Python 3. Another
>>> complicating factor is that we often have trouble updating some of the
>>> dependencies because instead of using `pip3 install` to download a
>>> dependency, we instead download the source and compile the packages. This
>>> may provide more flexibility to run on different architectures, but also
>>> can add an extra layer of complexity depending on each package's needs.
>>
>> Why wouldn’t we just use a requirements.txt with pip3?
>>
>>>
>>> So while I think the move from Pex to the more native Bazel Python binary
>>> rules might be better, I don't know if it directly addresses the
>> perceived
>>> issues.
>>>
>>> On Sun, Jan 2, 2022 at 1:12 AM Ning Wang <[email protected]> wrote:
>>>
>>>> it seems like pex has been painful to maintain. It would be great to try
>>>> the official rule. Thanks!
>>>>
>>>> On Sat, Jan 1, 2022 at 11:30 AM Saad Ur Rahman <[email protected]
>>>
>>>> wrote:
>>>>
>>>>> I will assist as much as I can with the migration.
>>>>>
>>>>> On Sat, Jan 1, 2022 at 12:47 PM Josh Fischer <[email protected]>
>>>> wrote:
>>>>>
>>>>>> Hey All,
>>>>>>
>>>>>> After doing work on the next RC we've found some issues related to
>>>> Python
>>>>>> and Pex. I'm thinking we need to remove the custom Pex rules from
>>>> repo
>>>>>> and depend on native bazel rules. I'll create a branch within the
>>>> Heron
>>>>>> repo to start work on it. I'll most likely have some questions for
>>>>>> everyone, and I'd like feedback as we go along.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Fri, Dec 24, 2021 at 6:53 PM Saad Ur Rahman <
>>>> [email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Thank you, Josh, that would be very interesting indeed.
>>>>>>>
>>>>>>> On Fri., Dec. 24, 2021, 5:48 p.m. Josh Fischer, <[email protected]
>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Saad,
>>>>>>>>
>>>>>>>> I welcome any help. If you are interested I can walk you through
>>>> the
>>>>>>>> process.
>>>>>>>>
>>>>>>>> On Fri, Dec 24, 2021 at 3:42 PM Saad Ur Rahman <
>>>>>> [email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thank you, Josh. Please let me know if there is anything I can do
>>>>> to
>>>>>>>> help.
>>>>>>>>>
>>>>>>>>> On Fri., Dec. 24, 2021, 2:52 p.m. Josh Fischer, <
>>>>> [email protected]
>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Nick,
>>>>>>>>>> I would appreciate the help if you are willing.
>>>>>>>>>>
>>>>>>>>>> Saad,
>>>>>>>>>>
>>>>>>>>>> Ok, I’ll start the process Sunday or Monday.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Dec 23, 2021 at 7:19 PM Saad <[email protected]>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I feel we are in a good place to cut a release as well.
>>>>>>>>>>>
>>>>>>>>>>> On Thu., Dec. 23, 2021, 6:44 p.m. Nicholas Nezis, <
>>>>>>>>>>> [email protected]>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Yes I think we should cut a release now. The remaining PRs
>>>>> can
>>>>>>> wait
>>>>>>>>> and
>>>>>>>>>>> may
>>>>>>>>>>>> change APIs.
>>>>>>>>>>>>
>>>>>>>>>>>> I'd like to help with publishing any release artifacts that
>>>>> we
>>>>>>>>> haven't
>>>>>>>>>>>> published previously. Specifically thinking of HeronPy and
>>>>> Helm
>>>>>>>>>>> artifacts.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Dec 22, 2021, 9:59 AM Josh Fischer <
>>>>>> [email protected]>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> The kubernetes scheduler PRs are now completed and
>>>>> merged. I
>>>>>>>> know
>>>>>>>>> we
>>>>>>>>>>>> have
>>>>>>>>>>>>> a couple of PRs that were recently opened, do we need to
>>>>> wait
>>>>>>> for
>>>>>>>>> any
>>>>>>>>>>> of
>>>>>>>>>>>>> those before we start the voting process?
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Josh
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Sent from A Mobile Device
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Sent from A Mobile Device
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>
