Re: Fwd: [DISCUSSION] Use github actions for python wheels ?
Hi Tobiasz, Thank you for your work on this. I think this is making great progress and I agree that it will simplify the release process. I asked a few clarifying questions on the PR. (I do not want to repeat it here to avoid diverging the conversation.) Thank you! Ahmet On Tue, Jun 9, 2020 at 2:39 AM Tobiasz Kędzierski < tobiasz.kedzier...@polidea.com> wrote: > Hi, > > I've added some important updates to > https://github.com/apache/beam/pull/11877 and I wanted to share some > thoughts with you about possible improvements: > > During releasing a new version of Beam the script > *build_release_candidate.sh* is executed. It builds sources and puts them > into the GCS staging bucket where they are consumed by separate repository > CI jobs (beam-wheels). Then they are downloaded and processed by > *sign_hash_python_wheels.sh* script. > > By using github actions this process could be simplified as follows: > 1. Within *build_release_candidate.sh* *release* and *release candidate* > branches are pushed to the remote repository (this is done by it now). > 2. gh-actions will build sources and wheels based on these branches. > 3. *build_release_candidate.sh* could verify status of the build by using > github api and corresponding data (name of the branch, commit hash) and > after successful build download sources and wheel files from gh-action > artifacts and use them in further actions. > > Happy to know your opinion on this > > BR > Tobiasz Kędzierski > > On Mon, Jun 1, 2020 at 11:02 PM Ahmet Altay wrote: > >> > @Ahmet Altay happy to understand the extent of what >> you had in mind, maybe the extensions are not as important to plan out, as >> they're straightforwardly bolted on (ex: daily builds). More tactically >> would be valuable to ensure I understand what all needs to occur. Any >> other source of info to consume other than >> https://github.com/apache/beam-wheels and >> https://beam.apache.org/contribute/release-guide/. >> >> I added a bit more details to >> https://issues.apache.org/jira/browse/BEAM-9388 as a comment, so that it >> is preserved in the JIRA. Thank you all for working on this. >> >> On Mon, Jun 1, 2020 at 9:20 AM Kamil Wasilewski < >> kamil.wasilew...@polidea.com> wrote: >> >>> "unistd.h" C header is present on POSIX systems (MacOS and Linux), but >>> not on Windows, therefore you can't build a wheel for Windows. >>> >>> I took a look and "statesampler_fast.pyx" uses "unistd.h" only because >>> of the `usleep` function. Unless we use C++ which offers [1], the solution >>> would be to search for the equivalent of `usleep` that works on Windows. >>> >> >> +Robert Bradshaw +Valentyn Tymofieiev >> - Do you have any suggestions on how building >> wheels could work on Windows? >> >> >>> >>> [1] https://en.cppreference.com/w/cpp/thread/sleep_for >>> >> >
Re: Fwd: [DISCUSSION] Use github actions for python wheels ?
Hi, I've added some important updates to https://github.com/apache/beam/pull/11877 and I wanted to share some thoughts with you about possible improvements: During releasing a new version of Beam the script *build_release_candidate.sh* is executed. It builds sources and puts them into the GCS staging bucket where they are consumed by separate repository CI jobs (beam-wheels). Then they are downloaded and processed by *sign_hash_python_wheels.sh* script. By using github actions this process could be simplified as follows: 1. Within *build_release_candidate.sh* *release* and *release candidate* branches are pushed to the remote repository (this is done by it now). 2. gh-actions will build sources and wheels based on these branches. 3. *build_release_candidate.sh* could verify status of the build by using github api and corresponding data (name of the branch, commit hash) and after successful build download sources and wheel files from gh-action artifacts and use them in further actions. Happy to know your opinion on this BR Tobiasz Kędzierski On Mon, Jun 1, 2020 at 11:02 PM Ahmet Altay wrote: > > @Ahmet Altay happy to understand the extent of what > you had in mind, maybe the extensions are not as important to plan out, as > they're straightforwardly bolted on (ex: daily builds). More tactically > would be valuable to ensure I understand what all needs to occur. Any > other source of info to consume other than > https://github.com/apache/beam-wheels and > https://beam.apache.org/contribute/release-guide/. > > I added a bit more details to > https://issues.apache.org/jira/browse/BEAM-9388 as a comment, so that it > is preserved in the JIRA. Thank you all for working on this. > > On Mon, Jun 1, 2020 at 9:20 AM Kamil Wasilewski < > kamil.wasilew...@polidea.com> wrote: > >> "unistd.h" C header is present on POSIX systems (MacOS and Linux), but >> not on Windows, therefore you can't build a wheel for Windows. >> >> I took a look and "statesampler_fast.pyx" uses "unistd.h" only because of >> the `usleep` function. Unless we use C++ which offers [1], the solution >> would be to search for the equivalent of `usleep` that works on Windows. >> > > +Robert Bradshaw +Valentyn Tymofieiev > - Do you have any suggestions on how building > wheels could work on Windows? > > >> >> [1] https://en.cppreference.com/w/cpp/thread/sleep_for >> >
Re: Fwd: [DISCUSSION] Use github actions for python wheels ?
> @Ahmet Altay happy to understand the extent of what you had in mind, maybe the extensions are not as important to plan out, as they're straightforwardly bolted on (ex: daily builds). More tactically would be valuable to ensure I understand what all needs to occur. Any other source of info to consume other than https://github.com/apache/beam-wheels and https://beam.apache.org/contribute/release-guide/. I added a bit more details to https://issues.apache.org/jira/browse/BEAM-9388 as a comment, so that it is preserved in the JIRA. Thank you all for working on this. On Mon, Jun 1, 2020 at 9:20 AM Kamil Wasilewski < kamil.wasilew...@polidea.com> wrote: > "unistd.h" C header is present on POSIX systems (MacOS and Linux), but not > on Windows, therefore you can't build a wheel for Windows. > > I took a look and "statesampler_fast.pyx" uses "unistd.h" only because of > the `usleep` function. Unless we use C++ which offers [1], the solution > would be to search for the equivalent of `usleep` that works on Windows. > +Robert Bradshaw +Valentyn Tymofieiev - Do you have any suggestions on how building wheels could work on Windows? > > [1] https://en.cppreference.com/w/cpp/thread/sleep_for >
Re: Fwd: [DISCUSSION] Use github actions for python wheels ?
"unistd.h" C header is present on POSIX systems (MacOS and Linux), but not on Windows, therefore you can't build a wheel for Windows. I took a look and "statesampler_fast.pyx" uses "unistd.h" only because of the `usleep` function. Unless we use C++ which offers [1], the solution would be to search for the equivalent of `usleep` that works on Windows. [1] https://en.cppreference.com/w/cpp/thread/sleep_for
Re: Fwd: [DISCUSSION] Use github actions for python wheels ?
Hi everyone, Austin Benett I would like cooperate with you with introducing GitHub actions for python wheels. Since last week I am trying to figure out how to do this and I was able to build wheels for Linux and MacOS with help of https://github.com/joerick/cibuildwheel/ <https://github.com/joerick/cibuildwheel/>. I’ve created draft PR to apache/beam: https://github.com/apache/beam/pull/11877 <https://github.com/apache/beam/pull/11877> and you can check GitHub actions logs on fork PR here: https://github.com/TobKed/beam/pull/3 <https://github.com/TobKed/beam/pull/3> I am struggling with building wheels for Windows due to the error: > Cannot open include file: 'unistd.h': No such file or directory caused by file sdks/python/apache_beam/runners/worker/statesampler_fast.pyx Adding this file to be excluded by cythonize method in setup.py allowed build to be completed however I don’t think it is a solution. Mentioned error on GitHub actions: https://github.com/TobKed/beam/runs/727709583?check_suite_focus=true#step:7:977 <https://github.com/TobKed/beam/runs/727709583?check_suite_focus=true#step:7:977> and my local PR: https://github.com/TobKed/beam/pull/2 <https://github.com/TobKed/beam/pull/2> What do you think about it? Tobiasz On 2020/05/30 20:53:51, Austin Bennett wrote: > There seems to be support here for this idea. After digging through> > things, I *think* I understand the moving pieces and can address (reason I> > was digging through the code, including beam-wheels repo [1]). Though, not> > 100% that I have not overlooked bits.> > > Would be happy to pickup> > https://issues.apache.org/jira/browse/BEAM-9388 (imagine> > would get to it over the course of a month or few -- so also wouldn't want> > to take it on if others would be looking to do it sooner) -- seems> > worthwhile to reduce manual steps for the release process, as well as the> > additional implications (daily builds) if can easily manage. I have some> > experience with GitHub Actions, but not with our (BEAM) deployment/release> > process, nor much with building wheels. I assume there are sufficient docs> > available to go through for the mechanics of the latter. If picking up> > this issue, I suspect may need to bother some of those that have done> > recent releases -- that would best occur on the jira ticket, once at that> > point?> > > @Ismail -- given your experience with GitHub actions,> > any reason to think GitHub Actions not appropriate/ready for this specific> > task?> > > @Ahmet Altay happy to understand the extent of what you> > had in mind, maybe the extensions are not as important to plan out, as> > they're straightforwardly bolted on (ex: daily builds). More tactically> > would be valuable to ensure I understand what all needs to occur. Any> > other source of info to consume other than> > https://github.com/apache/beam-wheels and> > https://beam.apache.org/contribute/release-guide/.> > > Also, open to the thought that this might be taking on too much, without> > more experience with the release process and such. Do advise...?> > > > [1]> > https://lists.apache.org/thread.html/r93884eb080297647207f7d2b8a393e224029fc2c3509017886e84051%40%3Cdev.beam.apache.org%3E> > > > > > -- Forwarded message -> > From: Ismaël Mejía > > Date: Wed, Feb 26, 2020 at 12:09 PM> > Subject: Re: [DISCUSSION] Use github actions for python wheels ?> > To: dev > > > > +1 I have been migrating multiple projects into github actions recently and> > even if for some tasks it is not as mature and polished as travis it has> > proven to be way more reliable.> > > > On Wed, Feb 26, 2020 at 7:07 PM Ahmet Altay wrote:> > > > I created https://issues.apache.org/jira/browse/BEAM-9388 to explore> > > this. To be explicit and not to do cookie licking, I would not be able to> > > work on this at the moment. If anyone is interested please take it.> > > Otherwise I will try to come back and explore this when I can.> > >> > > On Tue, Feb 25, 2020 at 2:57 PM Robert Bradshaw > > > wrote:> > >> > >> I'd be in favor of this, assuming it actually simplifies things.> > >> > >> > > This is also my concern. I do think that it will simplify things, but I am> > > not certain as I am not very familiar with the github actions.> > >> > >> > >> (Note> > >> that the wheels are for several variants of linux, presumably we could> > >> do cross-compiles. Also, manylinux is a "minimal" linux
Fwd: [DISCUSSION] Use github actions for python wheels ?
There seems to be support here for this idea. After digging through things, I *think* I understand the moving pieces and can address (reason I was digging through the code, including beam-wheels repo [1]). Though, not 100% that I have not overlooked bits. Would be happy to pickup https://issues.apache.org/jira/browse/BEAM-9388 (imagine would get to it over the course of a month or few -- so also wouldn't want to take it on if others would be looking to do it sooner) -- seems worthwhile to reduce manual steps for the release process, as well as the additional implications (daily builds) if can easily manage. I have some experience with GitHub Actions, but not with our (BEAM) deployment/release process, nor much with building wheels. I assume there are sufficient docs available to go through for the mechanics of the latter. If picking up this issue, I suspect may need to bother some of those that have done recent releases -- that would best occur on the jira ticket, once at that point? @Ismail -- given your experience with GitHub actions, any reason to think GitHub Actions not appropriate/ready for this specific task? @Ahmet Altay happy to understand the extent of what you had in mind, maybe the extensions are not as important to plan out, as they're straightforwardly bolted on (ex: daily builds). More tactically would be valuable to ensure I understand what all needs to occur. Any other source of info to consume other than https://github.com/apache/beam-wheels and https://beam.apache.org/contribute/release-guide/. Also, open to the thought that this might be taking on too much, without more experience with the release process and such. Do advise...? [1] https://lists.apache.org/thread.html/r93884eb080297647207f7d2b8a393e224029fc2c3509017886e84051%40%3Cdev.beam.apache.org%3E -- Forwarded message - From: Ismaël Mejía Date: Wed, Feb 26, 2020 at 12:09 PM Subject: Re: [DISCUSSION] Use github actions for python wheels ? To: dev +1 I have been migrating multiple projects into github actions recently and even if for some tasks it is not as mature and polished as travis it has proven to be way more reliable. On Wed, Feb 26, 2020 at 7:07 PM Ahmet Altay wrote: > I created https://issues.apache.org/jira/browse/BEAM-9388 to explore > this. To be explicit and not to do cookie licking, I would not be able to > work on this at the moment. If anyone is interested please take it. > Otherwise I will try to come back and explore this when I can. > > On Tue, Feb 25, 2020 at 2:57 PM Robert Bradshaw > wrote: > >> I'd be in favor of this, assuming it actually simplifies things. > > > This is also my concern. I do think that it will simplify things, but I am > not certain as I am not very familiar with the github actions. > > >> (Note >> that the wheels are for several variants of linux, presumably we could >> do cross-compiles. Also, manylinux is a "minimal" linux specifically >> built as to produce shared object libraries compatible with a wide >> variety of distributions--we can't just assume that a shared object >> library built on one modern linux will just work on another. (But >> maybe it's sufficient to do this within a docker environment?) >> > > There will be no change in this area. Both in Both Travis and github > actions offer a comparable set of options. > > >> >> On Tue, Feb 25, 2020 at 2:23 PM Kenneth Knowles wrote: >> > >> > +1 to exploring this. >> > >> > On bui...@apache.org there is lots of discussion and general approval >> for trying it. It is enabled and used by some projects. Calcite uses it to >> build their website, for example. >> > > Great. > > >> > >> > Kenn >> > >> > >> > On Tue, Feb 25, 2020 at 2:08 PM Ahmet Altay wrote: >> >> >> >> Hi all, >> >> >> >> I recently had a chance to look at the documentation for github >> actions. I think we could use github actions instead of travis to for >> building python wheels during releases. This will have the following >> advantages: >> >> >> >> - We will eliminate one repo. (If you don't know, we have >> https://github.com/apache/beam-wheels for the sole purpose of building >> wheels file.) >> >> - Workflow will be stored in the same repo. This will prevent bit rot >> that is only discovered at release times. (happened a few times, although >> usually easy to fix.) >> >> - github actions supports ubuntu, mac, windows environments. We could >> try to build wheels for windows as well. (Travis also supports the same >> environments but we only use linux and mac environments. Maybe there are >> other blockers for building wheels for Windows.) >> >> - We could do more, like daily python builds. >> >> >> >> Downsides would be: >> >> - I do not know if github actions will require some special set of >> permissions that require an approval from infra. >> >> - Travis works fine most of the time. This might be unnecessary work. >> >> >> >> What do you think? Is this feasible, would this be useful? >> >> >> >> Ahmet >> >
Re: [DISCUSSION] Use github actions for python wheels ?
+1 I have been migrating multiple projects into github actions recently and even if for some tasks it is not as mature and polished as travis it has proven to be way more reliable. On Wed, Feb 26, 2020 at 7:07 PM Ahmet Altay wrote: > I created https://issues.apache.org/jira/browse/BEAM-9388 to explore > this. To be explicit and not to do cookie licking, I would not be able to > work on this at the moment. If anyone is interested please take it. > Otherwise I will try to come back and explore this when I can. > > On Tue, Feb 25, 2020 at 2:57 PM Robert Bradshaw > wrote: > >> I'd be in favor of this, assuming it actually simplifies things. > > > This is also my concern. I do think that it will simplify things, but I am > not certain as I am not very familiar with the github actions. > > >> (Note >> that the wheels are for several variants of linux, presumably we could >> do cross-compiles. Also, manylinux is a "minimal" linux specifically >> built as to produce shared object libraries compatible with a wide >> variety of distributions--we can't just assume that a shared object >> library built on one modern linux will just work on another. (But >> maybe it's sufficient to do this within a docker environment?) >> > > There will be no change in this area. Both in Both Travis and github > actions offer a comparable set of options. > > >> >> On Tue, Feb 25, 2020 at 2:23 PM Kenneth Knowles wrote: >> > >> > +1 to exploring this. >> > >> > On bui...@apache.org there is lots of discussion and general approval >> for trying it. It is enabled and used by some projects. Calcite uses it to >> build their website, for example. >> > > Great. > > >> > >> > Kenn >> > >> > >> > On Tue, Feb 25, 2020 at 2:08 PM Ahmet Altay wrote: >> >> >> >> Hi all, >> >> >> >> I recently had a chance to look at the documentation for github >> actions. I think we could use github actions instead of travis to for >> building python wheels during releases. This will have the following >> advantages: >> >> >> >> - We will eliminate one repo. (If you don't know, we have >> https://github.com/apache/beam-wheels for the sole purpose of building >> wheels file.) >> >> - Workflow will be stored in the same repo. This will prevent bit rot >> that is only discovered at release times. (happened a few times, although >> usually easy to fix.) >> >> - github actions supports ubuntu, mac, windows environments. We could >> try to build wheels for windows as well. (Travis also supports the same >> environments but we only use linux and mac environments. Maybe there are >> other blockers for building wheels for Windows.) >> >> - We could do more, like daily python builds. >> >> >> >> Downsides would be: >> >> - I do not know if github actions will require some special set of >> permissions that require an approval from infra. >> >> - Travis works fine most of the time. This might be unnecessary work. >> >> >> >> What do you think? Is this feasible, would this be useful? >> >> >> >> Ahmet >> >
Re: [DISCUSSION] Use github actions for python wheels ?
I created https://issues.apache.org/jira/browse/BEAM-9388 to explore this. To be explicit and not to do cookie licking, I would not be able to work on this at the moment. If anyone is interested please take it. Otherwise I will try to come back and explore this when I can. On Tue, Feb 25, 2020 at 2:57 PM Robert Bradshaw wrote: > I'd be in favor of this, assuming it actually simplifies things. This is also my concern. I do think that it will simplify things, but I am not certain as I am not very familiar with the github actions. > (Note > that the wheels are for several variants of linux, presumably we could > do cross-compiles. Also, manylinux is a "minimal" linux specifically > built as to produce shared object libraries compatible with a wide > variety of distributions--we can't just assume that a shared object > library built on one modern linux will just work on another. (But > maybe it's sufficient to do this within a docker environment?) > There will be no change in this area. Both in Both Travis and github actions offer a comparable set of options. > > On Tue, Feb 25, 2020 at 2:23 PM Kenneth Knowles wrote: > > > > +1 to exploring this. > > > > On bui...@apache.org there is lots of discussion and general approval > for trying it. It is enabled and used by some projects. Calcite uses it to > build their website, for example. > Great. > > > > Kenn > > > > > > On Tue, Feb 25, 2020 at 2:08 PM Ahmet Altay wrote: > >> > >> Hi all, > >> > >> I recently had a chance to look at the documentation for github > actions. I think we could use github actions instead of travis to for > building python wheels during releases. This will have the following > advantages: > >> > >> - We will eliminate one repo. (If you don't know, we have > https://github.com/apache/beam-wheels for the sole purpose of building > wheels file.) > >> - Workflow will be stored in the same repo. This will prevent bit rot > that is only discovered at release times. (happened a few times, although > usually easy to fix.) > >> - github actions supports ubuntu, mac, windows environments. We could > try to build wheels for windows as well. (Travis also supports the same > environments but we only use linux and mac environments. Maybe there are > other blockers for building wheels for Windows.) > >> - We could do more, like daily python builds. > >> > >> Downsides would be: > >> - I do not know if github actions will require some special set of > permissions that require an approval from infra. > >> - Travis works fine most of the time. This might be unnecessary work. > >> > >> What do you think? Is this feasible, would this be useful? > >> > >> Ahmet >
Re: [DISCUSSION] Use github actions for python wheels ?
I'd be in favor of this, assuming it actually simplifies things. (Note that the wheels are for several variants of linux, presumably we could do cross-compiles. Also, manylinux is a "minimal" linux specifically built as to produce shared object libraries compatible with a wide variety of distributions--we can't just assume that a shared object library built on one modern linux will just work on another. (But maybe it's sufficient to do this within a docker environment?) On Tue, Feb 25, 2020 at 2:23 PM Kenneth Knowles wrote: > > +1 to exploring this. > > On bui...@apache.org there is lots of discussion and general approval for > trying it. It is enabled and used by some projects. Calcite uses it to build > their website, for example. > > Kenn > > > On Tue, Feb 25, 2020 at 2:08 PM Ahmet Altay wrote: >> >> Hi all, >> >> I recently had a chance to look at the documentation for github actions. I >> think we could use github actions instead of travis to for building python >> wheels during releases. This will have the following advantages: >> >> - We will eliminate one repo. (If you don't know, we have >> https://github.com/apache/beam-wheels for the sole purpose of building >> wheels file.) >> - Workflow will be stored in the same repo. This will prevent bit rot that >> is only discovered at release times. (happened a few times, although usually >> easy to fix.) >> - github actions supports ubuntu, mac, windows environments. We could try to >> build wheels for windows as well. (Travis also supports the same >> environments but we only use linux and mac environments. Maybe there are >> other blockers for building wheels for Windows.) >> - We could do more, like daily python builds. >> >> Downsides would be: >> - I do not know if github actions will require some special set of >> permissions that require an approval from infra. >> - Travis works fine most of the time. This might be unnecessary work. >> >> What do you think? Is this feasible, would this be useful? >> >> Ahmet
Re: [DISCUSSION] Use github actions for python wheels ?
+1 to exploring this. On bui...@apache.org there is lots of discussion and general approval for trying it. It is enabled and used by some projects. Calcite uses it to build their website, for example. Kenn On Tue, Feb 25, 2020 at 2:08 PM Ahmet Altay wrote: > Hi all, > > I recently had a chance to look at the documentation for github actions. I > think we could use github actions instead of travis to for building python > wheels during releases. This will have the following advantages: > > - We will eliminate one repo. (If you don't know, we have > https://github.com/apache/beam-wheels for the sole purpose of building > wheels file.) > - Workflow will be stored in the same repo. This will prevent bit rot that > is only discovered at release times. (happened a few times, although > usually easy to fix.) > - github actions supports ubuntu, mac, windows environments. We could try > to build wheels for windows as well. (Travis also supports the same > environments but we only use linux and mac environments. Maybe there are > other blockers for building wheels for Windows.) > - We could do more, like daily python builds. > > Downsides would be: > - I do not know if github actions will require some special set of > permissions that require an approval from infra. > - Travis works fine most of the time. This might be unnecessary work. > > What do you think? Is this feasible, would this be useful? > > Ahmet >
[DISCUSSION] Use github actions for python wheels ?
Hi all, I recently had a chance to look at the documentation for github actions. I think we could use github actions instead of travis to for building python wheels during releases. This will have the following advantages: - We will eliminate one repo. (If you don't know, we have https://github.com/apache/beam-wheels for the sole purpose of building wheels file.) - Workflow will be stored in the same repo. This will prevent bit rot that is only discovered at release times. (happened a few times, although usually easy to fix.) - github actions supports ubuntu, mac, windows environments. We could try to build wheels for windows as well. (Travis also supports the same environments but we only use linux and mac environments. Maybe there are other blockers for building wheels for Windows.) - We could do more, like daily python builds. Downsides would be: - I do not know if github actions will require some special set of permissions that require an approval from infra. - Travis works fine most of the time. This might be unnecessary work. What do you think? Is this feasible, would this be useful? Ahmet
