The main concern here seems to be whether using pre-release candidates would be too disruptive to our workflow. I think this is an easy hypothesis to test out--we can give using prerelease candidates a try, and if that indeed turns out to be a problem we can then do the work of trying to put together a separate test suite (TBD the balance between coverage and cost).
On Thu, Apr 20, 2023 at 10:20 AM Anand Inguva via dev <dev@beam.apache.org> wrote: > Friendly ping:) > > If anyone has experience testing the pre-released versions, your feedback > would be much appreciated. > > Thanks, > Anand > > > > On Wed, Apr 12, 2023 at 5:07 PM Yi Hu <ya...@google.com> wrote: > >> Sounds good, thanks! >> >> Best, >> Yi >> >> On Wed, Apr 12, 2023 at 2:20 PM Anand Inguva <ananding...@google.com> >> wrote: >> >>> @Yi Hu <ya...@google.com> I think adding them to Jenkins or github >>> actions is okay with me. With Github actions, since we don't use self >>> hosted runners yet, I worry that action workers might get queued up. >>> >>> Also, I plan to not run these on every commit but run it as a cron >>> job(maybe once per day) and also as trigger phrases and only on the lowest >>> and highest python version. Also, migrating this workflow to jenkins would >>> be trivial in the future once beam starts the migration. For now, I think >>> it might be best to run on jenkins. >>> >>> On Wed, Apr 12, 2023 at 1:32 PM Valentyn Tymofieiev <valen...@google.com> >>> wrote: >>> >>>> I think case in point dependency that would benefit from this testing >>>> is grpcio, which includes pre-releases, and broke us and multiple of it's >>>> released versions were yanked. https://pypi.org/project/grpcio/#history >>>> . >>>> >>>> We can look at how grpcio affected Beam previously. Couple of issues: >>>> >>>> - https://github.com/grpc/grpc/issues/30446 -- affected XLang tests >>>> - https://github.com/apache/beam/issues/23734 -- affected MacOS suites >>>> - https://github.com/apache/beam/issues/22159 -- (not detected by us, >>>> but potentially could have affected a performance test). >>>> >>>> I'm afraid a dedicated suite may not give us desired test coverage to >>>> catch regression at RC stage. >>>> >>>> On Wed, Apr 12, 2023 at 10:19 AM Yi Hu via dev <dev@beam.apache.org> >>>> wrote: >>>> >>>>> Thanks Anand, >>>>> >>>>> This would be very helpful to avoid experiencing multiple time ( >>>>> https://s.apache.org/beam-python-dependencies-pm). One thing to note >>>>> is that Beam Jenkins CI is experiencing many issues recently, mostly due >>>>> to >>>>> that multiple Jenkins plugins does not scale (draining GitHub API call >>>>> limit; disk usage, etc) so more PreCommit may add more pressures to >>>>> Jenkins >>>>> if going ahead with Option 1. As we have started GitHub Action migration, >>>>> is it considered to add these new tests to GitHub Action? >>>>> >>>>> Best, >>>>> Yi >>>>> >>>>> On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev < >>>>> dev@beam.apache.org> wrote: >>>>> >>>>>> Thanks for doing this Anand, I'm +1 on option 1 as well - I think >>>>>> having the clear signal of the normal suite succeeding and the prerelease >>>>>> one failing would be helpful and there shouldn't be too much additional >>>>>> code necessary. That makes it really easy to treat the prerelease suite >>>>>> as >>>>>> a (at least temporary) signal on needing upper bounds on our >>>>>> dependencies. >>>>>> >>>>>> Thanks, >>>>>> Danny >>>>>> >>>>>> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev < >>>>>> dev@beam.apache.org> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> For Apache Beam Python we are considering using pre-released >>>>>>> dependencies for unit testing by using the --pre flag to install >>>>>>> pre-released dependencies of packages. >>>>>>> >>>>>>> We believe that using pre-released dependencies may help us to >>>>>>> identify and resolve bugs more quickly, and to take advantage of new >>>>>>> features or bug fixes that are not yet available in stable releases. >>>>>>> However, we also understand that using pre-released dependencies may >>>>>>> introduce new risks and challenges, including potential code duplication >>>>>>> and stability issues. >>>>>>> >>>>>>> Before proceeding, we wanted to get your feedback on this approach. >>>>>>> >>>>>>> 1. Create a new PreCommit test suite and a PostCommit test suite >>>>>>> that runs tests by installing pre-released dependencies. >>>>>>> >>>>>>> Pros: >>>>>>> >>>>>>> - stable and pre-released test suites are separate and it will >>>>>>> be easier to debug if the pre-released test suite fails. >>>>>>> >>>>>>> Cons: >>>>>>> >>>>>>> - More test infra code to maintain. More tests to monitor. >>>>>>> >>>>>>> >>>>>>> 2. Make use of the current PreCommit and PostCommit test suite and >>>>>>> modify it so that it installs pre-released dependencies. >>>>>>> >>>>>>> Pros: >>>>>>> >>>>>>> - Less infra code and less tests to monitor. >>>>>>> >>>>>>> Cons: >>>>>>> >>>>>>> - Leads to noisy test signals if the pre-release candidate is >>>>>>> unstable. >>>>>>> >>>>>>> I am in favor of approach 1 since this approach would ensure that >>>>>>> any issues encountered during pre-release testing do not impact the >>>>>>> stable >>>>>>> release environment, and vice versa. >>>>>>> >>>>>>> If you have experience or done any testing work using pre-released >>>>>>> dependencies, please let me know if you took any different approaches. >>>>>>> It >>>>>>> will be really helpful. >>>>>>> >>>>>>> Thanks, >>>>>>> Anand >>>>>>> >>>>>>
