Update: some blockers were discovered and we have fixes, but I need help getting fixes reviewed and cherrypicked into the release branch. I'd appreciate any attention on the following:
Fix certain dependency errors and add analysis: https://github.com/apache/beam/pull/7700 Cherrypick the above to release branch: https://github.com/apache/beam/pull/7707 Cherrypick website fix to get green signal: https://github.com/apache/beam/pull/7713 Kenn On Fri, Jan 25, 2019 at 4:16 PM Kenneth Knowles <k...@apache.org> wrote: > Just a quick update - I'm working through the RC generating process still. > I've run into issues in the scripts / gradle release plugin and some flakes > but proceeding manually seems to be going OK. > > Kenn > > On Thu, Jan 24, 2019 at 12:21 PM Kenneth Knowles <k...@google.com> wrote: > >> My mistake - the problem was that the Dataflow container image was not >> yet built, but I did not dig deep enough into StackDriver to find that >> message. >> >> All issues are resolved and all PRs against the release branch are now >> merged. I am now in process of building those containers and RC1. >> >> Kenn >> >> On Wed, Jan 23, 2019 at 5:25 PM Ahmet Altay <al...@google.com> wrote: >> >>> Python precommit cron job ( >>> https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PreCommit_Python_Cron/) >>> seems to be healthy. Could you share one of the PRs that you noticed the >>> flakiness, I would be interested in debugging. >>> >>> On Wed, Jan 23, 2019 at 12:25 PM Kenneth Knowles <k...@google.com> wrote: >>> >>>> I've seen many PRs that I am reviewing with flakiness problems in the >>>> python precommit IT. Anyone have any insights? >>>> >>>> Kenn >>>> >>>> On Wed, Jan 23, 2019 at 12:23 PM Kenneth Knowles <k...@google.com> >>>> wrote: >>>> >>>>> Thanks for the reminder. I was reading from >>>>> https://issues.apache.org/jira/projects/BEAM/versions/12344540. RC1 >>>>> also gated on clearing >>>>> https://github.com/apache/beam/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+base%3Arelease-2.10.0 >>>>> . >>>>> >>>>> Kenn >>>>> >>>>> On Wed, Jan 23, 2019 at 9:44 AM Maximilian Michels <m...@apache.org> >>>>> wrote: >>>>> >>>>>> What about the revert of "Parse SDK-unknown pipeline options"? >>>>>> https://github.com/apache/beam/pull/7564 >>>>>> >>>>>> Should we merge this for the release? >>>>>> >>>>>> On 23.01.19 11:56, Scott Wegner wrote: >>>>>> > Cherry-pick PR for the last-remaining issue: >>>>>> > https://github.com/apache/beam/pull/7603 >>>>>> > >>>>>> > On Wed, Jan 23, 2019 at 7:15 AM Kenneth Knowles <k...@apache.org >>>>>> > <mailto:k...@apache.org>> wrote: >>>>>> > >>>>>> > The last remaining issue is split into a non-blocker and a >>>>>> revert that is >>>>>> > confirmed to fix the issue. >>>>>> > >>>>>> > Once https://github.com/apache/beam/pull/7600 is merged and >>>>>> cherry-picked, I >>>>>> > will cut RC1 today. >>>>>> > >>>>>> > Kenn >>>>>> > >>>>>> > On Tue, Jan 22, 2019 at 6:03 PM Kenneth Knowles < >>>>>> k...@apache.org >>>>>> > <mailto:k...@apache.org>> wrote: >>>>>> > >>>>>> > OK. There is just one release blocker remaining; >>>>>> > https://issues.apache.org/jira/browse/BEAM-6354 >>>>>> > >>>>>> > I have no insights yet, but I am bisecting. It was healthy >>>>>> in 2.9.0. >>>>>> > >>>>>> > Kenn >>>>>> > >>>>>> > On Tue, Jan 22, 2019 at 9:38 AM Scott Wegner < >>>>>> sweg...@google.com >>>>>> > <mailto:sweg...@google.com>> wrote: >>>>>> > >>>>>> > The rollback for BEAM-6352 is now in and cherry-picked >>>>>> into the >>>>>> > release branch. >>>>>> > >>>>>> > On Fri, Jan 18, 2019 at 9:04 AM Scott Wegner < >>>>>> sc...@apache.org >>>>>> > <mailto:sc...@apache.org>> wrote: >>>>>> > >>>>>> > For BEAM-6352, I have a rollback ready for review: >>>>>> > https://github.com/apache/beam/pull/7540 >>>>>> > Conversation about the decision to rollback vs. >>>>>> roll-forward for >>>>>> > this change is on the JIRA issue. >>>>>> > >>>>>> > On Fri, Jan 18, 2019 at 8:22 AM Maximilian Michels >>>>>> > <m...@apache.org <mailto:m...@apache.org>> wrote: >>>>>> > >>>>>> > I've created the revert for the pipeline >>>>>> options parsing >>>>>> > which we agreed on: >>>>>> > https://github.com/apache/beam/pull/7564 >>>>>> > >>>>>> > On 17.01.19 15:16, Maximilian Michels wrote: >>>>>> > > An issue with the Flink Runner when >>>>>> restarting streaming >>>>>> > pipelines: >>>>>> > > >>>>>> https://jira.apache.org/jira/browse/BEAM-6460 >>>>>> > > >>>>>> > > Looks like it will be easy to fix by >>>>>> invalidating the >>>>>> > Jackson cache. >>>>>> > > >>>>>> > > -Max >>>>>> > > >>>>>> > > On 16.01.19 23:00, Kenneth Knowles wrote: >>>>>> > >> Quick update on this. There are three >>>>>> remaining issues: >>>>>> > >> >>>>>> > >> - >>>>>> https://issues.apache.org/jira/browse/BEAM-6407: A >>>>>> > DirectRunner self-check >>>>>> > >> was broken from 2.8.0 to 2.9.0 - PR looks >>>>>> good modulo >>>>>> > our infra flakes >>>>>> > >> - >>>>>> https://issues.apache.org/jira/browse/BEAM-6354: >>>>>> > PAssert + DirectRunner + >>>>>> > >> Unbounded data busted? Investigation not >>>>>> started >>>>>> > >> - >>>>>> https://issues.apache.org/jira/browse/BEAM-6352: >>>>>> > Watch was broken from >>>>>> > >> 2.8.0 to 2.9.0 - will rollback if no >>>>>> forward fix by the >>>>>> > time everything else >>>>>> > >> is resolved >>>>>> > >> >>>>>> > >> Kenn >>>>>> > >> >>>>>> > >> On Wed, Jan 16, 2019 at 6:00 AM Kenneth >>>>>> Knowles >>>>>> > <k...@apache.org <mailto:k...@apache.org> >>>>>> > >> <mailto:k...@apache.org <mailto: >>>>>> k...@apache.org>>> wrote: >>>>>> > >> >>>>>> > >> Thanks, Ismaël! >>>>>> > >> >>>>>> > >> On Wed, Jan 16, 2019 at 2:13 AM Ismaël >>>>>> Mejía >>>>>> > <ieme...@gmail.com <mailto:ieme...@gmail.com> >>>>>> > >> <mailto:ieme...@gmail.com >>>>>> > <mailto:ieme...@gmail.com>>> wrote: >>>>>> > >> >>>>>> > >> Ok since there were not many issues >>>>>> I did the >>>>>> > 'update' for the >>>>>> > >> misplaced issues to version 2.10. >>>>>> We are good to >>>>>> > go. New resolved >>>>>> > >> issues in master musg go now into >>>>>> 2.11.0 >>>>>> > >> >>>>>> > >> On Wed, Jan 16, 2019 at 10:38 AM >>>>>> Ismaël Mejía >>>>>> > <ieme...@gmail.com <mailto:ieme...@gmail.com> >>>>>> > >> <mailto:ieme...@gmail.com >>>>>> > <mailto:ieme...@gmail.com>>> wrote: >>>>>> > >> > >>>>>> > >> > This means that the tickets >>>>>> resolved and >>>>>> > marked for 2.11 since January >>>>>> > >> > 2 should be reviewed and >>>>>> retargetted to >>>>>> > version 2.10. >>>>>> > >> > So this is a call for action for >>>>>> committers >>>>>> > who have merged fixes >>>>>> > >> > after the cut to update the >>>>>> tickets if required. >>>>>> > >> > >>>>>> > >> > Ismaël >>>>>> > >> > >>>>>> > >> > On Tue, Jan 15, 2019 at 9:22 PM >>>>>> Kenneth >>>>>> > Knowles <k...@apache.org <mailto: >>>>>> k...@apache.org> >>>>>> > >> <mailto:k...@apache.org >>>>>> > <mailto:k...@apache.org>>> wrote: >>>>>> > >> > > >>>>>> > >> > > As a heads up, I did not >>>>>> realize that the >>>>>> > release guide specified a >>>>>> > >> custom process for starting a >>>>>> release branch. It >>>>>> > makes sense; >>>>>> > >> cut_release_branch.sh consolidates >>>>>> knowledge >>>>>> > about all the places the >>>>>> > >> version is hardcoded in the >>>>>> codebase. To keep >>>>>> > the history simple, I will >>>>>> > >> re-cut the release branch at the >>>>>> point where >>>>>> > master moved from >>>>>> > >> 2.10.0-SNAPSHOT to 2.11.0-SNAPSHOT. >>>>>> All PRs to >>>>>> > the branch have been >>>>>> > >> cherry-picked from master, so they >>>>>> will all be >>>>>> > incorporated without any >>>>>> > >> action by their authors. >>>>>> > >> > > >>>>>> > >> > > Kenn >>>>>> > >> > > >>>>>> > >> > > On Tue, Jan 15, 2019 at 10:31 >>>>>> AM Kenneth >>>>>> > Knowles <k...@google.com <mailto:k...@google.com> >>>>>> > >> <mailto:k...@google.com <mailto: >>>>>> k...@google.com>>> >>>>>> > wrote: >>>>>> > >> > >> >>>>>> > >> > >> I'm on it. >>>>>> > >> > >> >>>>>> > >> > >> On Tue, Jan 15, 2019 at 8:10 >>>>>> AM Ismaël >>>>>> > Mejía <ieme...@gmail.com <mailto: >>>>>> ieme...@gmail.com> >>>>>> > >> <mailto:ieme...@gmail.com >>>>>> > <mailto:ieme...@gmail.com>>> wrote: >>>>>> > >> > >>> >>>>>> > >> > >>> There is also another issue, >>>>>> after the >>>>>> > 2.10.0 branch cut some >>>>>> > >> > >>> identifier in the build was >>>>>> not changed >>>>>> > and the Apache Beam >>>>>> > >> Snapshots >>>>>> > >> > >>> keep generating SNAPSHOTS >>>>>> for 2.10.0 >>>>>> > instead of the now current >>>>>> > >> > >>> 2.11.0-SNAPSHOT. Can >>>>>> somebody PTAL? >>>>>> > >> > >>> >>>>>> > >> > >>> On Thu, Jan 3, 2019 at 6:17 >>>>>> PM Maximilian >>>>>> > Michels <m...@apache.org <mailto:m...@apache.org> >>>>>> > >> <mailto:m...@apache.org <mailto: >>>>>> m...@apache.org>>> >>>>>> > wrote: >>>>>> > >> > >>> > >>>>>> > >> > >>> > Thanks for driving this >>>>>> Kenn! I'm in >>>>>> > favor of a strict cut off, >>>>>> > >> but I'd like to >>>>>> > >> > >>> > propose a week for >>>>>> cherry-picking >>>>>> > relevant changes to the >>>>>> > >> release branch. It >>>>>> > >> > >>> > looks like many people are >>>>>> returning >>>>>> > from holidays or are still >>>>>> > >> off. >>>>>> > >> > >>> > >>>>>> > >> > >>> > Cheers, >>>>>> > >> > >>> > Max >>>>>> > >> > >>> > >>>>>> > >> > >>> > On 02.01.19 17:20, Kenneth >>>>>> Knowles wrote: >>>>>> > >> > >>> > > Done. I've created the >>>>>> Jira tag for >>>>>> > 2.11.0. >>>>>> > >> > >>> > > >>>>>> > >> > >>> > > Previously, there was a >>>>>> few days >>>>>> > warning to get things in >>>>>> > >> before the branch is >>>>>> > >> > >>> > > cut. You can just >>>>>> cherry-pick them. >>>>>> > This is a bit better for >>>>>> > >> release stability >>>>>> > >> > >>> > > by avoiding all the >>>>>> other changes on >>>>>> > master. The timing of >>>>>> > >> the cut is always >>>>>> > >> > >>> > > going to include older >>>>>> and newer >>>>>> > changes anyhow. >>>>>> > >> > >>> > > >>>>>> > >> > >>> > > Kenn >>>>>> > >> > >>> > > >>>>>> > >> > >>> > > On Wed, Jan 2, 2019 at >>>>>> 1:08 PM Ismaël >>>>>> > Mejía >>>>>> > >> <ieme...@gmail.com <mailto: >>>>>> ieme...@gmail.com> >>>>>> > <mailto:ieme...@gmail.com <mailto: >>>>>> ieme...@gmail.com>> >>>>>> > >> > >>> > > <mailto: >>>>>> ieme...@gmail.com >>>>>> > <mailto:ieme...@gmail.com> <mailto: >>>>>> ieme...@gmail.com >>>>>> > <mailto:ieme...@gmail.com>>>> wrote: >>>>>> > >> > >>> > > >>>>>> > >> > >>> > > Can you please >>>>>> create 2.11 tag in >>>>>> > JIRA so we can move the >>>>>> > >> JIRAs that >>>>>> > >> > >>> > > are not blocking. I >>>>>> have quite a >>>>>> > bunch of pending code >>>>>> > >> reviews that >>>>>> > >> > >>> > > hoped to get into >>>>>> this one but >>>>>> > well now probably they >>>>>> > >> shall wait. (My >>>>>> > >> > >>> > > excuses for the >>>>>> people who may be >>>>>> > impacted, I had not >>>>>> > >> checked that the >>>>>> > >> > >>> > > date was in the >>>>>> first week). >>>>>> > >> > >>> > > >>>>>> > >> > >>> > > On Wed, Jan 2, 2019 >>>>>> at 4:45 PM >>>>>> > Jean-Baptiste Onofré >>>>>> > >> <j...@nanthrax.net <mailto: >>>>>> j...@nanthrax.net> >>>>>> > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net >>>>>> >> >>>>>> > >> > >>> > > <mailto: >>>>>> j...@nanthrax.net >>>>>> > <mailto:j...@nanthrax.net> <mailto: >>>>>> j...@nanthrax.net >>>>>> > <mailto:j...@nanthrax.net>>>> wrote: >>>>>> > >> > >>> > > > >>>>>> > >> > >>> > > > It sounds good to >>>>>> me. >>>>>> > >> > >>> > > > >>>>>> > >> > >>> > > > Regards >>>>>> > >> > >>> > > > JB >>>>>> > >> > >>> > > > >>>>>> > >> > >>> > > > On 02/01/2019 >>>>>> 16:16, Kenneth >>>>>> > Knowles wrote: >>>>>> > >> > >>> > > > > Hi All, >>>>>> > >> > >>> > > > > >>>>>> > >> > >>> > > > > According to >>>>>> the release >>>>>> > calendar [1] branch cut >>>>>> > >> date for >>>>>> > >> > >>> > > > > Beam 2.10.0 >>>>>> release is >>>>>> > today, 2019 January 2. I'd >>>>>> > >> like to volunteer to >>>>>> > >> > >>> > > > > manage this >>>>>> release. Does >>>>>> > anyone have any reason we >>>>>> > >> should not release >>>>>> > >> > >>> > > > > on schedule? >>>>>> > >> > >>> > > > > >>>>>> > >> > >>> > > > > Otherwise, if >>>>>> you know of >>>>>> > release-blocking bugs, >>>>>> > >> please mark their "Fix >>>>>> > >> > >>> > > > > Version" as >>>>>> 2.10.0 and they >>>>>> > will show up in the >>>>>> > >> burndown [2]. If you own >>>>>> > >> > >>> > > > > a bug currently >>>>>> in the >>>>>> > burndown, please double-check >>>>>> > >> if it is truly >>>>>> > >> > >>> > > > > >>>>>> release-blocking. >>>>>> > >> > >>> > > > > >>>>>> > >> > >>> > > > > I've gone ahead >>>>>> and cut a >>>>>> > release-2.10.0 branch from >>>>>> > >> the current master, >>>>>> > >> > >>> > > > > since it is >>>>>> green. If it >>>>>> > turns out to be an >>>>>> > >> inauspicious starting point, >>>>>> > >> > >>> > > > > we can always >>>>>> reset it. >>>>>> > >> > >>> > > > > >>>>>> > >> > >>> > > > > Kenn >>>>>> > >> > >>> > > > > >>>>>> > >> > >>> > > > > [1] >>>>>> > >> > >>> > > >>>>>> > >> >>>>>> > >> >>>>>> > >>>>>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com&ctz=America%2FLos_Angeles >>>>>> > >>>>>> > >> >>>>>> > >> > >>> > > > > [2] >>>>>> > >> >>>>>> > >>>>>> https://issues.apache.org/jira/projects/BEAM/versions/12344540 >>>>>> > >> > >>> > > > >>>>>> > >> > >>> > > > -- >>>>>> > >> > >>> > > > Jean-Baptiste >>>>>> Onofré >>>>>> > >> > >>> > > > >>>>>> jbono...@apache.org >>>>>> > <mailto:jbono...@apache.org> <mailto: >>>>>> jbono...@apache.org >>>>>> > <mailto:jbono...@apache.org>> >>>>>> > >> <mailto:jbono...@apache.org >>>>>> > <mailto:jbono...@apache.org> <mailto: >>>>>> jbono...@apache.org >>>>>> > <mailto:jbono...@apache.org>>> >>>>>> > >> > >>> > > > >>>>>> http://blog.nanthrax.net >>>>>> > >> > >>> > > > Talend - >>>>>> http://www.talend.com >>>>>> > >> > >>> > > >>>>>> > >> >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > Got feedback? tinyurl.com/swegner-feedback >>>>>> > <https://tinyurl.com/swegner-feedback> >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > Got feedback? tinyurl.com/swegner-feedback >>>>>> > <https://tinyurl.com/swegner-feedback> >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > Got feedback? tinyurl.com/swegner-feedback < >>>>>> https://tinyurl.com/swegner-feedback> >>>>>> >>>>>
