up, know it missed the 2.4 but can https://github.com/apache/beam/pull/4790 have some love? it really makes beam pretty unusable with direct runner, I start to have "// workaround for BEAM-3409" everywhere in my codebase which is quite bothering after 3 months :(
Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> 2018-03-06 14:44 GMT+01:00 Romain Manni-Bucau <rmannibu...@gmail.com>: > Hi guys, > > tried to reapply the waitUntilFinish fix - without breaking the > compilation this time ;) - in https://github.com/apache/beam/pull/4790, > anyone able to help to review and move that forward? > > > Romain Manni-Bucau > @rmannibucau <https://twitter.com/rmannibucau> | Blog > <https://rmannibucau.metawerx.net/> | Old Blog > <http://rmannibucau.wordpress.com> | Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> | Book > <https://www.packtpub.com/application-development/java-ee-8-high-performance> > > 2018-03-02 9:28 GMT+01:00 Robert Bradshaw <rober...@google.com>: > >> On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >>> Hi Ismaël, >>> >>> that's a good idea to show history. >>> >>> For me, the vote duration doesn't matter as we are in the release >>> process already. >>> >> >> A more relevant duration to track would probably be cut to final release. >> This both measures our investment in the release process, as well as how >> behind head is when we finally get the release out. >> >> >>> The gap between two releases is more significant. >> >> >> +1, this is what users see. (To clarify terminology, the "time between >> release" is the time between actually releasing x.y and x.y+1 that is most >> visible to end users, regardless of intermediate process like cutting and >> voting that we have.) Of course this gets thrown off if our >> time-to-prepare-a-release becomes a significant fraction of the desired >> time-between-releases. >> >> And clearly with an average of >>> 80 days (~ 3 months) it's two long. The idea is to reduce this clearly. I >>> propose two months previously (including the vote period), so meaning 6 >>> weeks >>> between releases. >>> >> >> It seems there have been proposals for monthly, every 6 weeks >> (sesquimonthly?), and bimonthly. >> >> >>> Regarding the time we take for the PR review and merge, I think it's a >>> fair time >>> to give us time to include new improvements and features, but also to >>> fix bugs. >>> >> >> Concrete deadlines can provide good motivation to get around to doing >> reviews, cleaning up PRs, fixing bugs, etc. that you've been meaning to do >> but for whatever reason keep putting off. So I think it's still good >> practice to have some lead time that a release is coming for a chance for >> folks to get stuff in, while still being clear that we're not holding >> things back for new features and if you don't make the cut another one is >> close behind. >> >> Regards >>> JB >>> >>> On 03/01/2018 06:17 PM, Ismaël Mejía wrote: >>> > The average time just in the vote process for Beam since we are out of >>> the >>> > incubator is 17.5 days with an average of 75 days between versions. >>> > >>> > Version Vote Period No. days >>> > 2.3.0 30/01-16/02 17 days (83 days since last) >>> > 2.2.0 27/10-25/11 29 days (101 days since last) >>> > 2.1.0 11/07-16/08 36 days (92 days since last) >>> > 2.0.0 08/05-16/05 8 days (62 days since last) >>> > 0.6.0 10/03-15/03 5 days (37 days since last) >>> > 0.5.0 27/01-06/02 10 days >>> > >>> > I think we should have these numbers into account to refine the >>> distance between >>> > releases. If we want to follow strict time-based releases, what we can >>> probably >>> > refine is how we cut the release so we try to reduce release overlaps >>> and avoid >>> > rushing unnecessarily. >>> > >>> > Maybe we should follow the proposed 6 weeks for the next release like >>> this: >>> > >>> > - 4 weeks let’s say just after succesful vote and then cut the release >>> branch. >>> > - 1 week to burn the blocker list (good to have ones that don’t make >>> will be >>> > moved to the next release). >>> > - 1 week for the vote + RCs (in case the vote takes longer at least >>> the overlap >>> > between vote + next dev cycle will be smaller). >>> > >>> > If we do this for the next cycle we will have a 6 week ‘dev’ period >>> and then we >>> > will have optimistically an average of 2 weeks of ‘releasing’ and 6 >>> weeks ‘dev’ >>> > cycles. >>> > >>> > On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré <j...@nanthrax.net> >>> wrote: >>> >> About BEAM-3409, I did a review yesterday and it looks good to me. We >>> are >>> >> waiting for Thomas' feedback. >>> >> >>> >> Regards >>> >> JB >>> >> Le 1 mars 2018, à 09:38, Robert Bradshaw <rober...@google.com> a >>> écrit: >>> >>> >>> >>> Looking at the burn-down list, we have 5 remaining issues. None of >>> these >>> >>> are blockers, but all look like they're really close (reviewed, >>> review >>> >>> comments were addressed, waiting for a final LGTM). Specifically: >>> >>> >>> >>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you >>> >>> verify they have all been addressed? >>> >>> BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor >>> >>> comments, looks like they were addressed, could you confirm? >>> >>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor >>> >>> comments, looks like they were addressed, could you confirm? >>> >>> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved. >>> >>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending >>> (currently >>> >>> running) tests passing. >>> >>> >>> >>> Let's see how many of these we can get in by, say, noon PST tomorrow. >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < >>> rober...@google.com> >>> >>> wrote: >>> >>>> >>> >>>> I tend to fall into the "release early, release often" camp in >>> general, >>> >>>> but for this one I'm particularly anxious to get the faster Python >>> direct >>> >>>> runner out in the hands of TFT/TFX users (and in particular have an >>> eye on >>> >>>> https://www.tensorflow.org/dev-summit/ which I think can be a >>> healthy source >>> >>>> of Beam users). >>> >>>> >>> >>>> >>> >>>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré < >>> j...@nanthrax.net> >>> >>>> wrote: >>> >>>>> >>> >>>>> Hi Gus, >>> >>>>> >>> >>>>> Thanks for the update, it makes sense. >>> >>>>> >>> >>>>> Regards >>> >>>>> JB >>> >>>>> >>> >>>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote: >>> >>>>>> Hi Jean-Baptiste, >>> >>>>>> >>> >>>>>> I can speak from the perspective of tf.transform >>> >>>>>> < https://github.com/tensorflow/transform> (TFT) in particular >>> and TFX >>> >>>>>> < https://research.google.com/pubs/pub46484.html> libs in >>> general, in >>> >>>>>> case it is >>> >>>>>> useful. >>> >>>>>> >>> >>>>>> TFX distributed computation has 2 "large" dependencies, namely >>> >>>>>> TensorFlow and >>> >>>>>> Apache Beam, each on their own release schedule. >>> >>>>>> As such, releasing of new TFX functionality often (but not always) >>> >>>>>> depends on >>> >>>>>> (and is blocked by) releases of *both* TensorFlow *and* Apache >>> Beam. >>> >>>>>> >>> >>>>>> Synchronizing releases across such large projects and >>> organizations is >>> >>>>>> likely >>> >>>>>> hard, so from our perspective having *frequent* releases of >>> Tensorflow >>> >>>>>> or Apache >>> >>>>>> Beam (and better yet both) decreases the time for which we are >>> blocked >>> >>>>>> on >>> >>>>>> releasing our features. >>> >>>>>> >>> >>>>>> In light of this, I would vote for more frequent releases in >>> general, >>> >>>>>> and for a >>> >>>>>> Beam 2.4 release soon in particular (as TFT 0.6 depends on it). >>> >>>>>> >>> >>>>>> Thanks, >>> >>>>>> Gus >>> >>>>>> >>> >>>>>> On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré < >>> >>>>>> j...@nanthrax.net >>> >>>>>> <mailto: j...@nanthrax.net>> wrote: >>> >>>>>> >>> >>>>>> By the way, if third party projects based on Beam (Google >>> >>>>>> Dataflow, Talend >>> >>>>>> DataStream, and others) need a release (including some >>> features), >>> >>>>>> it's better to >>> >>>>>> clearly state this on the mailing list. >>> >>>>>> >>> >>>>>> At Apache Karaf, I have lot of projects based on it >>> (OpenDaylight, >>> >>>>>> OpenHAB, >>> >>>>>> Websphere, ...). They just ask for the release schedule and >>> they >>> >>>>>> align with >>> >>>>>> these release. As a best effort, I'm always trying to move >>> fast >>> >>>>>> when a release >>> >>>>>> is asked. >>> >>>>>> >>> >>>>>> So, if 2.4.0 is required by third party, no problem to "ask >>> for a >>> >>>>>> release". >>> >>>>>> >>> >>>>>> Regards >>> >>>>>> JB >>> >>>>>> >>> >>>>>> On 02/28/2018 04:17 AM, Reuven Lax wrote: >>> >>>>>> > It's been six weeks since you proposed beam 2.3.0. so >>> assuming >>> >>>>>> the same time >>> >>>>>> > scale for this release, that's 1.5 months between releases. >>> >>>>>> Slightly faster than >>> >>>>>> > 2 months, but not by much. >>> >>>>>> > >>> >>>>>> > I do seem to remember that the original goal for beam was >>> >>>>>> monthly releases though. >>> >>>>>> > >>> >>>>>> > Reuven >>> >>>>>> > >>> >>>>>> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré < >>> >>>>>> j...@nanthrax.net <mailto: j...@nanthrax.net> >>> >>>>>> > <mailto: j...@nanthrax.net <mailto: j...@nanthrax.net>>> wrote: >>> >>>>>> > >>> >>>>>> > Hi Reuven, >>> >>>>>> > >>> >>>>>> > In a previous thread (about Beam project execution), I >>> >>>>>> proposed a release every >>> >>>>>> > two months (as a best effort), I will find the e-mail. >>> >>>>>> > >>> >>>>>> > Beam 2.3.0 has been released "officially" on February >>> 16th, >>> >>>>>> so two week ago >>> >>>>>> > roughly. I would have expected 2.4.0 not before end of >>> >>>>>> March. >>> >>>>>> > >>> >>>>>> > If we have issue we want to fix fast, then 2.3.1 is >>> good. If >>> >>>>>> it's a new release >>> >>>>>> > in the pace, it's pretty fast and might "confuse" our >>> users. >>> >>>>>> > >>> >>>>>> > That's why I'm curious ;) >>> >>>>>> > >>> >>>>>> > Regards >>> >>>>>> > JB >>> >>>>>> > >>> >>>>>> > On 02/28/2018 03:50 AM, Reuven Lax wrote: >>> >>>>>> > > Wasn't the original statement monthly releases? We've >>> >>>>>> never realistically >>> >>>>>> > > managed that, but Robert's proposed cut will be on a >>> >>>>>> 6-week pace. >>> >>>>>> > > >>> >>>>>> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré < >>> >>>>>> j...@nanthrax.net <mailto: j...@nanthrax.net> >>> >>>>>> > <mailto: j...@nanthrax.net <mailto: j...@nanthrax.net>> >>> >>>>>> > > <mailto: j...@nanthrax.net <mailto: j...@nanthrax.net> >>> >>>>>> <mailto: j...@nanthrax.net >>> >>>>>> <mailto: j...@nanthrax.net>>>> wrote: >>> >>>>>> > > >>> >>>>>> > > Hi Robert, >>> >>>>>> > > >>> >>>>>> > > I'm just curious: it's pretty fast compared to the >>> >>>>>> original plan of a >>> >>>>>> > release >>> >>>>>> > > every two months. What's the reason to cut 2.4.0 >>> now >>> >>>>>> instead of end of >>> >>>>>> > March ? >>> >>>>>> > > >>> >>>>>> > > I will do the Jira triage and update today. >>> >>>>>> > > >>> >>>>>> > > Regards >>> >>>>>> > > JB >>> >>>>>> > > >>> >>>>>> > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote: >>> >>>>>> > > > I'm planning on cutting the 2.4.0 release branch >>> >>>>>> soon (tomorrow?). I >>> >>>>>> > see 13 >>> >>>>>> > > > open issues on JIRA [1], none of which are >>> labeled >>> >>>>>> as blockers. If there >>> >>>>>> > > > are any that cannot be bumped to the next >>> release, >>> >>>>>> let me know soon. >>> >>>>>> > > > >>> >>>>>> > > > - Robert >>> >>>>>> > > > >>> >>>>>> > > > >>> >>>>>> > > > [1] >>> >>>>>> > > > >>> >>>>>> > > >>> >>>>>> > >>> >>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project% >>> 20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In% >>> 20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0 >>> >>>>>> < >>> >>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project% >>> 20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In% >>> 20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0> >>> >>>>>> > > > >>> >>>>>> > > >>> >>>>>> > > -- >>> >>>>>> > > Jean-Baptiste Onofré >>> >>>>>> > > jbono...@apache.org <mailto: jbono...@apache.org >>> > >>> >>>>>> <mailto: jbono...@apache.org >>> >>>>>> <mailto: jbono...@apache.org>> >>> >>>>>> > <mailto: jbono...@apache.org <mailto: >>> jbono...@apache.org> >>> >>>>>> <mailto: jbono...@apache.org <mailto: jbono...@apache.org>>> >>> >>>>>> > > http://blog.nanthrax.net >>> >>>>>> > > Talend - http://www.talend.com >>> >>>>>> > > >>> >>>>>> > >>> >>>>>> > -- >>> >>>>>> > Jean-Baptiste Onofré >>> >>>>>> > jbono...@apache.org <mailto: jbono...@apache.org> >>> >>>>>> <mailto: jbono...@apache.org <mailto: jbono...@apache.org>> >>> >>>>>> > http://blog.nanthrax.net >>> >>>>>> > Talend - http://www.talend.com >>> >>>>>> > >>> >>>>>> >>> >>>>>> -- >>> >>>>>> Jean-Baptiste Onofré >>> >>>>>> jbono...@apache.org <mailto: jbono...@apache.org> >>> >>>>>> http://blog.nanthrax.net >>> >>>>>> Talend - http://www.talend.com >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> -- >>> >>>>>> Gus Katsiapis | Software Engineer | katsia...@google.com >>> >>>>>> <mailto: katsia...@google.com> | 650-918-7487 >>> >>>>> >>> >>>>> -- >>> >>>>> Jean-Baptiste Onofré >>> >>>>> jbono...@apache.org >>> >>>>> http://blog.nanthrax.net >>> >>>>> Talend - http://www.talend.com >>> >>> -- >>> Jean-Baptiste Onofré >>> jbono...@apache.org >>> http://blog.nanthrax.net >>> Talend - http://www.talend.com >>> >> >
