Sounds good. I think we can add this under Technical Docs here <https://beam.apache.org/contribute/>.
Thanks, Cham On Tue, Jun 12, 2018 at 5:35 AM Kenneth Knowles <k...@google.com> wrote: > Makes sense. Then finding a good home on the web site is the way to go. > > Kenn > > On Mon, Jun 11, 2018 at 10:35 PM Bashir Sadjad <bas...@google.com> wrote: > >> FWIW, I also think that this has relevance for users. I am a user of Beam >> not a contributor and only monitor this list at a high level. But I think >> the dependency issue is something that many users have to deal with. It has >> bitten us at least twice over the last few months due to the fact that we >> depend on other libraries too and sometimes we get version conflicts (which >> is one of the issues highlighted in the doc >> <https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit> >> Cham shared). I usually go through file histories on GitHub to try to >> figure out why a certain version requirement is there. It would be nice if >> the reasons are maintained at a higher level easier to consume by users. >> >> Cheers >> >> -B >> >> On Tue, Jun 12, 2018 at 12:19 AM Ahmet Altay <al...@google.com> wrote: >> >>> I think this is relevant for users. It makes sense for users to know >>> about how Beam work with its dependencies and understand how conflicts will >>> be addressed and when dependencies will be upgraded. >>> >>> On Mon, Jun 11, 2018 at 9:09 PM, Kenneth Knowles <k...@google.com> wrote: >>> >>>> Do you think this has relevance for users? >>>> >>>> If not, it might be a good use of the new Confluence space. I'm not too >>>> familiar with the way permission work, but perhaps we can have a more >>>> locked down area that is for policy decisions like this. >>>> >>>> Kenn >>>> >>>> On Mon, Jun 11, 2018 at 3:58 PM Chamikara Jayalath < >>>> chamik...@google.com> wrote: >>>> >>>>> Hi All, >>>>> >>>>> Based on the vote (3 PMC +1s and no -1s) and based on the discussions >>>>> in the doc (seems to be mostly positive), I think we can go ahead and >>>>> implement some of the policies discussed so far. >>>>> >>>>> I have given some of the potential action items below. >>>>> >>>>> * Automatically generate human readable reports on status of Beam >>>>> dependencies weekly and share in dev list. >>>>> * Create JIRAs for significantly outdated dependencies based on above >>>>> reports. >>>>> * Copy some of the component level dependency version declarations to >>>>> top level. >>>>> * Try to identify owners for dependencies and specify owners in >>>>> comments close to dependency declarations. >>>>> * Vendor any dependencies that can cause issues if leaked to other >>>>> components. >>>>> * Add policies discussed so far to the Web site along with reasoning >>>>> (from doc). >>>>> >>>>> Of course, I'm happy to refine or add to these polices as needed. >>>>> >>>>> Thanks, >>>>> Cham >>>>> >>>>> >>>>> On Thu, Jun 7, 2018 at 9:40 AM Lukasz Cwik <lc...@google.com> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> On Thu, Jun 7, 2018 at 5:18 AM Kenneth Knowles <k...@google.com> >>>>>> wrote: >>>>>> >>>>>>> +1 to these. Thanks for clarifying! >>>>>>> >>>>>>> Kenn >>>>>>> >>>>>>> On Wed, Jun 6, 2018 at 10:40 PM Chamikara Jayalath < >>>>>>> chamik...@google.com> wrote: >>>>>>> >>>>>>>> Hi Kenn, >>>>>>>> >>>>>>>> On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <k...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> +0.5 >>>>>>>>> >>>>>>>>> I like the spirit of these policies. I think they need a little >>>>>>>>> wording work. Comments inline. >>>>>>>>> >>>>>>>>> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath < >>>>>>>>>> chamik...@google.com> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> (1) Human readable reports on status of Beam dependencies are >>>>>>>>>>> generated weekly and shared with the Beam community through the dev >>>>>>>>>>> list. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> Who is responsible for generating these? The mechanism or >>>>>>>>> responsibility should be made clear. >>>>>>>>> >>>>>>>>> I clicked through a doc -> thread -> doc to find even some >>>>>>>>> details. It looks like manual run of a gradle command was adopted. So >>>>>>>>> the >>>>>>>>> responsibility needs an owner, even if it is "unspecified volunteer >>>>>>>>> on dev@ >>>>>>>>> and feel free to complain or do it yourself if you don't see it" >>>>>>>>> >>>>>>>> >>>>>>>> This is described in following doc (referenced by my doc). >>>>>>>> >>>>>>>> https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit# >>>>>>>> >>>>>>>> Proposal is to run an automated Jenkins job that is run weekly, so >>>>>>>> no need for someone to manually generate these reports. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> (2) Beam components should define dependencies and their versions >>>>>>>>>>> at the top level. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> I think the big "should" works better with some guidance about >>>>>>>>> when something might be an exception, or at least explicit mention >>>>>>>>> that >>>>>>>>> there can be rare exceptions. Unless you think that is never the >>>>>>>>> case. If >>>>>>>>> there are no exceptions, then say "must" and if we hit a roadblock we >>>>>>>>> can >>>>>>>>> revisit the policy. >>>>>>>>> >>>>>>>> >>>>>>>> The idea was to allow exceptions. Added more details to the doc. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> (3) A significantly outdated dependency (identified manually or >>>>>>>>>>> through tooling) should result in a JIRA that is a blocker for the >>>>>>>>>>> next >>>>>>>>>>> release. Release manager may choose to push the blocker to the >>>>>>>>>>> subsequent >>>>>>>>>>> release or downgrade from a blocker. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> How is "significantly outdated" defined? By dev@ discussion? >>>>>>>>> Seems like the right way. Anyhow that's what will happen in practice >>>>>>>>> as >>>>>>>>> people debate the blocker bug. >>>>>>>>> >>>>>>>> >>>>>>>> This will be either through the automated Jenkins job (see the doc >>>>>>>> above, where the proposal is to flag new major versions and new minor >>>>>>>> versions that are more than six months old) or manually (for any >>>>>>>> critical >>>>>>>> updates that will not be captured by the Jenkins job) (more details in >>>>>>>> the >>>>>>>> doc). Manually identified critical dependency updates may involve a >>>>>>>> discussion in the dev list. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> (4) Dependency declarations may identify owners that are >>>>>>>>>>> responsible for upgrading the respective dependencies. >>>>>>>>>>> >>>>>>>>>>> (5) Dependencies of Java SDK components that may cause issues to >>>>>>>>>>> other components if leaked should be shaded. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> We previously agreed upon our intent to migrate to "pre-shaded" >>>>>>>>> aka "vendored" packages: >>>>>>>>> https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E >>>>>>>>> >>>>>>>>> With Maven, this involved a lot of boilerplate so I never did it. >>>>>>>>> With Gradle, we can easily build a re-usable rule to create such a >>>>>>>>> package >>>>>>>>> in a couple of lines. I just opened the first WIP PR here: >>>>>>>>> https://github.com/apache/beam/pull/5570 it is blocked by >>>>>>>>> deleting the poms anyhow so by then we should have a configuration >>>>>>>>> that >>>>>>>>> works to vendor our currently shaded artifacts. >>>>>>>>> >>>>>>>>> So I think this should be rephrased to "should be vendored" so we >>>>>>>>> don't have to revise the policy. >>>>>>>>> >>>>>>>> >>>>>>>> Thanks for the pointer. I agree that vendoring is a good approach. >>>>>>>> >>>>>>>> Here are the updated policies (and more details added to doc). I >>>>>>>> agree with Ahmet's point that votes should be converted to web sites >>>>>>>> where >>>>>>>> we can give more details and examples. >>>>>>>> >>>>>>>> (1.a) Human readable reports on status of Beam dependencies are >>>>>>>> generated weekly by an automated Jenkins job and shared with the Beam >>>>>>>> community through the dev list. >>>>>>>> >>>>>>>> (2.a) Beam components should define dependencies and their >>>>>>>> versions at the top level. There can be rare exceptions, but they >>>>>>>> should >>>>>>>> come with explanations. >>>>>>>> >>>>>>>> (3.a) A significantly outdated dependency (identified manually or >>>>>>>> through the automated Jenkins job) should result in a JIRA that is a >>>>>>>> blocker for the next release. Release manager may choose to push the >>>>>>>> blocker to the subsequent release or downgrade from a blocker. >>>>>>>> >>>>>>>> (4.a) Dependency declarations may identify owners that are >>>>>>>> responsible for upgrading the respective dependencies. >>>>>>>> >>>>>>>> (5.a) Dependencies of Java SDK components that may cause issues to >>>>>>>> other components if leaked should be vendored. >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Cham >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Kenn >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Please vote: >>>>>>>>>>> [ ] +1, Approve that we adapt these policies >>>>>>>>>>> [ ] -1, Do not approve (please provide specific comments) >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Cham >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E >>>>>>>>>>> [2] >>>>>>>>>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>
