This has been revised as https://github.com/apache/beam/pull/13075 and it's merged yesterday. Now many GCP-related dependencies are managed by the BOM. Thank you Kiley!
On Mon, Mar 30, 2020 at 10:59 AM Tomo Suzuki <suzt...@google.com> wrote: > Hi Chamikara, > > Ahmet is asking for your input in > https://github.com/apache/beam/pull/11156#issuecomment-602716275 . Would > you check this? > > On Mon, Mar 23, 2020 at 3:05 PM Tomo Suzuki <suzt...@google.com> wrote: > >> (Applied Ahmet's feedback in https://github.com/apache/beam/pull/11156) >> >> Hi Luke, >> >> > the pom files that are produced >> >> Build.gradle that uses the BOM will produce pom files that have >> corresponding dependencies without versions, and have a >> "dependencyManagement" section through which they import the GCP Libraries >> BOM. This fills the versions of the versionless dependencies when build >> systems evaluates the pom files. >> >> > how the GCP BOM impacts the release process >> >> I'm afraid I don't have knowledge on how dependencies affect the Beam >> release process. Would you be willing to break down this concern into some >> questions I can answer? >> >> >> >> >> On Fri, Mar 20, 2020 at 6:13 PM Luke Cwik <lc...@google.com> wrote: >> >>> My concern would be to validate how the GCP BOM impacts the release >>> process and the pom files that are produced otherwise the next person >>> running a release may run into trouble. >>> >>> On Fri, Mar 20, 2020 at 3:00 PM Pablo Estrada <pabl...@google.com> >>> wrote: >>> >>>> +1 >>>> >>>> On Fri, Mar 20, 2020 at 2:41 PM Ismaël Mejía <ieme...@gmail.com> wrote: >>>> >>>>> Well why we just don't merge it? I am unfamiliar with GCP deps to be >>>>> confident to LGTM it. but given that 22 checks pass and given that >>>>> Tomo addressed most comments and he has already a consistent track of >>>>> good work on dependency improvements I think it is worth to merge it >>>>> as it is. We still have some time to fix stuff if we find any >>>>> regression. WDYT? >>>>> >>>>> On Thu, Mar 19, 2020 at 9:36 PM Luke Cwik <lc...@google.com> wrote: >>>>> > >>>>> > As much as I would like to spend time on these reviews, I believe >>>>> I'll be delayed from reviewing them thoroughly till I finish other work >>>>> that I'm targeting for the 2.21 release related to portability. It would >>>>> be >>>>> greatly appreciated if there are others that could review this in the >>>>> meantime. >>>>> > >>>>> > On Thu, Mar 19, 2020 at 7:09 AM Tomo Suzuki <suzt...@google.com> >>>>> wrote: >>>>> >> >>>>> >> PR is ready (22 successful check) >>>>> >> https://github.com/apache/beam/pull/11156 >>>>> >> (Luke assigned himself as a reviewer) >>>>> >> >>>>> >> Regards, >>>>> >> Tomo >>>>> >> >>>>> >> On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki <suzt...@google.com> >>>>> wrote: >>>>> >>> >>>>> >>> Thank you for favorable responses. I'll start implementation. >>>>> >>> >>>>> >>> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki <suzt...@google.com> >>>>> wrote: >>>>> >>>> >>>>> >>>> > Do Spark or Flink have BOMs? >>>>> >>>> >>>>> >>>> Not that I know of. I couldn't find "bom" in their artifacts [1, >>>>> 2]. >>>>> >>>> >>>>> >>>> [1]: https://search.maven.org/search?q=g:org.apache.flink >>>>> >>>> [2]: https://search.maven.org/search?q=g:org.apache.spark >>>>> >>>> >>>>> >>>> >>>>> >>>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles <k...@apache.org> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> +1 and you have phrased the benefits and limitations well. We >>>>> have plenty of not-Google-related dependencies that use Guava and protobuf >>>>> (I know of Calcite, Cassandra, Kinesis, and Spark) so there's still work >>>>> in >>>>> managing deps, but the BOM should make it a lot easier to upgrade all >>>>> these >>>>> tightly coupled libraries that Google ships from their head. >>>>> >>>>> >>>>> >>>>> Do Spark or Flink have BOMs? I wonder if there's an opportunity >>>>> to catch incompatible deps at a larger scale by comparing and merging a >>>>> half dozen BOMs (although in the limit it approximately expands to one per >>>>> runner and one per IO and projects mature and become independent) >>>>> >>>>> >>>>> >>>>> Kenn >>>>> >>>>> >>>>> >>>>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik <lc...@google.com> >>>>> wrote: >>>>> >>>>>> >>>>> >>>>>> How would the Apache Beam BOM and GCP BOM work together? >>>>> >>>>>> >>>>> >>>>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas < >>>>> filiperega...@gmail.com> wrote: >>>>> >>>>>>> >>>>> >>>>>>> Big +1, this is a step in the right direction and checking >>>>> with other Beam's direct and transitive deps is crucial since the referred >>>>> bom only convers a small part of it. Apache Commons, Jackson, >>>>> `com.google.{api, apis, cloud}`, slf4j comes to mind. >>>>> >>>>>>> >>>>> >>>>>>> Filipe Regadas >>>>> >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ieme...@gmail.com> >>>>> wrote: >>>>> >>>>>>>> >>>>> >>>>>>>> +1 Sounds like a good improvement for users and maintainers ! >>>>> >>>>>>>> >>>>> >>>>>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel < >>>>> a...@vanboxel.be> wrote: >>>>> >>>>>>>> > >>>>> >>>>>>>> > +1, I can remember the countless hours that we fought with >>>>> Google dependencies. >>>>> >>>>>>>> > >>>>> >>>>>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath < >>>>> chamik...@google.com> wrote: >>>>> >>>>>>>> >> >>>>> >>>>>>>> >> +1 for this. >>>>> >>>>>>>> >> >>>>> >>>>>>>> >> This will make life easy for many of our users and will >>>>> help us keep GCP related dependencies compatible (which has not been easy >>>>> in the past). >>>>> >>>>>>>> >> >>>>> >>>>>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki < >>>>> suzt...@google.com> wrote: >>>>> >>>>>>>> >>> >>>>> >>>>>>>> >>> Hi Beam developers, >>>>> >>>>>>>> >>> >>>>> >>>>>>>> >>> Shall we use GCP Libraries BOM [1] to specify the >>>>> Google-related library versions in Beam? >>>>> >>>>>>>> >>> >>>>> >>>>>>>> >>> I've been working on Beam's dependency upgrades in the >>>>> past few months. It's time to consider a long-term solution to keep the >>>>> libraries up-to-date with small maintenance effort. To achieve that, I >>>>> propose Beam to use GCP Libraries BOM to set the Google-related library >>>>> versions, rather than the current way of making changes in each of ~30 >>>>> Google libraries with individual PRs [2]. >>>>> >>>>>>>> >>> >>>>> >>>>>>>> >>> After the proposal is implemented, Beam project upgrades >>>>> the BOM version to upgrade these Google-related libraries. This still >>>>> needs >>>>> to ensure the libraries in GCP Library BOM are compatible with Beam's >>>>> other >>>>> dependencies. (Linkage Checker will help with this job.) I believe >>>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we >>>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort >>>>> in >>>>> Beam's developers. >>>>> >>>>>>>> >>> >>>>> >>>>>>>> >>> Created an issue to track this: BEAM-9444 [3]. I >>>>> appreciate if you can share questions or feedback (thumbs-up / concerns). >>>>> >>>>>>>> >>> >>>>> >>>>>>>> >>> [1]: >>>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM >>>>> >>>>>>>> >>> [2]: >>>>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo >>>>> >>>>>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444 >>>>> >>>>>>>> >>> >>>>> >>>>>>>> >>> -- >>>>> >>>>>>>> >>> Regards, >>>>> >>>>>>>> >>> Tomo >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> -- >>>>> >>>> Regards, >>>>> >>>> Tomo >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> -- >>>>> >>> Regards, >>>>> >>> Tomo >>>>> >> >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> Regards, >>>>> >> Tomo >>>>> >>>> >> >> -- >> Regards, >> Tomo >> > > > -- > Regards, > Tomo > -- Regards, Tomo
