Is it possible to trigger a manual task to update the snapshot artifacts here in the interim until a decision has been made? It is blocking dev toward a public release.
On Thu, Apr 26, 2018 at 2:51 AM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > > 2018-04-26 7:54 GMT+02:00 Reuven Lax <re...@google.com>: > >> This seems more than just Gradle, as they are using internal details in >> the parent pom (all the Google Cloud dependency version). Saying that >> nothing should break means that we can never rename or refactor anything in >> our poms because some user might be depending on it; that would be true on >> Maven just as much as on Gradle. >> >> Maybe this particular use is important enough that it needs to be >> maintained, I'm not sure. However we also need to decide what our stable >> pom interface is for users. >> > > Browsing pom hierarchy to grab artifacts (including properties or not) > and plugin configurations (it is common to extract compiler config and > shade configs) is widely spread cause maven is the standard exchange format > so I hope beam will respect that at least. > The other usage is to check the dependencies between modules and determine > the impact of a change ("what is the common part" ~ parent+bom, "what does > it impact if we change that dep"). It can be key for beam to keep that to > ensure upstream projects can determine some of their impacts statically > through mavenrepo tools. > > >> >> Reuven >> >> On Wed, Apr 25, 2018 at 9:34 PM Romain Manni-Bucau <rmannibu...@gmail.com> >> wrote: >> >>> Hmm, gradle move not supposed to break any user and there are other >>> parents usages, as mentionned previously, so not sure beam can really >>> bypass it. Maven discussed consumer vs build pom since years for that >>> reason ;). >>> >>> Le 26 avr. 2018 01:59, "Eric Beach" <ebe...@google.com> a écrit : >>> >>>> Thanks Scott and Ben. >>>> >>>> This makes sense, and of course the same error happens when I tried >>>> beam-parent instead of beam-examples-parent. >>>> >>>> There is a big benefit for supporting explicit dependence on parent >>>> (either beam-examples-parent or beam-parent). In the project I'm involved >>>> in, I have Dataflow API (to query metrics for completed jobs), Google Cloud >>>> Storage, and Cloud Spanner (to execute queries outside of Beam). >>>> >>>> >>>> On Wed, Apr 25, 2018 at 7:55 PM Scott Wegner <sweg...@google.com> >>>> wrote: >>>> >>>>> Interesting. So two things: >>>>> >>>>> 1) As you correctly deduced, we switched to producing snapshots from >>>>> the Gradle build, which doesn't include parent pom's. As a result, the >>>>> parent pom snapshot you're referencing is stale. We should probably remove >>>>> it from the repository to prevent accidental usage until we get this >>>>> sorted >>>>> out. >>>>> >>>>> 2) Depending on the parent pom directly isn't an intentionally >>>>> supported scenario, but your use-case seems valid: successful execution on >>>>> Dataflow requires specific version sets for Google Cloud dependencies. And >>>>> those dependency versions are conveniently defined as properties in the >>>>> beam-parent >>>>> pom >>>>> <https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/beam-parent-2.5.0-20180408.070218-37.pom> >>>>> . >>>>> >>>>> Note that simply generating parent pom's from Gradle wouldn't fix >>>>> this; we need to explicitly generate these version properties in the pom >>>>> file. >>>>> >>>>> We could publish a special GCP parent pom just for this purpose. >>>>> Although forcing a parent relationship seems unnecessary; is there a >>>>> better >>>>> convention within Maven for importing a set property values? >>>>> >>>>> On Wed, Apr 25, 2018 at 4:37 PM Eric Beach <ebe...@google.com> wrote: >>>>> >>>>>> If I run $ mvn clean compile exec:java ... on my project, I get the >>>>>> following error stack trace: >>>>>> >>>>>> java.lang.reflect.InvocationTargetException >>>>>> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>>>> Method) >>>>>> >>>>>> at >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>> >>>>>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>> >>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>> >>>>>> at >>>>>> org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294) >>>>>> >>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>> >>>>>> Caused by: java.lang.NoClassDefFoundError: >>>>>> com/google/api/services/clouddebugger/v2/CloudDebugger >>>>>> >>>>>> at java.lang.Class.getDeclaredMethods0(Native Method) >>>>>> >>>>>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2703) >>>>>> >>>>>> at java.lang.Class.getDeclaredMethod(Class.java:2130) >>>>>> >>>>>> at >>>>>> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:206) >>>>>> >>>>>> at >>>>>> org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162) >>>>>> >>>>>> at >>>>>> org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55) >>>>>> >>>>>> at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150) >>>>>> >>>>>> at >>>>>> com.google.cloud.pontem.CloudSpannerDatabaseBackup.main(CloudSpannerDatabaseBackup.java:359) >>>>>> >>>>>> ... 6 more >>>>>> >>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>> com.google.api.services.clouddebugger.v2.CloudDebugger >>>>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>> >>>>>> >>>>>> When I dig deeper, the root problem seems to be that my project is >>>>>> pulling in v2-rev8-1.22.0 instead of v2-rev233-1.23.0 (changing >>>>>> commit here >>>>>> <https://github.com/apache/beam/commit/66fcf4bff19cda5831465ccedbbaa6fbaa648234#diff-600376dffeb79835ede4a0b285078036> >>>>>> ). >>>>>> >>>>>> In fact, all the version changes in this commit >>>>>> <https://github.com/apache/beam/commit/66fcf4bff19cda5831465ccedbbaa6fbaa648234#diff-600376dffeb79835ede4a0b285078036> >>>>>> are missing. >>>>>> >>>>>> My project's pom.xml contains the following: >>>>>> >>>>>> <parent> >>>>>> >>>>>> <artifactId>beam-examples-parent</artifactId> >>>>>> >>>>>> <groupId>org.apache.beam</groupId> >>>>>> >>>>>> <version>2.5.0-SNAPSHOT</version> >>>>>> >>>>>> </parent> >>>>>> >>>>>> <modelVersion>4.0.0</modelVersion> >>>>>> >>>>>> >>>>>> <groupId>com.google.cloud</groupId> >>>>>> >>>>>> <artifactId>xyz</artifactId> >>>>>> >>>>>> >>>>>> <repositories> >>>>>> >>>>>> <repository> >>>>>> >>>>>> <id>apache.snapshots</id> >>>>>> >>>>>> <name>Apache Development Snapshot Repository</name> >>>>>> >>>>>> <url> >>>>>> https://repository.apache.org/content/repositories/snapshots/</url> >>>>>> >>>>>> <releases> >>>>>> >>>>>> <enabled>false</enabled> >>>>>> >>>>>> </releases> >>>>>> >>>>>> <snapshots> >>>>>> >>>>>> <enabled>true</enabled> >>>>>> >>>>>> </snapshots> >>>>>> >>>>>> </repository> >>>>>> >>>>>> </repositories> >>>>>> >>>>>> My first round of attempts to debug included removing ~/.m2/ and a >>>>>> number of other techniques. >>>>>> >>>>>> When I look at >>>>>> https://repository.apache.org/content/repositories/snapshots, I notice >>>>>> that beam-parent with 2.5.0-SNAPSHOT >>>>>> <https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/> >>>>>> is >>>>>> very outdated and so all the version numbers inherited from the POM are >>>>>> wrong (see >>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/). >>>>>> If you look at >>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/ >>>>>> you will see that the beam-parent is significantly old. >>>>>> >>>>>> So, it seems imperative that the snapshot be rebuilt before the >>>>>> finalized 2.5.0 version is set or any projects depending on the Beam >>>>>> parent >>>>>> will break. (I depend upon the Beam parent because trying to keep the >>>>>> versions of all the different Google projects in sync is a total >>>>>> nightmare). >>>>>> >>>>>> >>>>>> On Wed, Apr 25, 2018 at 6:58 PM Chamikara Jayalath < >>>>>> chamik...@google.com> wrote: >>>>>> >>>>>>> Ccing Eric for providing more context. >>>>>>> >>>>>>> Thanks, >>>>>>> Cham >>>>>>> >>>>>>> On Wed, Apr 25, 2018 at 3:38 PM Scott Wegner <sweg...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Do you have any more context on how they were using the parent pom? >>>>>>>> In the Gradle build, instead of using a parent pom hierarchy we are >>>>>>>> embedding the complete set of dependencies and metadata in each >>>>>>>> generated >>>>>>>> pom. They should be functionally equivalent. >>>>>>>> >>>>>>>> On Wed, Apr 25, 2018 at 3:09 PM Chamikara Jayalath < >>>>>>>> chamik...@google.com> wrote: >>>>>>>> >>>>>>>>> At least one user was depending on 'beam-parent' and ran into >>>>>>>>> issues due to dependency update >>>>>>>>> https://github.com/apache/beam/pull/5046 not being reflected >>>>>>>>> there. I'm not sure if this is a usage pattern that we should >>>>>>>>> discourage. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Cham >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Apr 25, 2018 at 2:36 PM Alan Myrvold <amyrv...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> The gradle build is currently only generating pom files when >>>>>>>>>> there is a jar file. >>>>>>>>>> What are these parent poms used for? Do all the parent poms need >>>>>>>>>> to be generated or just the top level beam-parent? >>>>>>>>>> >>>>>>>>>> On Wed, Apr 25, 2018 at 2:07 PM Chamikara Jayalath < >>>>>>>>>> chamik...@google.com> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks. Should this be a 2.5.0 blocker ? >>>>>>>>>>> >>>>>>>>>>> On Wed, Apr 25, 2018 at 2:05 PM Alan Myrvold < >>>>>>>>>>> amyrv...@google.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> I think this corresponds with the move from maven to gradle >>>>>>>>>>>> snapshot releases. >>>>>>>>>>>> Issue logged as https://issues.apache.org/jira/browse/BEAM-4170 >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Apr 25, 2018 at 1:57 PM Chamikara Jayalath < >>>>>>>>>>>> chamik...@google.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Seems like we are not building some of the SNAPSHOT artifacts >>>>>>>>>>>>> since 4/9/2018. >>>>>>>>>>>>> For example: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/ >>>>>>>>>>>>> >>>>>>>>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-examples-parent/2.5.0-SNAPSHOT/ >>>>>>>>>>>>> >>>>>>>>>>>>> Users who depend on these artifacts are getting outdated >>>>>>>>>>>>> dependencies. >>>>>>>>>>>>> >>>>>>>>>>>>> Any idea why ? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Cham >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>> >>>>>>>> >>>>>>>> Got feedback? http://go/swegner-feedback >>>>>>>> >>>>>>> -- >>>>> >>>>> >>>>> Got feedback? http://go/swegner-feedback >>>>> >>>> >