Is it possible to trigger a manual task to update the snapshot artifacts
here in the interim until a decision has been made? It is blocking dev
toward a public release.
On Thu, Apr 26, 2018 at 2:51 AM Romain Manni-Bucau <rmannibu...@gmail.com>
wrote:
>
>
> 2018-04-26 7:54 GMT+02:00 Reuven Lax <re...@google.com>:
>
>> This seems more than just Gradle, as they are using internal details in
>> the parent pom (all the Google Cloud dependency version). Saying that
>> nothing should break means that we can never rename or refactor anything in
>> our poms because some user might be depending on it; that would be true on
>> Maven just as much as on Gradle.
>>
>> Maybe this particular use is important enough that it needs to be
>> maintained, I'm not sure. However we also need to decide what our stable
>> pom interface is for users.
>>
>
> Browsing pom hierarchy to grab artifacts (including properties or not)
> and plugin configurations (it is common to extract compiler config and
> shade configs) is widely spread cause maven is the standard exchange format
> so I hope beam will respect that at least.
> The other usage is to check the dependencies between modules and determine
> the impact of a change ("what is the common part" ~ parent+bom, "what does
> it impact if we change that dep"). It can be key for beam to keep that to
> ensure upstream projects can determine some of their impacts statically
> through mavenrepo tools.
>
>
>>
>> Reuven
>>
>> On Wed, Apr 25, 2018 at 9:34 PM Romain Manni-Bucau <rmannibu...@gmail.com>
>> wrote:
>>
>>> Hmm, gradle move not supposed to break any user and there are other
>>> parents usages, as mentionned previously, so not sure beam can really
>>> bypass it. Maven discussed consumer vs build pom since years for that
>>> reason ;).
>>>
>>> Le 26 avr. 2018 01:59, "Eric Beach" <ebe...@google.com> a écrit :
>>>
>>>> Thanks Scott and Ben.
>>>>
>>>> This makes sense, and of course the same error happens when I tried
>>>> beam-parent instead of beam-examples-parent.
>>>>
>>>> There is a big benefit for supporting explicit dependence on parent
>>>> (either beam-examples-parent or beam-parent). In the project I'm involved
>>>> in, I have Dataflow API (to query metrics for completed jobs), Google Cloud
>>>> Storage, and Cloud Spanner (to execute queries outside of Beam).
>>>>
>>>>
>>>> On Wed, Apr 25, 2018 at 7:55 PM Scott Wegner <sweg...@google.com>
>>>> wrote:
>>>>
>>>>> Interesting. So two things:
>>>>>
>>>>> 1) As you correctly deduced, we switched to producing snapshots from
>>>>> the Gradle build, which doesn't include parent pom's. As a result, the
>>>>> parent pom snapshot you're referencing is stale. We should probably remove
>>>>> it from the repository to prevent accidental usage until we get this
>>>>> sorted
>>>>> out.
>>>>>
>>>>> 2) Depending on the parent pom directly isn't an intentionally
>>>>> supported scenario, but your use-case seems valid: successful execution on
>>>>> Dataflow requires specific version sets for Google Cloud dependencies. And
>>>>> those dependency versions are conveniently defined as properties in the
>>>>> beam-parent
>>>>> pom
>>>>> <https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/beam-parent-2.5.0-20180408.070218-37.pom>
>>>>> .
>>>>>
>>>>> Note that simply generating parent pom's from Gradle wouldn't fix
>>>>> this; we need to explicitly generate these version properties in the pom
>>>>> file.
>>>>>
>>>>> We could publish a special GCP parent pom just for this purpose.
>>>>> Although forcing a parent relationship seems unnecessary; is there a
>>>>> better
>>>>> convention within Maven for importing a set property values?
>>>>>
>>>>> On Wed, Apr 25, 2018 at 4:37 PM Eric Beach <ebe...@google.com> wrote:
>>>>>
>>>>>> If I run $ mvn clean compile exec:java ... on my project, I get the
>>>>>> following error stack trace:
>>>>>>
>>>>>> java.lang.reflect.InvocationTargetException
>>>>>>
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>> Method)
>>>>>>
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>
>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>>
>>>>>> at
>>>>>> org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>>>>>>
>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>
>>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>>> com/google/api/services/clouddebugger/v2/CloudDebugger
>>>>>>
>>>>>> at java.lang.Class.getDeclaredMethods0(Native Method)
>>>>>>
>>>>>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2703)
>>>>>>
>>>>>> at java.lang.Class.getDeclaredMethod(Class.java:2130)
>>>>>>
>>>>>> at
>>>>>> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:206)
>>>>>>
>>>>>> at
>>>>>> org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
>>>>>>
>>>>>> at
>>>>>> org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
>>>>>>
>>>>>> at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
>>>>>>
>>>>>> at
>>>>>> com.google.cloud.pontem.CloudSpannerDatabaseBackup.main(CloudSpannerDatabaseBackup.java:359)
>>>>>>
>>>>>> ... 6 more
>>>>>>
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> com.google.api.services.clouddebugger.v2.CloudDebugger
>>>>>>
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>>>
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>
>>>>>>
>>>>>> When I dig deeper, the root problem seems to be that my project is
>>>>>> pulling in v2-rev8-1.22.0 instead of v2-rev233-1.23.0 (changing
>>>>>> commit here
>>>>>> <https://github.com/apache/beam/commit/66fcf4bff19cda5831465ccedbbaa6fbaa648234#diff-600376dffeb79835ede4a0b285078036>
>>>>>> ).
>>>>>>
>>>>>> In fact, all the version changes in this commit
>>>>>> <https://github.com/apache/beam/commit/66fcf4bff19cda5831465ccedbbaa6fbaa648234#diff-600376dffeb79835ede4a0b285078036>
>>>>>> are missing.
>>>>>>
>>>>>> My project's pom.xml contains the following:
>>>>>>
>>>>>> <parent>
>>>>>>
>>>>>> <artifactId>beam-examples-parent</artifactId>
>>>>>>
>>>>>> <groupId>org.apache.beam</groupId>
>>>>>>
>>>>>> <version>2.5.0-SNAPSHOT</version>
>>>>>>
>>>>>> </parent>
>>>>>>
>>>>>> <modelVersion>4.0.0</modelVersion>
>>>>>>
>>>>>>
>>>>>> <groupId>com.google.cloud</groupId>
>>>>>>
>>>>>> <artifactId>xyz</artifactId>
>>>>>>
>>>>>>
>>>>>> <repositories>
>>>>>>
>>>>>> <repository>
>>>>>>
>>>>>> <id>apache.snapshots</id>
>>>>>>
>>>>>> <name>Apache Development Snapshot Repository</name>
>>>>>>
>>>>>> <url>
>>>>>> https://repository.apache.org/content/repositories/snapshots/</url>
>>>>>>
>>>>>> <releases>
>>>>>>
>>>>>> <enabled>false</enabled>
>>>>>>
>>>>>> </releases>
>>>>>>
>>>>>> <snapshots>
>>>>>>
>>>>>> <enabled>true</enabled>
>>>>>>
>>>>>> </snapshots>
>>>>>>
>>>>>> </repository>
>>>>>>
>>>>>> </repositories>
>>>>>>
>>>>>> My first round of attempts to debug included removing ~/.m2/ and a
>>>>>> number of other techniques.
>>>>>>
>>>>>> When I look at
>>>>>> https://repository.apache.org/content/repositories/snapshots, I notice
>>>>>> that beam-parent with 2.5.0-SNAPSHOT
>>>>>> <https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/>
>>>>>> is
>>>>>> very outdated and so all the version numbers inherited from the POM are
>>>>>> wrong (see
>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/).
>>>>>> If you look at
>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/
>>>>>> you will see that the beam-parent is significantly old.
>>>>>>
>>>>>> So, it seems imperative that the snapshot be rebuilt before the
>>>>>> finalized 2.5.0 version is set or any projects depending on the Beam
>>>>>> parent
>>>>>> will break. (I depend upon the Beam parent because trying to keep the
>>>>>> versions of all the different Google projects in sync is a total
>>>>>> nightmare).
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 25, 2018 at 6:58 PM Chamikara Jayalath <
>>>>>> chamik...@google.com> wrote:
>>>>>>
>>>>>>> Ccing Eric for providing more context.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Cham
>>>>>>>
>>>>>>> On Wed, Apr 25, 2018 at 3:38 PM Scott Wegner <sweg...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Do you have any more context on how they were using the parent pom?
>>>>>>>> In the Gradle build, instead of using a parent pom hierarchy we are
>>>>>>>> embedding the complete set of dependencies and metadata in each
>>>>>>>> generated
>>>>>>>> pom. They should be functionally equivalent.
>>>>>>>>
>>>>>>>> On Wed, Apr 25, 2018 at 3:09 PM Chamikara Jayalath <
>>>>>>>> chamik...@google.com> wrote:
>>>>>>>>
>>>>>>>>> At least one user was depending on 'beam-parent' and ran into
>>>>>>>>> issues due to dependency update
>>>>>>>>> https://github.com/apache/beam/pull/5046 not being reflected
>>>>>>>>> there. I'm not sure if this is a usage pattern that we should
>>>>>>>>> discourage.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Cham
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Apr 25, 2018 at 2:36 PM Alan Myrvold <amyrv...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> The gradle build is currently only generating pom files when
>>>>>>>>>> there is a jar file.
>>>>>>>>>> What are these parent poms used for? Do all the parent poms need
>>>>>>>>>> to be generated or just the top level beam-parent?
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 25, 2018 at 2:07 PM Chamikara Jayalath <
>>>>>>>>>> chamik...@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks. Should this be a 2.5.0 blocker ?
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 25, 2018 at 2:05 PM Alan Myrvold <
>>>>>>>>>>> amyrv...@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I think this corresponds with the move from maven to gradle
>>>>>>>>>>>> snapshot releases.
>>>>>>>>>>>> Issue logged as https://issues.apache.org/jira/browse/BEAM-4170
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Apr 25, 2018 at 1:57 PM Chamikara Jayalath <
>>>>>>>>>>>> chamik...@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Seems like we are not building some of the SNAPSHOT artifacts
>>>>>>>>>>>>> since 4/9/2018.
>>>>>>>>>>>>> For example:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-examples-parent/2.5.0-SNAPSHOT/
>>>>>>>>>>>>>
>>>>>>>>>>>>> Users who depend on these artifacts are getting outdated
>>>>>>>>>>>>> dependencies.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any idea why ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Cham
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>> Got feedback? http://go/swegner-feedback
>>>>>>>>
>>>>>>> --
>>>>>
>>>>>
>>>>> Got feedback? http://go/swegner-feedback
>>>>>
>>>>
>
