Hi Daniel,
(You helped me apply some change to this strange setup a few months back.
Thank you for working on rectifying the situation.)
I like that idea overall.
Question 1: How are you going to approach testing/CI?
The pull requests in the java-pubsublite repo do not trigger Beam repo's
CI. You want to deliver things to your customers after they are tested as
much as possible.
Question2 : in the code below, what is the purpose of keeping the
PubsubLiteIO in the Beam repo?
```
class PubsubLiteIO extends com.google.cloud.pubsublite.beam.PubsubLiteIO {}
````
The backward compatibility came to my mind but I thought you may have more
reasons.
My memo:
java-pubsublite repsitory has:
https://github.com/googleapis/java-pubsublite/blob/master/pubsublite-beam-io/src/main/java/com/google/cloud/pubsublite/beam/PubsubLiteIO.java
beam repo has:
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteIO.java
(and other files in the same directory)
google-cloud-pubsublite is not part of the Libraries BOM (yet) because of
its pre-1.0 status.
On Thu, Jun 17, 2021 at 5:07 PM Daniel Collins <[email protected]> wrote:
> I don't know that the cycle would cause a problem- wouldn't it override
> and cause it to use beam-sdks-java-core:2.30.0 (at least until beam goes to
> 3.X.X)?
>
> Something we can do if this is an issue is mark pubsublite-beam-io's dep
> on beam-sdks-java-core as 'provided'. But I'd prefer to avoid this and just
> let overriding fix it if that works.
>
> On Thu, Jun 17, 2021 at 4:15 PM Andrew Pilloud <[email protected]>
> wrote:
>
>> How do you plan to address the circular dependency? Won't this end up
>> with Beam depending on older versions of itself?
>>
>> beam-sdks-java-io-google-cloud-platform:2.30.0 ->
>> pubsublite-beam-io:0.16.0 -> beam-sdks-java-core:2.29.0
>>
>> On Thu, Jun 17, 2021 at 11:56 AM Daniel Collins <[email protected]>
>> wrote:
>>
>>> Hello beam developers,
>>>
>>> I'm the primary author of the Pub/Sub Lite I/O, and I'd like to get some
>>> feedback on a change to the model for hosting this I/O in beam. Our team
>>> has been frustrated by the fact that we have no way to release features or
>>> fixes for bugs to customers on time scales shorter than the 1-2 months of
>>> the beam release cycle, and that those fixes are necessarily coupled with a
>>> beam version upgrade. To work around this, I forked the I/O in beam to our
>>> own repo about 6 months ago and have been maintaining both copies in
>>> parallel.
>>>
>>> I'd like to retain our ability to quickly fix and improve the I/O while
>>> retaining end-user visibility within the beam repo. To do this, I'd like
>>> to remove all the implementation from the beam repo, and leave the I/O
>>> there implemented as:
>>>
>>> ```
>>> class PubsubLiteIO extends com.google.cloud.pubsublite.beam.PubsubLiteIO
>>> {}
>>> ````
>>> , and add a dependency on our beam artifact.
>>>
>>> This enables beam users who want to just use the
>>> beam-sdks-java-io-google-cloud-platform artifact to do so, but they can
>>> also track the canonical version separately in our repo to get fixes and
>>> improvements at a faster rate. All static methods from the parent class
>>> would be available on the class in the beam repo.
>>>
>>> I'd be interested to hear anyones thoughts and suggestions surrounding
>>> this.
>>>
>>> -Daniel
>>>
>>
--
Regards,
Tomo
