Re: DebeziumIO dependency update

2021-10-15 Thread Judah Rand
This PR was later reverted due to issues it caused in the Python X-lang
wrapper. I've had a look and I can't work out what the issue is. Would
someone be able to lend a hand? Pablo, you seem to have a lot of experience
with Debezium?

Judah

On Wed, 6 Oct 2021 at 01:38, Luke Cwik  wrote:

> I think this was a double post as I merged an upgrade a couple of days ago.
>
> On Mon, Oct 4, 2021 at 12:02 PM Pablo Estrada  wrote:
>
>> Hi Judah,
>> you should have received an invite for Slack. Let me know if you were
>> able to access it.
>>
>> I think it makes sense to update our Debezium dependency - would you be
>> willing to make the pull request to make the update? If you make it in the
>> next couple of days, we can have it updated on time for Beam 2.34.0.
>> Let me know if you don't have the time so I can pick that up instead.
>> Best
>> -P.
>>
>> On Fri, Oct 1, 2021 at 1:47 PM Judah Rand  wrote:
>>
>>> Hi all,
>>>
>>> I identified the other day that Debezium does not work with many cloud
>>> providers Postgres instances. This includes Azure and Google CloudSQL. I
>>> determined that this is due to the version of Postgres JDBC that Debezium
>>> uses (42.2.14).
>>>
>>> I've been working with the Debezium devs to update this dependency to
>>> 42.2.22 which should solve the connection problems. This dependency update
>>> should be released imminently in Debezium 1.7.0.Final (today or Monday I
>>> suspect).
>>>
>>> See:
>>> https://issues.redhat.com/browse/DBZ-4060
>>> https://github.com/pgjdbc/pgjdbc/issues/1868
>>>
>>> However, this means that for the DebeziumIO connector dependency in
>>> Apache Beam will also need updating. Is this something which people will be
>>> happy with?
>>>
>>> Is it something which could likely be included in the next Beam release?
>>>
>>> Many thanks,
>>> Judah
>>>
>>> PS. Any chance I could get invited to the ASF Slack?
>>>
>>


Re: [VOTE] Release 2.33.0, release candidate 1

2021-10-01 Thread Judah Rand
Might happy with development. Dataflow will pose problems, however.

On Fri, 1 Oct 2021, 20:51 Chamikara Jayalath,  wrote:

>
>
> On Fri, Oct 1, 2021 at 11:49 AM Judah Rand  wrote:
>
>> That's very disappointing as I'd worked very hard over the last few days
>> to get the dependency updated in Debezium and 6 weeks is quite a
>> significant and problematic delay for me.
>>
>> On Fri, 1 Oct 2021, 19:44 Udi Meiri,  wrote:
>>
>>> RC2 progress: I've merged 3 cherrypicks, and then reverted 1 (it
>>> required a BOM (Java dependendencies) update which is too risky).
>>>
>>> I would like to start RC2 ASAP. It is marked as a recent regression
>>> (2.32.0), though the PR that introduced that version [1] is from 2.31.0 so
>>> I assume the problem exists from that version.
>>> I'm leaning towards not cherry picking to avoid potential issues with a
>>> new dependency, but OTOH this would only affect Debezium users using the
>>> expansion service and Postgres, which is both an argument for and against
>>> cherrypicking at this late stage.
>>>
>>> In conclusion, I'll skip this cherrypick.
>>>
>>> [1] https://github.com/apache/beam/pull/14829
>>>
>>>
>>> On Fri, Oct 1, 2021 at 11:36 AM Judah Rand 
>>> wrote:
>>>
>>>> I need to use it via the Python `ReadFromDebezium` interface which
>>>> makes that more tricky as it just goes and fetches the prebuilt jar.
>>>>
>>>
> You can start a custom expansion service with a jar built from HEAD and
> specify it when using transforms in debezium.py [1].
> For example,
>
> ./gradlew :sdks:java:io:debezium:expansion-service:build
> java -jar
> sdks/java/io/debezium/expansion-service/build/libs/beam-sdks-java-io-debezium-expansion-service-2.34.0-SNAPSHOT.jar
> 12345
>
> [1]
> https://github.com/apache/beam/blob/e4b400b5d73e90fc5529a407eac4022eebe95175/sdks/python/apache_beam/io/debezium.py#L142
>
>
>>
>>>> On Fri, 1 Oct 2021, 19:26 Luke Cwik,  wrote:
>>>>
>>>>> Based upon the PR, it doesn't look like there were any code changes.
>>>>> Would overriding the debezium version in your application to 1.7 solve the
>>>>> issue?
>>>>>
>>>>> On Fri, Oct 1, 2021 at 10:08 AM Judah Rand 
>>>>> wrote:
>>>>>
>>>>>> I was wondering if there might be any possibility of cherry-picking
>>>>>> the changes from https://github.com/apache/beam/pull/15636 into the
>>>>>> 2.33.0 release for next week?
>>>>>>
>>>>>> Currently DebeziumIO is completely unusable for both Azure and Google
>>>>>> CloudSQL Postgres instances due to a bug in the Postgres JDBC driver 
>>>>>> which
>>>>>> has only recently been updated in Debezium.
>>>>>>
>>>>>> I’m currently completely blocked from using Beam for our intended use
>>>>>> case.
>>>>>>
>>>>>> Appreciate it’s probably quite late in the day for changes but
>>>>>> consideration would be appreciated.
>>>>>>
>>>>>> On 2021/09/21 17:32:40, Udi Meiri  wrote:
>>>>>> > Hi everyone,>
>>>>>> > Please review and vote on the release candidate #1 for the version
>>>>>> 2.33.0,>
>>>>>> > as follows:>
>>>>>> > [ ] +1, Approve the release>
>>>>>> > [ ] -1, Do not approve the release (please provide specific
>>>>>> comments)>
>>>>>> >
>>>>>> >
>>>>>> > Reviewers are encouraged to test their own use cases with the
>>>>>> release>
>>>>>> > candidate, and vote +1 if>
>>>>>> > no issues are found.>
>>>>>> >
>>>>>> > The complete staging area is available for your review, which
>>>>>> includes:>
>>>>>> > * JIRA release notes [1],>
>>>>>> > * the official Apache source release to be deployed to
>>>>>> dist.apache.org [2],>
>>>>>> > which is signed with the key with fingerprint 587B049C36DAAFE6
>>>>>> [3],>
>>>>>> > * all artifacts to be deployed to the Maven Central Repository
>>>>>> [4],>
>>>>>> > * source code tag "v2.33.0-RC1" [5],>
>>>>&

DebeziumIO dependency update

2021-10-01 Thread Judah Rand
Hi all,

I identified the other day that Debezium does not work with many cloud
providers Postgres instances. This includes Azure and Google CloudSQL. I
determined that this is due to the version of Postgres JDBC that Debezium
uses (42.2.14).

I've been working with the Debezium devs to update this dependency to
42.2.22 which should solve the connection problems. This dependency update
should be released imminently in Debezium 1.7.0.Final (today or Monday I
suspect).

See:
https://issues.redhat.com/browse/DBZ-4060
https://github.com/pgjdbc/pgjdbc/issues/1868

However, this means that for the DebeziumIO connector dependency in Apache
Beam will also need updating. Is this something which people will be happy
with?

Is it something which could likely be included in the next Beam release?

Many thanks,
Judah

PS. Any chance I could get invited to the ASF Slack?


Re: [VOTE] Release 2.33.0, release candidate 1

2021-10-01 Thread Judah Rand
That's very disappointing as I'd worked very hard over the last few days to
get the dependency updated in Debezium and 6 weeks is quite a significant
and problematic delay for me.

On Fri, 1 Oct 2021, 19:44 Udi Meiri,  wrote:

> RC2 progress: I've merged 3 cherrypicks, and then reverted 1 (it required
> a BOM (Java dependendencies) update which is too risky).
>
> I would like to start RC2 ASAP. It is marked as a recent regression
> (2.32.0), though the PR that introduced that version [1] is from 2.31.0 so
> I assume the problem exists from that version.
> I'm leaning towards not cherry picking to avoid potential issues with a
> new dependency, but OTOH this would only affect Debezium users using the
> expansion service and Postgres, which is both an argument for and against
> cherrypicking at this late stage.
>
> In conclusion, I'll skip this cherrypick.
>
> [1] https://github.com/apache/beam/pull/14829
>
>
> On Fri, Oct 1, 2021 at 11:36 AM Judah Rand  wrote:
>
>> I need to use it via the Python `ReadFromDebezium` interface which makes
>> that more tricky as it just goes and fetches the prebuilt jar.
>>
>> On Fri, 1 Oct 2021, 19:26 Luke Cwik,  wrote:
>>
>>> Based upon the PR, it doesn't look like there were any code changes.
>>> Would overriding the debezium version in your application to 1.7 solve the
>>> issue?
>>>
>>> On Fri, Oct 1, 2021 at 10:08 AM Judah Rand 
>>> wrote:
>>>
>>>> I was wondering if there might be any possibility of cherry-picking the
>>>> changes from https://github.com/apache/beam/pull/15636 into the 2.33.0
>>>> release for next week?
>>>>
>>>> Currently DebeziumIO is completely unusable for both Azure and Google
>>>> CloudSQL Postgres instances due to a bug in the Postgres JDBC driver which
>>>> has only recently been updated in Debezium.
>>>>
>>>> I’m currently completely blocked from using Beam for our intended use
>>>> case.
>>>>
>>>> Appreciate it’s probably quite late in the day for changes but
>>>> consideration would be appreciated.
>>>>
>>>> On 2021/09/21 17:32:40, Udi Meiri  wrote:
>>>> > Hi everyone,>
>>>> > Please review and vote on the release candidate #1 for the version
>>>> 2.33.0,>
>>>> > as follows:>
>>>> > [ ] +1, Approve the release>
>>>> > [ ] -1, Do not approve the release (please provide specific
>>>> comments)>
>>>> >
>>>> >
>>>> > Reviewers are encouraged to test their own use cases with the
>>>> release>
>>>> > candidate, and vote +1 if>
>>>> > no issues are found.>
>>>> >
>>>> > The complete staging area is available for your review, which
>>>> includes:>
>>>> > * JIRA release notes [1],>
>>>> > * the official Apache source release to be deployed to
>>>> dist.apache.org [2],>
>>>> > which is signed with the key with fingerprint 587B049C36DAAFE6 [3],>
>>>> > * all artifacts to be deployed to the Maven Central Repository [4],>
>>>> > * source code tag "v2.33.0-RC1" [5],>
>>>> > * website pull request listing the release [6], the blog post [6],
>>>> and>
>>>> > publishing the API reference manual [7].>
>>>> > * Java artifacts were built with Maven 3.6.3 and OpenJDK 1.8.0_181.>
>>>> > * Python artifacts are deployed along with the source release to the>
>>>> > dist.apache.org [2] and pypy[8].>
>>>> > * Validation sheet with a tab for 2.33.0 release to help with
>>>> validation>
>>>> > [9].>
>>>> > * Docker images published to Docker Hub [10].>
>>>> >
>>>> > The vote will be open for at least 72 hours. It is adopted by
>>>> majority>
>>>> > approval, with at least 3 PMC affirmative votes.>
>>>> >
>>>> > For guidelines on how to try the release in your projects, check out
>>>> our>
>>>> > blog post at https://beam.apache.org/blog/validate-beam-release/.>
>>>> >
>>>> > Thanks,>
>>>> > Release Manager>
>>>> >
>>>> > [1]>
>>>> >
>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12350404>
>>>>
>>>> > [2] https://dist.apache.org/repos/dist/dev/beam/2.33.0/>
>>>> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS>
>>>> > [4]
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1234/>
>>>>
>>>> > [5] https://github.com/apache/beam/tree/v2.33.0-RC1>
>>>> > [6] https://github.com/apache/beam/pull/15543>
>>>> > [7] https://github.com/apache/beam-site/pull/619>
>>>> > [8] https://pypi.org/project/apache-beam/2.33.0rc1/>
>>>> > [9]>
>>>> >
>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1705275493>
>>>>
>>>> > [10] https://hub.docker.com/search?q=apache%2Fbeamtype=image>
>>>> >
>>>
>>>


Re: [VOTE] Release 2.33.0, release candidate 1

2021-10-01 Thread Judah Rand
I need to use it via the Python `ReadFromDebezium` interface which makes
that more tricky as it just goes and fetches the prebuilt jar.

On Fri, 1 Oct 2021, 19:26 Luke Cwik,  wrote:

> Based upon the PR, it doesn't look like there were any code changes. Would
> overriding the debezium version in your application to 1.7 solve the issue?
>
> On Fri, Oct 1, 2021 at 10:08 AM Judah Rand  wrote:
>
>> I was wondering if there might be any possibility of cherry-picking the
>> changes from https://github.com/apache/beam/pull/15636 into the 2.33.0
>> release for next week?
>>
>> Currently DebeziumIO is completely unusable for both Azure and Google
>> CloudSQL Postgres instances due to a bug in the Postgres JDBC driver which
>> has only recently been updated in Debezium.
>>
>> I’m currently completely blocked from using Beam for our intended use
>> case.
>>
>> Appreciate it’s probably quite late in the day for changes but
>> consideration would be appreciated.
>>
>> On 2021/09/21 17:32:40, Udi Meiri  wrote:
>> > Hi everyone,>
>> > Please review and vote on the release candidate #1 for the version
>> 2.33.0,>
>> > as follows:>
>> > [ ] +1, Approve the release>
>> > [ ] -1, Do not approve the release (please provide specific comments)>
>> >
>> >
>> > Reviewers are encouraged to test their own use cases with the release>
>> > candidate, and vote +1 if>
>> > no issues are found.>
>> >
>> > The complete staging area is available for your review, which
>> includes:>
>> > * JIRA release notes [1],>
>> > * the official Apache source release to be deployed to dist.apache.org
>> [2],>
>> > which is signed with the key with fingerprint 587B049C36DAAFE6 [3],>
>> > * all artifacts to be deployed to the Maven Central Repository [4],>
>> > * source code tag "v2.33.0-RC1" [5],>
>> > * website pull request listing the release [6], the blog post [6], and>
>> > publishing the API reference manual [7].>
>> > * Java artifacts were built with Maven 3.6.3 and OpenJDK 1.8.0_181.>
>> > * Python artifacts are deployed along with the source release to the>
>> > dist.apache.org [2] and pypy[8].>
>> > * Validation sheet with a tab for 2.33.0 release to help with
>> validation>
>> > [9].>
>> > * Docker images published to Docker Hub [10].>
>> >
>> > The vote will be open for at least 72 hours. It is adopted by majority>
>> > approval, with at least 3 PMC affirmative votes.>
>> >
>> > For guidelines on how to try the release in your projects, check out
>> our>
>> > blog post at https://beam.apache.org/blog/validate-beam-release/.>
>> >
>> > Thanks,>
>> > Release Manager>
>> >
>> > [1]>
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12350404>
>>
>> > [2] https://dist.apache.org/repos/dist/dev/beam/2.33.0/>
>> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS>
>> > [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1234/>
>> > [5] https://github.com/apache/beam/tree/v2.33.0-RC1>
>> > [6] https://github.com/apache/beam/pull/15543>
>> > [7] https://github.com/apache/beam-site/pull/619>
>> > [8] https://pypi.org/project/apache-beam/2.33.0rc1/>
>> > [9]>
>> >
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1705275493>
>>
>> > [10] https://hub.docker.com/search?q=apache%2Fbeamtype=image>
>> >
>
>


Re: [VOTE] Release 2.33.0, release candidate 1

2021-10-01 Thread Judah Rand
I was wondering if there might be any possibility of cherry-picking the changes 
from https://github.com/apache/beam/pull/15636 into the 2.33.0 release for next 
week? 

Currently DebeziumIO is completely unusable for both Azure and Google CloudSQL 
Postgres instances due to a bug in the Postgres JDBC driver which has only 
recently been updated in Debezium. 

I’m currently completely blocked from using Beam for our intended use case. 

Appreciate it’s probably quite late in the day for changes but consideration 
would be appreciated.

On 2021/09/21 17:32:40, Udi Meiri  wrote: 
> Hi everyone,> 
> Please review and vote on the release candidate #1 for the version 2.33.0,> 
> as follows:> 
> [ ] +1, Approve the release> 
> [ ] -1, Do not approve the release (please provide specific comments)> 
> 
> 
> Reviewers are encouraged to test their own use cases with the release> 
> candidate, and vote +1 if> 
> no issues are found.> 
> 
> The complete staging area is available for your review, which includes:> 
> * JIRA release notes [1],> 
> * the official Apache source release to be deployed to dist.apache.org [2],> 
> which is signed with the key with fingerprint 587B049C36DAAFE6 [3],> 
> * all artifacts to be deployed to the Maven Central Repository [4],> 
> * source code tag "v2.33.0-RC1" [5],> 
> * website pull request listing the release [6], the blog post [6], and> 
> publishing the API reference manual [7].> 
> * Java artifacts were built with Maven 3.6.3 and OpenJDK 1.8.0_181.> 
> * Python artifacts are deployed along with the source release to the> 
> dist.apache.org [2] and pypy[8].> 
> * Validation sheet with a tab for 2.33.0 release to help with validation> 
> [9].> 
> * Docker images published to Docker Hub [10].> 
> 
> The vote will be open for at least 72 hours. It is adopted by majority> 
> approval, with at least 3 PMC affirmative votes.> 
> 
> For guidelines on how to try the release in your projects, check out our> 
> blog post at https://beam.apache.org/blog/validate-beam-release/.> 
> 
> Thanks,> 
> Release Manager> 
> 
> [1]> 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12350404>
>  
> [2] https://dist.apache.org/repos/dist/dev/beam/2.33.0/> 
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS> 
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1234/> 
> [5] https://github.com/apache/beam/tree/v2.33.0-RC1> 
> [6] https://github.com/apache/beam/pull/15543> 
> [7] https://github.com/apache/beam-site/pull/619> 
> [8] https://pypi.org/project/apache-beam/2.33.0rc1/> 
> [9]> 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1705275493>
>  
> [10] https://hub.docker.com/search?q=apache%2Fbeamtype=image> 
> 

DebeziumIO dependency update

2021-10-01 Thread Judah Rand
Hi all,

I identified the other day that Debezium does not work with many cloud
providers Postgres instances. This includes Azure and Google CloudSQL. I
determined that this is due to the version of Postgres JDBC that Debezium
uses (42.2.14).

I've been working with the Debezium devs to update this dependency to
42.2.22 which should solve the connection problems. This dependency update
should be released imminently in Debezium 1.7.0.Final (today or Monday I
suspect).

See:
https://issues.redhat.com/browse/DBZ-4060
https://github.com/pgjdbc/pgjdbc/issues/1868
https://github.com/apache/beam/pull/15636

However, this means that for the DebeziumIO connector dependency in Apache
Beam will also need updating. Is this something which people will be happy
with?

Is it something which could likely be included in the next Beam release?

Many thanks,
Judah

PS. Any chance I could get invited to the ASF Slack?