Re: [DISCUSS] Migrate Jira to GitHub Issues?

[Jean-Baptiste Onofre]

Hi,

No problem for me. The only thing I don’t like with GitHub issues is that fact 
that it’s not possible to “assign” several milestones to an issue.
When we maintain several active branch/version, it sucks (one issue == one 
milestone), as we have to create several issue.

Regards
JB

> Le 10 déc. 2021 à 01:28, Kyle Weaver  a écrit :
> 
> I’m in favor of switching to Github issues. I can’t think of a single thing 
> jira does better.
> 
> Thanks Jarek, this is a really great resource [1]. For another reference, the 
> Calcite project is engaged in the same discussion right now [2]. I came up 
> with many of the same points independently before I saw their thread.
> 
> When evaluating feature parity, we should make a distinction between 
> non-structured (text) and structured data. And we don’t need a strict 
> mechanical mapping for everything unless we’re planning on automatically 
> migrating all existing issues. I don’t see the point in automatic migration, 
> though; as Jarek pointed out, we’d end up perpetuating a ton of obsolete 
> issues.
> 
>   • We use nested issues and issue relations in jira, but as far as I 
> know robots don’t use them and we don’t query them much, so we’re not losing 
> anything by moving from an API to plain English descriptions: “This issue is 
> blocked by issue #n.” Mentions show up automatically on other issues.
>   • For component, type, priority, etc., we can use Github labels.
>   • Version(s) affected is used inconsistently, and as far as I know only 
> by humans, so a simple English description is fine. We can follow the example 
> of other projects and make the version affected a part of the issue template.
>   • For fix version, which we use to track which issues we want to fix in 
> upcoming releases, as well as automatically generate release notes: Github 
> has “milestones,” which can be marked on PRs or issues, or both.
>   • IMO the automatically generated JIRA release notes are not 
> especially useful anyway. They are too detailed for a quick summary, and not 
> precise enough to show everything. For a readable summary, we use CHANGES.md 
> to highlight changes we especially want users to know about. For a complete 
> list of changes, there’s the git commit log, which is the ultimate source of 
> truth.
>   • We’d only want to preserve reporter and assignee if we’re planning on 
> migrating everything automatically, and even then I think it’d be fine to 
> compile a map of active contributors and drop the rest.
> 
> As for the advantages of switching (just the ones off the top of my head):
>   • As others have mentioned, it’s less burden for new contributors to 
> create new issues and comment on existing ones.
>   • Effortless linking between issues and PRs.
>   • Github -> jira links were working for a short while, but they 
> seem to be broken at the moment.
>   • Jira -> github links only show: “links to GitHub Pull Request 
> #x”. They don’t say the status of the PR, so you have to follow the link 
> to find out. Especially inconvenient when one jira maps to several PRs, and 
> you have to open all the links to get a summary of what work was done.
>   • When you mention a GH issue in a pull request, a link to the 
> PR will automatically appear on the issue, including not just the ID but also 
> the PR’s description and status (open/closed/draft/merged/etc.), and if you 
> hover it will show a preview as well.
>   • We frequently merge a PR and then forget to mark the jira as 
> closed. Whereas if a PR is linked to a GH issue using the “closes” keyword, 
> the GH issue will automatically be closed [3].
>   • I don’t have to look up or guess whether a github account and jira 
> account belong to the same person.
>   • There’s a single unified search bar to find issues, PRs, and code.
>   • Github enables markdown formatting everywhere, which is more or less 
> the industry standard, whereas Jira has its own bespoke system [4]. 
>   • In GH issues, links to Github code snippets will automatically 
> display the code snippet inline.
>   • GH labels are scoped to each project, whereas ASF Jira labels are an 
> unmanageable, infinitely growing namespace (see “flake,” “flaky,” “flakey,” 
> “Flaky,” “flaky-test”...).
> 
> [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191332632
> [2] 
> https://mail-archives.apache.org/mod_mbox/calcite-dev/202112.mbox/%3CCAB%3DJe-EuaijDjwb6umU_N2TaqFZawE%2BUbgZAgZYvrgPFypfAYQ%40mail.gmail.com%3E
> [3] 
> https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
> [4] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191332632
> https://jira.atlassian.com/secure/WikiRendererHelpAction.jspa?section=all
> 
> 
> On Wed, Dec 8, 2021 at 9:13 AM Alexey Romanenko  
> wrote:
> Many thanks for 

Re: Intro

[Jean-Baptiste Onofre]

Welcome ;)

Regards
JB

> Le 22 oct. 2021 à 15:52, Moritz Mack  a écrit :
> 
> Hi all,
>  
> I’m very much looking forward to start contributing to Beam and just want to 
> briefly introduce myself.
>  
> My name is Moritz (mosche) and I’m working together with Alexey and Etienne. 
> Having worked mostly with Spark in the past, I’m excited to dive deeper into 
> Beam 
>  
> Looking forward to working with all of you!
>  
> Kind regards from Munich,
> Moritz
>  
> As a recipient of an email from Talend, your contact personal data will be on 
> our systems. Please see our privacy notice (updated August 2020) at Talend, 
> Inc. 
> 

Re: [VOTE] Vendored Dependencies Release Byte Buddy 1.11.0

[Jean-Baptiste Onofre]

+1 (binding)

Regards
JB

> Le 20 mai 2021 à 10:56, Etienne Chauchot  a écrit :
> 
> +1 (binding) on releasing vendored bytebuddy for testing in 
> https://github.com/apache/beam/pull/14824 
> 
> Etienne
> 
> On 19/05/2021 23:43, Kai Jiang wrote:
>> +1 (non-binding)
>> 
>> On Wed, May 19, 2021 at 12:23 PM Jan Lukavský > > wrote:
>> +1 (non-binding)
>> 
>> verified correct shading.
>> 
>>  Jan
>> 
>> On 5/19/21 8:53 PM, Ismaël Mejía wrote:
>>> This release is only to publish the vendored dependency artifacts. We need 
>>> those to integrate it and be able to verify if it causes problems or not. 
>>> The PR for this is already opened but it needs the artifacts of this vote 
>>> to be ran.
>>> https://github.com/apache/beam/pull/14824 
>>> 
>>> 
>>> For ref there is a document on how to release and validate releases of 
>>> Beam's vendored dependencies that can be handy to anyone wishing to help 
>>> validate:
>>> https://s.apache.org/beam-release-vendored-artifacts 
>>> 
>>> On Wed, May 19, 2021 at 8:45 PM Tyson Hamilton >> > wrote:
>>> I'd like to help, but I don't know how to determine whether this upgrade is 
>>> going to cause problems or not. Are there tests I should look at, or some 
>>> validation I should perform?
>>> 
>>> On Wed, May 19, 2021 at 11:29 AM Ismaël Mejía >> > wrote:
>>> Kind reminder, the vote is ongoing
>>> 
>>> On Mon, May 17, 2021 at 5:32 PM Ismaël Mejía >> > wrote:
>>> Please review the release of the following artifacts that we vendor:
>>>  * beam-vendor-bytebuddy-1_11_0
>>> 
>>> Hi everyone,
>>> Please review and vote on the release candidate #1 for the version 0.1, as 
>>> follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>> 
>>> The complete staging area is available for your review, which includes:
>>> * the official Apache source release to be deployed to dist.apache.org 
>>>  [1], which is signed with the key with 
>>> fingerprint 3415631729E15B33051ADB670A9DAF6713B86349 [2],
>>> * all artifacts to be deployed to the Maven Central Repository [3],
>>> * commit hash "d93c591deb21237ddb656583d7ef7a4debba" [4],
>>> 
>>> The vote will be open for at least 72 hours. It is adopted by majority 
>>> approval, with at least 3 PMC affirmative votes.
>>> 
>>> Thanks,
>>> Release Manager
>>> 
>>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/ 
>>> 
>>> [2] https://dist.apache.org/repos/dist/release/beam/KEYS 
>>> 
>>> [3] https://repository.apache.org/content/repositories/orgapachebeam-1166/ 
>>> 
>>> [4] 
>>> https://github.com/apache/beam/commit/d93c591deb21237ddb656583d7ef7a4debba
>>>  
>>> 
>>> 

Re: LGPL-2.1 in beam-vendor-grpc

[Jean-Baptiste Onofre]

+1 fully agree.

Regards
JB

> Le 10 mai 2021 à 16:02, Jan Lukavský  a écrit :
> 
> +1 for blocking the release - I think we should not release something about 
> which we _know_ that it might be legally problematic. And we should 
> definitely create a check in the build process that will warn about such 
> issues in the future.
> 
>  Jan
> 
> On 5/10/21 3:44 PM, Ismaël Mejía wrote:
>> Tomo just confirmed in the ticket that if we update the gRPC vendored 
>> version we won't need the JBoss dependency anymore so we should be good to 
>> go with the upgrade. The open question is if this should be blocking for the 
>> upcoming Beam 2.31.0 release or we can fix it afterwards.
>> 
>> 
>> On Mon, May 10, 2021 at 2:46 PM Ismaël Mejía > > wrote:
>> We have been discussing about updating the vendored dependency in BEAM-11227 
>> , if I remember correctly 
>> the newer version of gRPC does not require the jboss dependency, so probably 
>> is the best upgrade path, can you confirm Tomo Suzuki 
>>  ?
>> 
>> On Mon, May 10, 2021 at 2:33 PM Jarek Potiuk > > wrote:
>> Also we have very similar discussion about it in 
>> https://issues.apache.org/jira/browse/LEGAL-572 
>>  
>> Just to be clear about the context of it, it's not a legal requirement of 
>> Apache Licence, it's Apache Software Foundation policy, that we should not 
>> limit   our users in using our software. If the LGPL 
>> dependency is "optional", it's fine to add such optional dependency. If it 
>> is "required" to run your software, then it is not allowed as it limits the 
>> users of ASF software in further redistributing the software in the way they 
>> want (this is at least my understanding of it). 
>> 
>> On Mon, May 10, 2021 at 12:58 PM JB Onofré > > wrote:
>> Hi
>> 
>> You can take a look on
>> 
>> https://www.apache.org/legal/resolved.html 
>> 
>> 
>> Regards 
>> JB
>> 
>>> Le 10 mai 2021 à 12:56, Elliotte Rusty Harold >> > a écrit :
>>> 
>>> Anyone have a link to the official Apache policy about this? Thanks.
>>> 
>>> On Mon, May 10, 2021 at 10:07 AM Jan Lukavský >> > wrote:
 
 Hi,
 
 we are bundling dependencies with LGPL-2.1, according to license header
 in META-INF/maven/org.jboss.modules/jboss-modules/pom.xml. I think is
 might be an issue, already reported here: [1]. I created [2] to track it
 on our side.
 
  Jan
 
 [1] https://issues.apache.org/jira/browse/FLINK-22555 
 
 
 [2] https://issues.apache.org/jira/browse/BEAM-12316 
 
 
>>> 
>>> 
>>> -- 
>>> Elliotte Rusty Harold
>>> elh...@ibiblio.org 
>> 
>> 
>> -- 
>> +48 660 796 129

Re: Lots of branches

[Jean-Baptiste Onofre]

+1

On other projects, I have script mostly doing:

$ git branch —merged
$ git branch -d MERGED
$ git remote prune REMOTE

Regards
JB

> Le 8 mai 2021 à 02:35, Ahmet Altay  a écrit :
> 
> Hello all,
> 
> Our git repo has lots of branches for cherry picks, patches, reverts etc. I 
> believe these are artifacts of github's easy to use online editor. If you no 
> longer need those, could you please clean them?
> 
> Have a great weekend!
> Ahmet

Re: [VOTE] Release 2.29.0, release candidate #1

[Jean-Baptiste Onofre]

+1 (binding)

Regards
JB

> Le 16 avr. 2021 à 06:01, Kenneth Knowles  a écrit :
> 
> Hi everyone,
> 
> Please review and vote on the release candidate #1 for the version 2.29.0, as 
> follows:
> 
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1 if no issues are found.
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org 
>  [2], which is signed with the key with fingerprint 
> 03DBA3E6ABDD04BFD1558DC16ED551A8AE02461C [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.29.0-RC1" [5],
> * website pull request listing the release [6], publishing the API reference 
> manual [7], and the blog post [8].
> * Java artifacts were built with Maven MAVEN_VERSION and OpenJDK/Oracle JDK 
> JDK_VERSION.
> * Python artifacts are deployed along with the source release to the 
> dist.apache.org  [2].
> * Validation sheet with a tab for 2.29.0 release to help with validation [9].
> * Docker images published to Docker Hub [10].
> 
> New this time around: the Python artifacts are published to pypi as a 
> pre-release version [11].
> 
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> I want to let everyone know that I will be offline on vacation tomorrow, so 
> if you find an issue it would be great if you can also propose a fix and it 
> is fine for any committer to merge it to the release branch in my absence so, 
> if necessary, I can get right on RC2 when I return.
> 
> Kenn
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12349629
>  
> 
> [2] https://dist.apache.org/repos/dist/dev/beam/2.29.0 
> 
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS 
> 
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1165/ 
> 
> [5] https://github.com/apache/beam/tree/v2.29.0-RC1 
> 
> [6] https://github.com/apache/beam/pull/14556 
> 
> [7] https://github.com/apache/beam-site/pull/612 
> 
> [8] https://github.com/apache/beam/pull/14562 
> 
> [9] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1652881637
>  
> 
> [10] https://hub.docker.com/search?q=apache%2Fbeam=image 
> 
> [11] https://pypi.org/project/apache-beam/2.29.0rc1/ 
> 

Re: [DISCUSS] Drop support for Flink 1.8 and 1.9

[Jean-Baptiste Onofre]

+1

Regards
JB

> Le 11 mars 2021 à 19:18, Ismaël Mejía  a écrit :
> 
> Hello,
> 
> We have been supporting older versions of Flink that we had agreed in previous
> discussions where we said we will be supporting only the latest three releases
> [1].
> 
> I would like to propose that for Beam 2.30.0 we stop supporting Flink 1.8 and
> 1.9 [2].  I prepared a PR for this [3] but of course I wanted to bring the
> subject here (and to user@) for your attention and in case someone has a
> different opinion or reason to still support the older versions.
> 
> WDYT?
> 
> Regards,
> Ismael
> 
> [1] 
> https://lists.apache.org/thread.html/rfb5ac9d889d0e3f4400471de3c25000a15352bde879622c899d97581%40%3Cdev.beam.apache.org%3E
> [2] https://issues.apache.org/jira/browse/BEAM-11948
> [3] https://github.com/apache/beam/pull/14203

Re: [VOTE] Release vendor-calcite-1_26_0 version 0.1, release candidate #1

[Jean-Baptiste Onofre]

+1 (binding)

Regards
JB

> Le 8 mars 2021 à 20:37, Andrew Pilloud  a écrit :
> 
> Hi everyone,
> Please review and vote on the release candidate #1 for 
> beam-vendor-calcite-1_26_0 version 0.1, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1 if no issues are found.
> 
> The complete staging area is available for your review, which includes:
> * the official Apache source release to be deployed to dist.apache.org 
>  [1], which is signed with the key with fingerprint 
> 9E7CEC0661EFD610B632C610AE8FE17F9F8AE3D4 [2],
> * all artifacts to be deployed to the Maven Central Repository [3],
> * source code commit "0f52187e344dad9bca4c183fe51151b733b24e35" [4].
> 
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Andrew
> 
> [1] 
> https://dist.apache.org/repos/dist/dev/beam/vendor/beam-vendor-calcite-1_26_0/0.1/
>  
> 
> [2] https://dist.apache.org/repos/dist/release/beam/KEYS 
> 
> [3] https://repository.apache.org/content/repositories/orgapachebeam-1163/ 
> 
> [4] 
> https://github.com/apache/beam/tree/0f52187e344dad9bca4c183fe51151b733b24e35 
> 

Re: [VOTE] Release 2.28.0, release candidate #1

[Jean-Baptiste Onofre]

+1 (binding)

Regards
JB

> Le 15 févr. 2021 à 21:01, Chamikara Jayalath  a écrit :
> 
> Hi everyone,
> 
> Please review and vote on the release candidate #1 for the version 2.28.0, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1 if no issues are found.
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org 
>  [2], which is signed with the key with fingerprint 
> 0CCD5EB2A718A56E [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.28.0-RC1" [5],
> * website pull request listing the release [6], publishing the API reference 
> manual [7], and the blog post [8].
> * Java artifacts were built with Maven 3.3.9 and OpenJDK/Oracle JDK 1.8.0_181.
> * Python artifacts are deployed along with the source release to the 
> dist.apache.org  [2].
> * Validation sheet with a tab for 2.28.0 release to help with validation [9].
> * Docker images published to Docker Hub [10].
> 
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Cham
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12349499
>  
> 
> [2] https://dist.apache.org/repos/dist/dev/beam/2.28.0/ 
> 
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS 
> 
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1161/ 
> 
> [5] https://github.com/apache/beam/tree/v2.28.0-RC1 
> 
> [6] https://github.com/apache/beam/pull/13991 
> 
> [7] https://github.com/apache/beam-site/pull/611 
> 
> [8] https://github.com/apache/beam/pull/13992 
> 
> [9] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=283321283
>  
> 
> [10] https://hub.docker.com/search?q=apache%2Fbeam=image 
> 

Re: BEAM-6516 - RabbitMqIO.Read - Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method(reply-code=406, reply-text=PRECONDITION_FAILED - unknow

[Jean-Baptiste Onofre]

Hi,

Let me take a look.

Regards
JB

> Le 15 janv. 2021 à 20:43, Rafael Ribeiro  a écrit :
> 
> hi All,
> 
> I'm facing the problem reported on 
> https://issues.apache.org/jira/browse/BEAM-6516 
> 
> 
> running a flex template based on kafka-to-bigquery and adapted to rabbitmq
> 
> java.lang.RuntimeException: Exception while finalizing checkpoint
> at 
> org.apache.beam.runners.dataflow.worker.StreamingModeExecutionContext.lambda$flushState$0
>  (StreamingModeExecutionContext.java:403 
> )
> at 
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.lambda$callFinalizeCallbacks$3
>  (StreamingDataflowWorker.java:1218 
> )
> at java.util.concurrent.ThreadPoolExecutor.runWorker 
> (ThreadPoolExecutor.java:1149 
> )
> at java.util.concurrent.ThreadPoolExecutor$Worker.run 
> (ThreadPoolExecutor.java:624 
> )
> at java.lang.Thread.run (Thread.java:748 
> )
> Caused by: java.io.IOException
> at com.rabbitmq.client.impl.AMQChannel.wrap (AMQChannel.java:129 
> )
> at com.rabbitmq.client.impl.AMQChannel.wrap (AMQChannel.java:125 
> )
> at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc (AMQChannel.java:147 
> )
> at com.rabbitmq.client.impl.ChannelN.txCommit (ChannelN.java:1540 
> )
> at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.txCommit 
> (AutorecoveringChannel.java:663 
> )
> Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; 
> protocol method: #method(reply-code=406, 
> reply-text=PRECONDITION_FAILED - unknown delivery tag 2, class-id=60, 
> method-id=80)
> at com.rabbitmq.utility.ValueOrException.getValue (ValueOrException.java:66 
> )
> at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue 
> (BlockingValueOrException.java:36 
> )
> at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply 
> (AMQChannel.java:502 
> )
> at com.rabbitmq.client.impl.AMQChannel.privateRpc (AMQChannel.java:293 
> )
> at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc (AMQChannel.java:141 
> )
> Caused by: 

Re: [VOTE] Release 2.27.0, release candidate #4

[Jean-Baptiste Onofre]

+1 (binding)

Regards
JB

> Le 6 janv. 2021 à 08:17, Pablo Estrada  a écrit :
> 
> Hi everyone,
> Please review and vote on the release candidate #4 for the version 2.27.0, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> NOTE. What happened to RC #2? I started building RC2 before completing all 
> the cherry-picks, so the tag for RC2 was created on an incorrect commit.
> 
> NOTE. What happened to RC #3? I started building RC3, but a new bug was 
> discovered (BEAM-11569) that required amending the branch. Thus this is now 
> RC4.
> 
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1
>  if no issues are found.
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org 
>  [2], which is signed with the key with fingerprint 
> C79DDD47DAF3808F0B9DDFAC02B2D9F742008494 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.27.0-RC4" [5],
> * website pull request listing the release [6], publishing the API reference 
> manual [7], and the blog post [8].
> * Python artifacts are deployed along with the source release to the 
> dist.apache.org  [2].
> * Validation sheet with a tab for 2.27.0 release to help with validation [9].
> * Docker images published to Docker Hub [10].
> 
> The vote will be open for at least 72 hours, but given the holidays, we will 
> likely extend for a few more days. The release will be adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> -P.
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12349380
>  
> 
>  
> [2] https://dist.apache.org/repos/dist/dev/beam/2.27.0/ 
> 
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS 
> 
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1149/ 
>  
> [5] https://github.com/apache/beam/tree/v2.27.0-RC4 
>  
> [6] https://github.com/apache/beam/pull/13602 
>  
> [7] https://github.com/apache/beam-site/pull/610 
>  
> [8] https://github.com/apache/beam/pull/13603 
>  
> [9] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=194829106
>  
> 
>  
> [10] https://hub.docker.com/search?q=apache%2Fbeam=image 
> 

Re: [VOTE] Release 2.27.0, release candidate #1

[Jean-Baptiste Onofre]

+1 (binding)

Regards
JB

> Le 23 déc. 2020 à 06:46, Pablo Estrada  a écrit :
> 
> Hi everyone,
> Please review and vote on the release candidate #1 for the version 2.27.0, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1
>  if no issues are found.
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org 
>  [2], which is signed with the key with fingerprint 
> C79DDD47DAF3808F0B9DDFAC02B2D9F742008494 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.27.0-RC1" [5],
> * website pull request listing the release [6], publishing the API reference 
> manual [7], and the blog post [8].
> * Python artifacts are deployed along with the source release to the 
> dist.apache.org  [2].
> * Validation sheet with a tab for 2.27.0 release to help with validation [9].
> * Docker images published to Docker Hub [10].
> 
> The vote will be open for at least 72 hours, but given the holidays, we will 
> likely extend for a few more days. The release will be adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> -P.
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12349380
>  
> 
>  
> [2] https://dist.apache.org/repos/dist/dev/beam/2.27.0/ 
> 
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS 
> 
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1145/ 
> 
> [5] https://github.com/apache/beam/tree/v2.27 
> .0-RC1
> [6] https://github.com/apache/beam/pull/13602 
>  
> [7] https://github.com/apache/beam-site/pull/610 
>  
> [8] https://github.com/apache/beam/pull/13603 
>  
> [9] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=194829106
>  
> 
>  
> [10] https://hub.docker.com/search?q=apache%2Fbeam=image 
> 

Re: [VOTE] Release 2.26.0, release candidate #1

[Jean-Baptiste Onofre]

+1 (binding)

Sorry for the delay.

Regards
JB

> Le 10 déc. 2020 à 17:40, Tyson Hamilton  a écrit :
> 
> +1 from me. I validated Nexmark performance tests.
> 
> On Tue, Dec 8, 2020 at 7:53 PM Robert Burke  > wrote:
> I'm +1 on RC1 based on the 7 tests I know I can check successfully.  I'll be 
> trying more tomorrow, but remember that release validation requires the 
> community to validate it meets our standards, and I can't do it alone.
> 
> Remember you can participate in the release validation by reviewing parts of 
> the documentation being published as well, not just by running the Pyhton and 
> Java artifacts.
> 
>  If you have contributed new python or java docs into this release, they'll 
> appear in the to be published docs.
> 
> Cheers,
> Robert Burke
> 2.26.0 release manager
> 
> On Mon, Dec 7, 2020, 6:25 PM Robert Burke  > wrote:
> Turns out no changes required affecting the dataflow artifacts this time 
> around, so Dataflow is cleared for testing.
> 
> Cheers.
> Robert Burke
> 2.26.0 Release Manager
> 
> On Mon, Dec 7, 2020, 6:03 PM Robert Burke  > wrote:
> 
> Robert Burke mailto:r...@google.com>>
> Thu, Dec 3, 8:01 PM (4 days ago)
> 
> to dev
> Hi everyone,
> Please review and vote on the release candidate #1 for the version 2.26.0, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> Reviewers are encouraged to test their own use cases with the release 
> candidate, and vote +1
>  if no issues are found.
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org 
>  [2], which is signed with the key with fingerprint 
> A52F5C83BAE26160120EC25F3D56ACFBFB2975E1 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.26.0-RC1" [5],
> * website pull request listing the release [6], publishing the API reference 
> manual [7], and the blog post [8].
> * Java artifacts were built with Maven 3.6.0 and OpenJDK 1.8.0_275.
> * Python artifacts are deployed along with the source release to the 
> dist.apache.org  [2].
> * Validation sheet with a tab for 2.26.0 release to help with validation [9].
> * Docker images published to Docker Hub [10].
> 
> The vote will be open for at least 72 hours (10th ~6pm PST). It is adopted by 
> majority approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Robert Burke
> 2.26.0 Release Manager
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12348833
>  
> 
> [2] https://dist.apache.org/repos/dist/dev/beam/2.26.0/ 
> 
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS 
> 
> [4] https://repository.apache.org/content/repositories/org 
>  apache beam-1144/
> [5] https://github.com/apache/beam/tree/v2.26.0-RC1 
> 
> [6] https://github.com/apache/beam/pull/13481 
> 
> [7] https://github.com/apache/beam-site/pull/609 
> 
> [8] https://github.com/apache/beam/pull/13482 
> 
> [9] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=475997301
>  
> 
> [10] https://hub.docker.com/search?q=apache%2Fbeam=image 
> 
> 
> PS. New Dataflow artifacts likely need to be built and published but this 
> doesn't block vetting the remainder of the RC at this time. Thank you for 
> your patience.
> 

Re: DRAFT - Beam board report June 2020

[Jean-Baptiste Onofre]

Hi,

It looks good with the latest proposed changes.

Regards
JB

> Le 9 juin 2020 à 20:36, Kenneth Knowles  a écrit :
> 
> Ping! It is now June, and time to submit this report. Please add interesting 
> tidbits from the last quarter. Perhaps find highlights in 
> https://github.com/apache/beam/blob/master/CHANGES.md 
> 
> 
> Kenn
> 
> On Wed, Mar 25, 2020 at 10:40 AM Kenneth Knowles  > wrote:
> Hi all,
> 
> I just finally got a chance to finish and submit the March board report 
> (late, sorry).
> 
> I want to have the board report draft available earlier so we can make notes 
> whenever things happen. Just like CHANGES.md is for the code, this is for the 
> project/community.
> 
> https://s.apache.org/beam-report-2020-06 
> 
> 
> You can read past reports at 
> https://whimsy.apache.org/board/minutes/Beam.html 
>  to get a feel for it. 
> Here are some specific examples of things that are good to add:
> 
>  - interesting technical discussions that steer the project
>  - major integrations with other projects
>  - community events
>  - major user facing addition/deprecation (like the Flink and Python version 
> and LTS discussions)
> 
> It is OK to add very rough data and not be too careful with language. I will 
> play editor and make it all fit together.
> 
> Kenn

Re: Difficulties with triggering PRs Jenkins jobs

[Jean-Baptiste Onofre]

Hi,


Actually, it’s because of the PR plugin we are using on Jenkins.

We have the choice between two plugins:
- the "old/deprecated" one that allow key sentence like "retest this please". 
I’m still using it on projects like Karaf just for that ;)
- the "new" plugin just trigger build when the PR is created or updated (push 
forced), but it doesn’t support (yet) key sentence

Regards
JB

> Le 8 juin 2020 à 14:22, Alexey Romanenko  a écrit :
> 
> Hello,
> 
> I think we have an issue, that seems to happen from time to time, to trigger 
> the Jenkins jobs from PRs. 
> 
> For me, usually, I need to type and enter 2 or even more times the same 
> command as a PR comment (like ‘retest this please’ or at least 'Run Java 
> PreCommit’) to see that it finally started. Sometimes it doesn’t start at all 
> .
> 
> Does anyone know what cause this and what we can do to fix/improve it?

Re: How to build Beam locally (with just the bare minimum for Java SDK and runners)?

[Jean-Baptiste Onofre]

Hi,

Personally I’m use:

./gradlew :sdks:java:build …

And ./gradlew projects gives you the name of the module.

Regards
JB

> Le 24 mai 2020 à 11:32, Jacek Laskowski  a écrit :
> 
> Hi Thomas,
> 
> > Most of the developer documentation can be found on the cwiki, in this case
> 
> That's the purpose of the email as I couldn't find the answer. 
> 
> My question is how to build Beam locally with just the bare minimum for Java 
> SDK and the runners? After reading the aforementioned page I ended up with 
> the following:
> 
> ./gradlew -p sdks/java build -x test
> 
> The reason I've been asking about it was that I hoped it would fix the 
> issue(s) with IDEA (thinking that the issue could be with auto-generated 
> files). I'm still struggling with importing the sources in IDEA and have to 
> manually remove 2.20.0-SNAPSHOT directory-based deps and add the sdks/java 
> module.
> 
> Thanks for your help Thomas. Danke schon!
> 
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski 
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski 
> 
> 
>  
> 
> On Sat, May 23, 2020 at 8:17 PM Thomas Weise  > wrote:
> Hi Jacek,
> 
> Most of the developer documentation can be found on the cwiki, in this case
> 
> https://cwiki.apache.org/confluence/display/BEAM/Gradle+Tips 
> 
> 
> To build just a runner, for example, Flink:
> 
> ./gradlew :runners:flink:1.10:job-server:runShadow
> 
> Thomas
> 
> On Sat, May 23, 2020 at 10:29 AM Jacek Laskowski  > wrote:
> Hi,
> 
> I've been wondering how the devs build Beam locally. I'm new to gradle and 
> can't seem to find it in the docs [1].
> 
> I thought the following would be enough, but turns out it executes go-, 
> website- and python-related tasks among others.
> 
> ./gradlew build -x test
> 
> I thought about disabling the tasks I don't want and ended up with the 
> following:
> 
> ./gradlew build -x test -x check -x website -x docker -x 
> :sdks:python:setupVirtualenv
> 
> but that still builds other projects I don't need and so I doubt if -x's is 
> the way to go. There must be a more clever approach. What's that? Please 
> help. Thanks.
> 
> [1] https://beam.apache.org/contribute/ 
> 
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski 
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski 
> 
> 
>  

Re: Companies using Beam?

[Jean-Baptiste Onofre]

Hi,

We already have some testimonials on Beam home page (I did the one about Beam 
use at Talend).

It makes sense to have a dedicated section as it gives ideas about use case and 
production system running with Beam.

Regards
JB

> Le 28 avr. 2020 à 23:42, Austin Bennett  a écrit 
> :
> 
> Hi All,
> 
> Have we considered getting onto our website or our our GitHub repo the 
> ability for individuals to share that their company is using Beam?  Seeing - 
> what I believe to be a reasonable list of - companies productively using Beam 
> would be helpful to point others to.  For instance, a common question I get 
> is whether anyone or who is using?  I'm not sure that's the best metric or 
> datapoint in many cases for adoption, but a heuristic that some rely upon.  
> 
> Naturally, we could ask for a roll-call, esp. via user list, but imagining  a 
> persistent web-list would be of interest.   
> 
> Cheers,
> Austin
> 
> 
> P.S.  If putting such a list into our repo, that would also get some people 
> to submit PRs (so more contributors!) :-)
> 
> 

Re: [VOTE] Release 2.20.0, release candidate #2

[Jean-Baptiste Onofre]

+1 (binding)

Quickly checked on beam-samples.

Regards
JB

> Le 10 avr. 2020 à 17:25, Kenneth Knowles  a écrit :
> 
> +1
> 
> Ran a small Java pipeline.
> 
> Noting that the version in gradle.properties is 2.20.0-RC2: 
> https://github.com/apache/beam/blob/v2.20.0-RC2/gradle.properties#L26 
> . The 
> versions in the built Java artifacts all seem to be the desired value of 
> 2.20.0.
> 
> Kenn
> 
> On Thu, Apr 9, 2020 at 5:19 PM Robert Bradshaw  > wrote:
> +1, the artifacts and signatures all look good, and I also checked that the 
> Python wheels work with a simple pipeline in a fresh virtual environment. 
> 
> On Thu, Apr 9, 2020 at 5:11 PM Ahmet Altay  > wrote:
> +1 - validated python quickstarts batch/streaming with python 2.7. 
> 
> Thank you Rui!
> 
> On Thu, Apr 9, 2020 at 12:28 PM Valentyn Tymofieiev  > wrote:
> +1. Checked mobile gaming batch examples, and a streaming quickstart on 
> Dataflow, on Python 3.7 using Linux wheels.
> 
> On Thu, Apr 9, 2020 at 11:13 AM Rui Wang  > wrote:
> Hi everyone,
> Please review and vote on the release candidate #2 for the version 2.20.0, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org 
>  [2], which is signed with the key with fingerprint 
> 699A 22D2 D4F0 0AD3 957B  6A88 38B1 C6B4 25EB A67C [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.20.0-RC2" [5],
> * website pull request listing the release [6], publishing the API reference 
> manual [7], and the blog post [8].
> * Java artifacts were built with Maven 3.6.2 and OpenJDK 1.8.0_181.
> * Python artifacts are deployed along with the source release to the 
> dist.apache.org  [2].
> * Validation sheet with a tab for 2.20.0 release to help with validation [9].
> * Docker images published to Docker Hub [10].
> 
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Release Manager
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12346780
>  
> 
> [2] https://dist.apache.org/repos/dist/dev/beam/2.20.0/ 
> 
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS 
> 
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1101/ 
> 
> [5] https://github.com/apache/beam/tree/v2.20.0-RC2 
> 
> [6] https://github.com/apache/beam/pull/11285 
> 
> [7] https://github.com/apache/beam-site/pull/602 
> 
> [8] https://github.com/apache/beam/pull/11298 
> 
> [9] 
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=318600984
>  
> 
> [10] https://hub.docker.com/search?q=apache%2Fbeam=image 
> 

Re: Question about updating documents on MongoDB

[Jean-Baptiste Onofre]

Hi,

Not sure I understand fully your use case, but you can add/update document with 
Write:

pipeline
   .apply(...)
   .apply(MongoDbIO.write()
   .withUri("mongodb://localhost:27017")
  .withDatabase("my-database")
.withCollection("my-collection")
.withNumSplits(30))

It’s using the Java SDK. Your question is specifically to Python ?

Regards
JB

> Le 31 mars 2020 à 11:32, Emilien HENON  a écrit :
> 
> Hello,
> 
> I would like to contact you concerning a question about the MongoDBIO 
> connector. As part of our Dataflow / Apache Beam project, we need to create 
> and update documents in our MongoDB database. However, the update operation 
> seems impossible with the current connector (either we read or we create, but 
> it seems impossible to modify/overwrite a document from an already existing 
> ObjectId).
> 
> We would have liked to know if it was really impossible to update or 
> overwrite a document with this library (because according to the 
> documentation, it seems possible on the Python side). If it is, is it a 
> deliberate choice on your side? Or would it be possible to open an associated 
> Jira to possibly contribute to an implementation?
> 
> Regards,
> Emilien Henon

Re: [PROPOSAL] Snowflake Java Connector for Apache Beam

[Jean-Baptiste Onofre]

Hi,

It’s very interesting. +1 to create a Jira and prepare a PR for review.

Thanks !
Regards
JB

> Le 23 mars 2020 à 15:23, Katarzyna Kucharczyk  a 
> écrit :
> 
> Hi all,
> 
> Me and my colleagues have developed a new Java connector for Snowflake that 
> we would like to add to Beam.
> 
> Snowflake is an analytic data warehouse provided as Software-as-a-Service 
> (SaaS). It uses a new SQL database engine with a unique architecture designed 
> for the cloud. To read more details please check [1] and [2].
> 
> Proposed Snowflake IOs use JDBC Snowflake library [3]. The IOs are batch 
> write and batch read that use the Snowflake COPY [4] operation underneath. In 
> both cases ParDo IOs load files on a stage and then they are inserted into 
> the Snowflake table of choice using the COPY API. The currently supported 
> stage is Google Cloud Storage[5].
> 
> The schema how Snowflake Read IO works (write operation works similarly but 
> in opposite direction):
> 
> 
> 
> Here is an Apache Beam fork [6] with current work of the Snowflake IO.
> 
> In the near future we would like to also add IO for writing streams which 
> will use SnowPipe - Snowflake mechanism for continuous loading[7]. Also, we 
> would like to use cross language to provide Python connectors as well.
> 
> We are open for all opinions and suggestions. In case of any 
> questions/comments please do not hesitate to post them.
> 
> In case of no objection I will create jira tickets and share them in this 
> thread.
> 
> Cheers,
> Kasia
> 
> [1] https://www.snowflake.com  
> [2] https://docs.snowflake.net/manuals/user-guide/intro-key-concepts.html 
>  
> [3] https://docs.snowflake.net/manuals/user-guide/jdbc.html 
>  
> [4] https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html 
>  
> [5] 
> https://github.com/PolideaInternal/beam/tree/snowflake-io/sdks/java/io/snowflake
>  
> 
>  
> [6] https://cloud.google.com/storage  
> [7] https://docs.snowflake.net/manuals/user-guide/data-load-snowpipe.html 
>  
> 

Re: A new reworked Elasticsearch 7+ IO module

[Jean-Baptiste Onofre]

Hi,

I think WARN makes sense and the safest approach. It allows users to be notify 
and eventually update or back on previous Beam IO version.

Regards
JB

> Le 6 mars 2020 à 18:49, Kenneth Knowles  a écrit :
> 
> Since the user provides backendVersion, here are some possible levels of 
> things to add in expand() based on that (these are extra niceties beyond the 
> agreed number of releases to remove)
> 
>  - WARN for backendVersion < n
>  - reject for backendVersion < n with opt-in pipeline option to keep it 
> working one more version (gets their attention and indicates urgency)
>  - reject completely
> 
> Kenn
> 
> On Fri, Mar 6, 2020 at 2:26 AM Etienne Chauchot  > wrote:
> Hi all, 
> 
> it's been 3 weeks since the survey on ES versions the users use. 
> 
> The survey received very few responses: only 9 responses for now (multiple 
> versions possible of course). The responses are the following:
> 
> ES2: 0 clients, ES5: 1, ES6: 5, ES7: 8 
> 
> It tends to go toward a drop of ES2 support but for now it is still not very 
> representative.
> 
> I'm cross-posting to @users to let you know that I'm closing the survey 
> within 1 or 2 weeks. So please respond if you're using ESIO.
> 
> Best
> 
> Etienne
> 
> On 13/02/2020 12:37, Etienne Chauchot wrote:
>> Hi Cham, thanks for your comments !
>> 
>> I just sent an email to user ML with a survey link to count ES uses per 
>> version:
>> 
>> https://lists.apache.org/thread.html/rc8185afb8af86a2a032909c13f569e18bd89e75a5839894d5b5d4082%40%3Cuser.beam.apache.org%3E
>>  
>> 
>> Best
>> 
>> Etienne
>> 
>> On 10/02/2020 19:46, Chamikara Jayalath wrote:
>>> 
>>> 
>>> On Thu, Feb 6, 2020 at 8:13 AM Etienne Chauchot >> > wrote:
>>> Hi,
>>> 
>>> please see my comments inline
>>> 
>>> On 06/02/2020 16:24, Alexey Romanenko wrote:
 Please, see my comments inline.
 
> On 6 Feb 2020, at 10:50, Etienne Chauchot  > wrote:
 1. regarding version support: ES v2 is no more maintained by Elastic 
 since 2018/02 so we plan to remove it from the IO. In the past we 
 already retired versions (like spark 1.6 for instance).
 
>> 
>> 
>> My only concern here is that there might be users who use the existing 
>> module who might not be able to easily upgrade the Beam version if we 
>> remove it. But given that V2 is 5 versions behind the latest release 
>> this might be OK.
>> 
>> It seems we have a consensus on this.
>> I think there should be another general discussion on the long term 
>> support of our prefered tool IO modules.
> => yes, consensus, let's drop ESV2
> 
 We had (and still have) a similar problem with KafkaIO to support 
 different versions of Kafka, especially very old version 0.9. We raised 
 this question on user@ and it appears that there are users who for some 
 reasons still use old Kafka versions. So, before dropping a support of any 
 ES versions, I’d suggest to ask it user@ and see if any people will be 
 affected by this.
>>> Yes we can do a survey among users but the question is, should we support 
>>> an ES version that is no more supported by Elastic themselves ?
>>> 
>>> +1 for asking in the user list. I guess this is more about whether users 
>>> need this specific version that we hope to drop support for. Whether we 
>>> need to support unsupported versions is a more generic question that should 
>>> prob. be addressed in the dev list. (and I personally don't think we should 
>>> unless there's a large enough user base for a given version).
>>> 
> 
 2. regarding the user: the aim is to unlock some new features (listed 
 by Ludovic) and give the user more flexibility on his request. For 
 that, it requires to use high level java ES client in place of the low 
 level REST client (that was used because it is the only one compatible 
 with all ES versions). We plan to replace the API (json document in 
 and out) by more complete standard ES objects that contain de request 
 logic (insert/update, doc routing etc...) and the data. There are 
 already IOs like SpannerIO that use similar objects in input 
 PCollection rather than pure POJOs. 
 
>> 
>> 
>> Won't this be a breaking change for all users ? IMO using POJOs in 
>> PCollections is safer since we have to worry about changes to the 
>> underlying client library API. Exception would be when underlying client 
>> library offers a backwards compatibility guarantee that we can rely on 
>> for the foreseeable future (for example, BQ TableRow).
>> 
>> Agreed but actually, there will be POJOs in order to abstract 
>> Elasticsearch's version support. The 

Re: [VOTE] Vendored Dependencies Release gRPC 1.26.0 v0.3 for BEAM-9288 RC #3

[Jean-Baptiste Onofre]

+1 (binding)

Regards
JB

> Le 5 mars 2020 à 19:55, Luke Cwik  a écrit :
> 
> Please review the release of the following artifacts that we vendor:
>  * beam-vendor-grpc-1_26_0
> 
> Hi everyone,
> Please review and vote on the release candidate #1 for the version 0.3, as 
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> The complete staging area is available for your review, which includes:
> * the official Apache source release to be deployed to dist.apache.org 
>  [1], which is signed with the key with fingerprint 
> EAD5DE293F4A03DD2E77565589E68A56E371CCA2 [2],
> * all artifacts to be deployed to the Maven Central Repository [3],
> * commit hash "28d05d317a39b5d60a31988c69d1fb8a9c5006fc" [4],
> 
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Release Manager
> 
> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/ 
> 
> [2] https://dist.apache.org/repos/dist/release/beam/KEYS 
> 
> [3] https://repository.apache.org/content/repositories/orgapachebeam-1098/ 
> 
> [4] 
> https://github.com/apache/beam/commit/28d05d317a39b5d60a31988c69d1fb8a9c5006fc
>  
> 

Re: [VOTE] Upgrade gradle to 6.2

[Jean-Baptiste Onofre]

Fair enough, we have the consensus, so agree to create Jira and move forward 
about this update.

Regards
JB

> Le 5 mars 2020 à 20:16, Ismaël Mejía  a écrit :
> 
> Looks like we have consensus on this one. Can you create a JIRA to
> track this Alex.
> I found this interesting presentation and associated repo, for the
> interested on new improvements we can win with the move to version
> 6.x.x
> https://melix.github.io/gradle-6-whats-new/#/
> https://github.com/melix/gradle-6-whats-new/tree/master/demos/hello-gradle-6
> 
> On Tue, Feb 25, 2020 at 7:24 PM Luke Cwik  wrote:
>> 
>> +1
>> 
>> On Tue, Feb 25, 2020 at 12:49 AM Gleb Kanterov  wrote:
>>> 
>>> +1 (non-binding)
>>> 
>>> On Tue, Feb 25, 2020 at 9:38 AM Ismaël Mejía  wrote:
 
 +1 great to have our build updated, please share if there are new 
 interesting features/plugin advantages we can benefit from too.
 
 On Tue, Feb 25, 2020 at 8:24 AM Jean-Baptiste Onofré  
 wrote:
> 
> Hi Alex
> 
> I also have couple of contacts at Gradle. Let me know if needed.
> 
> Regards
> JB
> 
> Le mar. 25 f?vr. 2020 ? 08:20, Alex Van Boxel  a ?crit :
>> 
>> OK, great. I know someone that works at gradle, so I can ping them when 
>> I have some problems.
>> 
>> Any other know pitfalls I can expect, let me know :-)
>> 
>> _/
>> _/ Alex Van Boxel
>> 
>> 
>> On Tue, Feb 25, 2020 at 7:20 AM Jean-Baptiste Onofr?  
>> wrote:
>> 
>> +1
>> 
>> It makes sense.
>> 
>> Thanks.
>> Regards
>> JB
>> 
>> Le lun. 24 f?vr. 2020 ? 22:37, Alex Van Boxel  a ?crit 
>> :
>> 
>> Anyone objections that I upgrade gradle to 6.2. If ok this will be done 
>> over several commits where I will:
>> 
>> Upgrade plugins
>> Upgrade gradle to 6.2
>> See where we can use some of the new features
>> 
>> 
>> _/
>> _/ Alex Van Boxel

Re: JdbcIO for writing to Dynamic Schemas in Postgres

[Jean-Baptiste Onofre]

Hi

You have the setPrepareStatement() method where you define the target tables.
However, it’s in the same database (datasource) per pipeline.

You can define several datasources and use a different datasource in each 
JdbcIO write. Meaning that you can divide in sub pipelines.

Regards
JB

> Le 29 févr. 2020 à 17:52, Vasu Gupta  a écrit :
> 
> Hey folks,
> 
> Can we use JdbcIO for writing data to multiple Schemas(For Postgres Database) 
> dynamically using Apache beam Java Framework? Currently, I can't find any 
> property that I could set to JdbcIO transform for providing schema or maybe I 
> am missing something.
> 
> Thanks

Re: [VOTE][BIP-1] Beam Schema Options

[Jean-Baptiste Onofre]

+1 (binding)

Very interesting. It remembers me when we start the discussion about schema 
support ;)

Regards
JB

> Le 20 févr. 2020 à 08:36, Alex Van Boxel  a écrit :
> 
> Hi all,
> 
> let's do a vote on the very first Beam Improvement Proposal. If you have a -1 
> or -1 (binding) please add your concern to the open issues section to the 
> wiki. Thanks.
> 
> This is the proposal:
> https://cwiki.apache.org/confluence/display/BEAM/%5BBIP-1%5D+Beam+Schema+Options
>  
> 
> 
> Can I have your votes.
> 
>  _/
> _/ Alex Van Boxel

Re: Compile error on Java 11 when running :examples:java:test

[Jean-Baptiste Onofre]

Hi,

AFAIR I had the same issue on my Linux.

Let me do a new run.

Regards
JB

> Le 7 févr. 2020 à 21:35, Kenneth Knowles  a écrit :
> 
> The expected class file version 53 is for Java 9, I believe. So is the right 
> javac being invoked?
> 
> I hit some issues like this on mac a while back, unrelated to Java 11. 
> Suspected something wonky in Mac's Java setup not working well with the 
> Gradle wrapper. Never resolved them actually. Have been working on linux 
> lately.
> 
> Kenn
> 
> On Fri, Feb 7, 2020 at 11:32 AM Jean-Baptiste Onofré  > wrote:
> Hi
> 
> No jdk 11 is not yet fully supported.
> 
> I?ve started to work on it but it?s not yet ready.
> 
> Regards
> JB
> 
> Le ven. 7 f?vr. 2020 ? 20:20, David Cavazos  > a ?crit :
> Hi Beamers,
> 
> I'm trying to run the tests for the Java examples using Java 11 and there is 
> a compilation error due to an incompatible version.
> 
> I'm using the latest version of master.
> 
> 
> 
> If I downgrade to Java 8, it works. But isn't Java 11 supported?
> 
> Thanks!

Re: Transitive dependency from external repository

[Jean-Baptiste Onofre]

Like this:
repositories {
  jcenter()
  maven { url "https://plugins.gradle.org/m2/; }
  maven {
url "https://repo.spring.io/plugins-release/;
content { includeGroup "io.spring.gradle" }
  }
  maven { url "foo" }
}


> Le 6 févr. 2020 à 18:37, Jean-Baptiste Onofre  a écrit :
> 
> Great, thanks !
> 
> Back on your question, I guess we can add the repository in 
> buildSrc/build.gradle (repositories property).
> 
> Regards
> JB
> 
>> Le 6 févr. 2020 à 18:33, Alexey Romanenko > <mailto:aromanenko@gmail.com>> a écrit :
>> 
>> Yes, it's Apache License 2.0
>> 
>> https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.4.0/kafka-avro-serializer-5.4.0.pom
>>  
>> <https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.4.0/kafka-avro-serializer-5.4.0.pom>
>> 
>>> On 6 Feb 2020, at 18:12, Jean-Baptiste Onofre >> <mailto:j...@nanthrax.net>> wrote:
>>> 
>>> Hi,
>>> 
>>> Just a side note: did you check the license of the dependency (just to be 
>>> sure it’s not a Cat X dependency) ?
>>> 
>>> Regards
>>> JB
>>> 
>>>> Le 6 févr. 2020 à 18:06, Alexey Romanenko >>> <mailto:aromanenko@gmail.com>> a écrit :
>>>> 
>>>> Hi,
>>>> 
>>>> To add support of Confluent Registry Schema in KafkaIO we added new 
>>>> dependency on “io.confluent:kafka-avro-serializer”. The artifacts of this 
>>>> dependency exist in external repository [1]. So, it should not be a 
>>>> problem to add this repository into the list of available repositories of 
>>>> Beam build system - it works fine to build Beam KafkaIO artifacts. 
>>>> 
>>>> The actual problem is with transitive dependency of 
>>>> “io.confluent:kafka-avro-serializer” in user code. We add this dependency 
>>>> into generated and then published KafkaIO's pom.xml but, to successfully 
>>>> resolve it, we need to add a new repository [1] as well (or user should 
>>>> add that manually in its pom which is definitevly not a perfect solution).
>>>> 
>>>> So, my questions to grade/build experts:
>>>> 
>>>> 1) How to add more repositories into published pom.xml with gradle, like 
>>>> we do it in maven?
>>>> 
>>>> For example:
>>>> 
>>>> 
>>>> confluent
>>>> https://packages.confluent.io/maven/ 
>>>> <https://packages.confluent.io/maven/>
>>>> 
>>>> 
>>>> 
>>>> I tried several ways to do that, like adding "repositories { maven { url 
>>>> "https://packages.confluent.io/maven 
>>>> <https://packages.confluent.io/maven>/“ } }” into KafkaIO build.gradle but 
>>>> seems it doesn’t work (I don’t see any additional repositories in 
>>>> published pom file). 
>>>> 
>>>> 2) Another option - would it better to vendor 
>>>> “io.confluent:kafka-avro-serializer” along with KafkaIO and do not add an 
>>>> addition dependency? Wdyt?
>>>> 
>>>> 3) Any other recommendations of better solution for such case?
>>>> 
>>>> Any help on this topic will be very appreciated.
>>>> 
>>>> Alexey
>>>> 
>>>> [1] https://packages.confluent.io/maven/ 
>>>> <https://packages.confluent.io/maven/>
>> 
> 

Re: Transitive dependency from external repository

[Jean-Baptiste Onofre]

Great, thanks !

Back on your question, I guess we can add the repository in 
buildSrc/build.gradle (repositories property).

Regards
JB

> Le 6 févr. 2020 à 18:33, Alexey Romanenko  a écrit :
> 
> Yes, it's Apache License 2.0
> 
> https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.4.0/kafka-avro-serializer-5.4.0.pom
>  
> <https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.4.0/kafka-avro-serializer-5.4.0.pom>
> 
>> On 6 Feb 2020, at 18:12, Jean-Baptiste Onofre > <mailto:j...@nanthrax.net>> wrote:
>> 
>> Hi,
>> 
>> Just a side note: did you check the license of the dependency (just to be 
>> sure it’s not a Cat X dependency) ?
>> 
>> Regards
>> JB
>> 
>>> Le 6 févr. 2020 à 18:06, Alexey Romanenko >> <mailto:aromanenko@gmail.com>> a écrit :
>>> 
>>> Hi,
>>> 
>>> To add support of Confluent Registry Schema in KafkaIO we added new 
>>> dependency on “io.confluent:kafka-avro-serializer”. The artifacts of this 
>>> dependency exist in external repository [1]. So, it should not be a problem 
>>> to add this repository into the list of available repositories of Beam 
>>> build system - it works fine to build Beam KafkaIO artifacts. 
>>> 
>>> The actual problem is with transitive dependency of 
>>> “io.confluent:kafka-avro-serializer” in user code. We add this dependency 
>>> into generated and then published KafkaIO's pom.xml but, to successfully 
>>> resolve it, we need to add a new repository [1] as well (or user should add 
>>> that manually in its pom which is definitevly not a perfect solution).
>>> 
>>> So, my questions to grade/build experts:
>>> 
>>> 1) How to add more repositories into published pom.xml with gradle, like we 
>>> do it in maven?
>>> 
>>> For example:
>>> 
>>> 
>>> confluent
>>> https://packages.confluent.io/maven/ 
>>> <https://packages.confluent.io/maven/>
>>> 
>>> 
>>> 
>>> I tried several ways to do that, like adding "repositories { maven { url 
>>> "https://packages.confluent.io/maven 
>>> <https://packages.confluent.io/maven>/“ } }” into KafkaIO build.gradle but 
>>> seems it doesn’t work (I don’t see any additional repositories in published 
>>> pom file). 
>>> 
>>> 2) Another option - would it better to vendor 
>>> “io.confluent:kafka-avro-serializer” along with KafkaIO and do not add an 
>>> addition dependency? Wdyt?
>>> 
>>> 3) Any other recommendations of better solution for such case?
>>> 
>>> Any help on this topic will be very appreciated.
>>> 
>>> Alexey
>>> 
>>> [1] https://packages.confluent.io/maven/ 
>>> <https://packages.confluent.io/maven/>
> 

Re: Transitive dependency from external repository

[Jean-Baptiste Onofre]

Hi,

Just a side note: did you check the license of the dependency (just to be sure 
it’s not a Cat X dependency) ?

Regards
JB

> Le 6 févr. 2020 à 18:06, Alexey Romanenko  a écrit :
> 
> Hi,
> 
> To add support of Confluent Registry Schema in KafkaIO we added new 
> dependency on “io.confluent:kafka-avro-serializer”. The artifacts of this 
> dependency exist in external repository [1]. So, it should not be a problem 
> to add this repository into the list of available repositories of Beam build 
> system - it works fine to build Beam KafkaIO artifacts. 
> 
> The actual problem is with transitive dependency of 
> “io.confluent:kafka-avro-serializer” in user code. We add this dependency 
> into generated and then published KafkaIO's pom.xml but, to successfully 
> resolve it, we need to add a new repository [1] as well (or user should add 
> that manually in its pom which is definitevly not a perfect solution).
> 
> So, my questions to grade/build experts:
> 
> 1) How to add more repositories into published pom.xml with gradle, like we 
> do it in maven?
> 
> For example:
> 
> 
> confluent
> https://packages.confluent.io/maven/ 
> 
> 
> 
> 
> I tried several ways to do that, like adding "repositories { maven { url 
> "https://packages.confluent.io/maven /“ 
> } }” into KafkaIO build.gradle but seems it doesn’t work (I don’t see any 
> additional repositories in published pom file). 
> 
> 2) Another option - would it better to vendor 
> “io.confluent:kafka-avro-serializer” along with KafkaIO and do not add an 
> addition dependency? Wdyt?
> 
> 3) Any other recommendations of better solution for such case?
> 
> Any help on this topic will be very appreciated.
> 
> Alexey
> 
> [1] https://packages.confluent.io/maven/ 
> 

Re: A new reworked Elasticsearch 7+ IO module

[Jean-Baptiste Onofre]

Hi,

Let’s sync together about this IO.

Regarding mock and IOs, and Etienne’s comment, there are two things:

1. Of course, it’s always preferable to use concrete backend, but several times 
it’s not possible. It’s there mock is required.
2. The mock can be smart enough to cover core IO behavior

So, I still think that mock is interesting as core first level test.

In the case of ES, we can easily bootstrap instance, but I think we can move 
forward without a mock.

Regards
JB

> Le 6 févr. 2020 à 09:47, Ludovic Boutros  a écrit :
> 
> Hi all,
> 
> First, thank you all for your answers and especially, Etienne for your time, 
> advises and kindness :)
> @Jean-Baptiste, any help on this module is welcome of course.
> 
> @Chamikara Jayalath, my aswers are inline.
> 
> Have a good day !
> 
> Ludovic
> 
> Le mer. 5 févr. 2020 à 20:15, Chamikara Jayalath  <mailto:chamik...@google.com>> a écrit :
> 
> 
> On Wed, Feb 5, 2020 at 6:35 AM Etienne Chauchot  <mailto:echauc...@apache.org>> wrote:
> Still there is something I don't agree with is that IOs can be tested on 
> mock. We don't really test IO behavior with mocks: there is always special 
> behaviors that cannot be reproduced in mocks (split, load, with corner cases 
> etc...). There was in the past IOs that were tested using mocks and that 
> happened to be nonfunctional.
> 
> Regarding ITests we have very few comparing to UTests and they are not as 
> closely observed as UTests.
> 
> Etienne
> 
> On 05/02/2020 11:32, Jean-Baptiste Onofre wrote:
>> Hi,
>> 
>> We talked in the past about multiple/single module.
>> 
>> IMHO the always preferred goal is to have a single module. However, it’s 
>> tricky when we have such difference, including on the user facing API. So, I 
>> would go with module per version, or use a specified version for a target 
>> Beam release.
>> 
>> For the test, we should distinguish utest from itest. Utest can be done with 
>> mock, the purpose is really to test the IO behavior. Then we can have itest 
>> using concrete ES instance.
>> 
>> Anyway, I’m OK with the proposal and I would like to work on this IO (I have 
>> other improvements coming on other IOs anyway) with you guys (and Ludovic 
>> especially).
>> 
>> Regards
>> JB
>> 
>>> Le 5 févr. 2020 à 10:44, Etienne Chauchot >> <mailto:echauc...@apache.org>> a écrit :
>>> 
>>> Hi all, 
>>> 
>>> We had a long discussion with Ludovic about this IO. I'd like to share with 
>>> you to keep you informed and also gather your opinions
>>> 
>>> 1. regarding version support: ES v2 is no more maintained by Elastic since 
>>> 2018/02 so we plan to remove it from the IO. In the past we already retired 
>>> versions (like spark 1.6 for instance).
>>> 
> 
> 
> My only concern here is that there might be users who use the existing module 
> who might not be able to easily upgrade the Beam version if we remove it. But 
> given that V2 is 5 versions behind the latest release this might be OK.
> 
> It seems we have a consensus on this.
> I think there should be another general discussion on the long term support 
> of our prefered tool IO modules.
>  
>  
>>> 
>>> 2. regarding the user: the aim is to unlock some new features (listed by 
>>> Ludovic) and give the user more flexibility on his request. For that, it 
>>> requires to use high level java ES client in place of the low level REST 
>>> client (that was used because it is the only one compatible with all ES 
>>> versions). We plan to replace the API (json document in and out) by more 
>>> complete standard ES objects that contain de request logic (insert/update, 
>>> doc routing etc...) and the data. There are already IOs like SpannerIO that 
>>> use similar objects in input PCollection rather than pure POJOs. 
>>> 
> 
> 
> Won't this be a breaking change for all users ? IMO using POJOs in 
> PCollections is safer since we have to worry about changes to the underlying 
> client library API. Exception would be when underlying client library offers 
> a backwards compatibility guarantee that we can rely on for the foreseeable 
> future (for example, BQ TableRow).
> 
> Agreed but actually, there will be POJOs in order to abstract Elasticsearch's 
> version support. The following third point explains this.
>  
>  
>>> 
>>> 3. regarding multiple/single module: the aim is to have only one production 
>>> code to ease the maintenance.  The problem is that using high level client 
>>> makes the code dependent to an ES lib ve

Re: A new reworked Elasticsearch 7+ IO module

[Jean-Baptiste Onofre]

Hi,

We talked in the past about multiple/single module.

IMHO the always preferred goal is to have a single module. However, it’s tricky 
when we have such difference, including on the user facing API. So, I would go 
with module per version, or use a specified version for a target Beam release.

For the test, we should distinguish utest from itest. Utest can be done with 
mock, the purpose is really to test the IO behavior. Then we can have itest 
using concrete ES instance.

Anyway, I’m OK with the proposal and I would like to work on this IO (I have 
other improvements coming on other IOs anyway) with you guys (and Ludovic 
especially).

Regards
JB

> Le 5 févr. 2020 à 10:44, Etienne Chauchot  a écrit :
> 
> Hi all, 
> 
> We had a long discussion with Ludovic about this IO. I'd like to share with 
> you to keep you informed and also gather your opinions
> 
> 1. regarding version support: ES v2 is no more maintained by Elastic since 
> 2018/02 so we plan to remove it from the IO. In the past we already retired 
> versions (like spark 1.6 for instance).
> 
> 2. regarding the user: the aim is to unlock some new features (listed by 
> Ludovic) and give the user more flexibility on his request. For that, it 
> requires to use high level java ES client in place of the low level REST 
> client (that was used because it is the only one compatible with all ES 
> versions). We plan to replace the API (json document in and out) by more 
> complete standard ES objects that contain de request logic (insert/update, 
> doc routing etc...) and the data. There are already IOs like SpannerIO that 
> use similar objects in input PCollection rather than pure POJOs. 
> 
> 3. regarding multiple/single module: the aim is to have only one production 
> code to ease the maintenance.  The problem is that using high level client 
> makes the code dependent to an ES lib version. We would like to make it 
> invisible to the user. He should select only one jar and the IO should decide 
> the lib to use behind the scene. We are thinking about using one module and 
> sub-modules per version and use relocation, wrappers and a factory that 
> detects the version the IO actually points to to instantiate the correct 
> client version. It would also require to have DTOs in the IO because the high 
> level ES java objects are not exactly the same among the ES versions.
> 
> 4. regarding tests: the aim is always to target real ES backends to have 
> relevant tests (for reasons I already explained in another thread). The 
> problem is that es-test-framework used today is version dependent and is a 
> pain to use. We plan on using test containers per version (validated by ES 
> dev advocate) and launching them as part of the UTests. Obviously we will 
> launch only one container at the time per version and do all the test with it 
> to avoid paying the cost of launch too much. And the tests will be shipped in 
> per-version sub-modules and not in test dedicated modules like it is now.
> 
> WDYT ?
> 
> Best !
> 
> Etienne
> 
> On 30/01/2020 17:55, Alexey Romanenko wrote:
>> I’m second for this question. We have a similar (maybe a bit less painful) 
>> issue for KafkaIO and it would be useful to have a general strategy for such 
>> cases about how to deal with that.
>> 
>>> On 24 Jan 2020, at 21:54, Kenneth Knowles >> > wrote:
>>> 
>>> Would it make sense to have different version-specialized connectors with a 
>>> common core library and common API package?
>>> 
>>> On Fri, Jan 24, 2020 at 11:52 AM Chamikara Jayalath >> > wrote:
>>> Thanks for the contribution. I agree with Alexey that we should try to add 
>>> any new features brought in with the new PR into existing connector instead 
>>> of trying to maintain two implementations.  
>>> 
>>> Thanks,
>>> Cham
>>> 
>>> On Fri, Jan 24, 2020 at 9:01 AM Alexey Romanenko >> > wrote:
>>> Hi Ludovic,
>>> 
>>> Thank you for working on this and sharing the details with us. This is 
>>> really great job!
>>> 
>>> As I recall, we already have some support of Elasticsearch7 in current 
>>> ElasticsearchIO (afaik, at least they are compatible), thanks to Zhong Chen 
>>> and Etienne Chauchot, who were working on adding this [1][2] and it should 
>>> be released in Beam 2.19.
>>> 
>>> Would you think you can leverage this in your work on adding new 
>>> Elasticsearch7 features? IMHO, supporting two different related IOs can be 
>>> quite tough task and I‘d rather raise my hand to add a new functionality 
>>> into existing IO than creating a new one, if it’s possible.
>>> 
>>> [1] https://issues.apache.org/jira/browse/BEAM-5192 
>>> 
>>> [2] https://github.com/apache/beam/pull/10433 
>>> 
>>> 
 On 22 Jan 2020, at 19:23, Ludovic Boutros >>> > wrote:
 
 Dear all,
 
 I have written a