Small DNS update for Beam Playground

[Pablo Estrada via dev]

Hi all,
I would like to update everyone on a small change for the Beam Playground.
We've worked to move it from App Engine to Kubernetes. This means that we
have to 'flip the switch', and change the domain name to point to the k8s
endpoint.

This does not require any action from you, but rather it's an update to the
community about a change to our infra. Feel free to ask questions if there
are any.
Best
-P.

Re: Incorrect python bigquery storage read API docs

[Pablo Estrada via dev]

("This is correct" meaning that you're correct Parth)

On Mon, Feb 6, 2023 at 12:39 PM Pablo Estrada  wrote:

> This is correct! The Bigquery Storage Read API is supported in Python. We
> need to fix that doc.
> -P.
>
> On Mon, Feb 6, 2023 at 11:58 AM Ahmet Altay  wrote:
>
>> Adding @Pablo Estrada  who reviewed that issue,
>> hopefully he could help.
>>
>> On Sun, Feb 5, 2023 at 4:37 PM Parth Doshi 
>> wrote:
>>
>>> Hi,
>>>
>>> I was reading through the documentation for the python bigquery read API
>>> 
>>>  and
>>> it says
>>> """
>>> The Beam SDK for Python does not support the BigQuery Storage API. See
>>> Issue 20687).
>>> """
>>>
>>> However, that issue has since been closed
>>>  and it seems it is now
>>> supported.
>>>
>>> Best,
>>> Parth
>>>
>>

Re: Incorrect python bigquery storage read API docs

[Pablo Estrada via dev]

This is correct! The Bigquery Storage Read API is supported in Python. We
need to fix that doc.
-P.

On Mon, Feb 6, 2023 at 11:58 AM Ahmet Altay  wrote:

> Adding @Pablo Estrada  who reviewed that issue,
> hopefully he could help.
>
> On Sun, Feb 5, 2023 at 4:37 PM Parth Doshi  wrote:
>
>> Hi,
>>
>> I was reading through the documentation for the python bigquery read API
>> 
>>  and
>> it says
>> """
>> The Beam SDK for Python does not support the BigQuery Storage API. See
>> Issue 20687).
>> """
>>
>> However, that issue has since been closed
>>  and it seems it is now
>> supported.
>>
>> Best,
>> Parth
>>
>

Re: [fyi][discuss] Making jackson-dataformat-yaml a provided/optional dependency

[Pablo Estrada via dev]

It's worth mentioning that neither of the libraries
(jackson-dataformat-yaml + snakeyaml) have a newer version without the CVE.
-P.

On Mon, Feb 6, 2023 at 9:19 AM Pablo Estrada  wrote:

> Hi all,
> I am proposing that we make the jackson-dataformat-yaml dependency
> optional in our expansion service module[1]. This is because it depends on
> SnakeYAML, and there is a known CVE for it[2].
>
> It seems that given the way we use SnakeYAML, the CVE is not feasible to
> exploit[2], but this will not stop tooling/user policies from being
> alerted, so it may be convenient to simply make the dependency optional.
>
> I looked around for documentation on this code path (loading an allow list
> for the expansion service's classpath), but it's not very widely
> documented, so this feature may only be used by Beam devs, and not much by
> Beam users.
>
> Thoughts on making the dependency optional?
> Thanks!
> -P.
>
> [1] https://github.com/apache/beam/pull/25350
> [2] https://github.com/snakeyaml/snakeyaml#cve
>

[fyi][discuss] Making jackson-dataformat-yaml a provided/optional dependency

[Pablo Estrada via dev]

Hi all,
I am proposing that we make the jackson-dataformat-yaml dependency optional
in our expansion service module[1]. This is because it depends on
SnakeYAML, and there is a known CVE for it[2].

It seems that given the way we use SnakeYAML, the CVE is not feasible to
exploit[2], but this will not stop tooling/user policies from being
alerted, so it may be convenient to simply make the dependency optional.

I looked around for documentation on this code path (loading an allow list
for the expansion service's classpath), but it's not very widely
documented, so this feature may only be used by Beam devs, and not much by
Beam users.

Thoughts on making the dependency optional?
Thanks!
-P.

[1] https://github.com/apache/beam/pull/25350
[2] https://github.com/snakeyaml/snakeyaml#cve

[fyi] Updating ip address for Playground staging

[Pablo Estrada via dev]

Hi all,
this email is just to inform contributors that due to resource constraints
in the apache-beam-testing project, we want to update the IP address for
the staging environment of the Beam playground from a us-west one to a
us-east one.

If you have any concerns about this, please reach out to me.
Best
-P.

Re: Access to Confluence

[Pablo Estrada via dev]

+Kenneth Knowles 
I believe Beam is still built based on JDK 8. I suppose our development
manuals could be moved to Java 11 as long as Beam continues to build with
JDK 8 in our CI/CD.
I don't know enough about Java to know what's the best way of dealing with
this : )
-P.

On Mon, Jan 23, 2023 at 4:57 AM AIT-ZEOUAY Amrane <
amrane.ait-zeo...@renault.com> wrote:

> Hello Pablo,
>
>
>
> I hope you are doing well, and you had a great weekend.
>
>
>
> I just noticed that in the project Apache beam, the class Module is being
> used which was introduced since Java 9, but in the Java documentation
> ,
> we are recommending to use JDK 8, wouldn’t be better to modify the
> documentation to change JDK 8 to 11 ?
>
>
>
> Have a nice day and a great beginning of the week.
>
>
>
> Best Regards,
>
>
>
> [image: signature_3530061176]
>
> [image: signature_2506645423]
>
> Amrane AIT ZEOUAY
>
> Data Engineer
>
> + 33 6 03 05 52 78 <+33603055278>
>
> amrane.ait-zeo...@renault.com
>
>
>
>
>
>
>
> *De : *Pablo Estrada 
> *Date : *jeudi, 12 janvier 2023 à 18:59
> *À : *AIT-ZEOUAY Amrane 
> *Cc : *dev@beam.apache.org 
> *Objet : *Re: Access to Confluence
>
> [image: vwP6KQExYeP8ewASUVORK5CYII=]
>
> [EXT]
>
> Okay I've granted you edit privileges! Thanks again for offering to make
> improvements Ait Zeouay!
>
> Best
>
> -P.
>
>
>
>
>
> On Thu, Jan 12, 2023 at 7:59 AM AIT-ZEOUAY Amrane <
> amrane.ait-zeo...@renault.com> wrote:
>
> Hello Pablo,
>
>
>
> Thank you for your reply. I tried to edit it by just signing up, but it
> didn’t work. Here is the email I used for signing up
> a.zeouayam...@gmail.com and the username is
>
> Amraneze.
>
>
>
> PS: You almost got it, it’s Ait Zeouay 😃
>
>
>
> Best Regards,
>
>
>
> [image: signature_777096298]
>
> [image: signature_3125182988]
>
> Amrane AIT ZEOUAY
>
> Data Engineer
>
> + 33 6 03 05 52 78 <+33603055278>
>
> amrane.ait-zeo...@renault.com
>
>
>
>
>
> *De : *Pablo Estrada 
> *Date : *mercredi, 11 janvier 2023 à 19:36
> *À : *dev@beam.apache.org , AIT-ZEOUAY Amrane <
> amrane.ait-zeo...@renault.com>
> *Objet : *Re: Access to Confluence
>
> [image: vwP6KQExYeP8ewASUVORK5CYII=]
>
> [EXT]
>
> Hello Ait-Zelouai! (did I get that right?:))
>
>
>
> Thanks for the help. You can sign up for an account here:
> https://cwiki.apache.org/confluence/signup.action
>
>
>
> After signing up, you can share you username with me, and I'll grant you
> edit privileges.
>
> Best
>
> -P.
>
>
>
> On Wed, Jan 11, 2023 at 10:11 AM AIT-ZEOUAY Amrane via dev <
> dev@beam.apache.org> wrote:
>
> Hello,
>
>
>
> I tried cloning the project locally, it couldn’t be built. After sometime,
> I got a hang of it and I wanted to update the document
> https://cwiki.apache.org/confluence/display/BEAM/Using+IntelliJ+IDE to
> help others who want to contribute to apache beam. If it’s not possible, I
> can send you a Markdown document which will contains the missing part of
> the page.
>
>
>
> Best Regards,
>
>
>
> [image: signature_948923254]
>
> [image: signature_2646937704]
>
> Amrane AIT ZEOUAY
>
> Data Engineer
>
> + 33 6 03 05 52 78 <+33603055278>
>
> amrane.ait-zeo...@renault.com
>
>
>
>
>
>
>
> Confidential C
>
> -- Disclaimer 
> Ce message ainsi que les eventuelles pieces jointes constituent une
> correspondance privee et confidentielle a l'attention exclusive du
> destinataire designe ci-dessus. Si vous n'etes pas le destinataire du
> present message ou une personne susceptible de pouvoir le lui delivrer, il
> vous est signifie que toute divulgation, distribution ou copie de cette
> transmission est strictement interdite. Si vous avez recu ce message par
> erreur, nous vous remercions d'en informer l'expediteur par telephone ou de
> lui retourner le present message, puis d'effacer immediatement ce message
> de votre systeme.
>
> *** This e-mail and any attachments is a confidential correspondence
> intended only for use of the individual or entity named above. If you are
> not the intended recipient or the agent responsible for delivering the
> message to the intended recipient, you are hereby notified that any
> disclosure, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please notify
> the sender by phone or by replying this message, and then delete this
> message from your system.
>
>
>
>
>
> Confidential C
>
> -- Disclaimer 
> Ce message ainsi que les eventuelles pieces jointes constituent une
> correspondance privee et confidentielle a l'attention exclusive du
> destinataire designe ci-dessus. Si vous n'etes pas le destinataire du
> present message ou une personne susceptible de pouvoir le lui delivrer, il
> vous est signifie que toute divulgation, distribution ou copie de cette
> transmission est strictement interdite. Si vous avez recu ce message par
> erreur, nous vous remercions d'en i

Re: Access to Confluence

[Pablo Estrada via dev]

Okay I've granted you edit privileges! Thanks again for offering to make
improvements Ait Zeouay!
Best
-P.


On Thu, Jan 12, 2023 at 7:59 AM AIT-ZEOUAY Amrane <
amrane.ait-zeo...@renault.com> wrote:

> Hello Pablo,
>
>
>
> Thank you for your reply. I tried to edit it by just signing up, but it
> didn’t work. Here is the email I used for signing up
> a.zeouayam...@gmail.com and the username is
>
> Amraneze.
>
>
>
> PS: You almost got it, it’s Ait Zeouay 😃
>
>
>
> Best Regards,
>
>
>
> [image: signature_777096298]
>
> [image: signature_3125182988]
>
> Amrane AIT ZEOUAY
>
> Data Engineer
>
> + 33 6 03 05 52 78 <+33603055278>
>
> amrane.ait-zeo...@renault.com
>
>
>
>
>
> *De : *Pablo Estrada 
> *Date : *mercredi, 11 janvier 2023 à 19:36
> *À : *dev@beam.apache.org , AIT-ZEOUAY Amrane <
> amrane.ait-zeo...@renault.com>
> *Objet : *Re: Access to Confluence
>
> [image: vwP6KQExYeP8ewASUVORK5CYII=]
>
> [EXT]
>
> Hello Ait-Zelouai! (did I get that right?:))
>
>
>
> Thanks for the help. You can sign up for an account here:
> https://cwiki.apache.org/confluence/signup.action
>
>
>
> After signing up, you can share you username with me, and I'll grant you
> edit privileges.
>
> Best
>
> -P.
>
>
>
> On Wed, Jan 11, 2023 at 10:11 AM AIT-ZEOUAY Amrane via dev <
> dev@beam.apache.org> wrote:
>
> Hello,
>
>
>
> I tried cloning the project locally, it couldn’t be built. After sometime,
> I got a hang of it and I wanted to update the document
> https://cwiki.apache.org/confluence/display/BEAM/Using+IntelliJ+IDE to
> help others who want to contribute to apache beam. If it’s not possible, I
> can send you a Markdown document which will contains the missing part of
> the page.
>
>
>
> Best Regards,
>
>
>
> [image: signature_948923254]
>
> [image: signature_2646937704]
>
> Amrane AIT ZEOUAY
>
> Data Engineer
>
> + 33 6 03 05 52 78 <+33603055278>
>
> amrane.ait-zeo...@renault.com
>
>
>
>
>
>
>
> Confidential C
>
> -- Disclaimer 
> Ce message ainsi que les eventuelles pieces jointes constituent une
> correspondance privee et confidentielle a l'attention exclusive du
> destinataire designe ci-dessus. Si vous n'etes pas le destinataire du
> present message ou une personne susceptible de pouvoir le lui delivrer, il
> vous est signifie que toute divulgation, distribution ou copie de cette
> transmission est strictement interdite. Si vous avez recu ce message par
> erreur, nous vous remercions d'en informer l'expediteur par telephone ou de
> lui retourner le present message, puis d'effacer immediatement ce message
> de votre systeme.
>
> *** This e-mail and any attachments is a confidential correspondence
> intended only for use of the individual or entity named above. If you are
> not the intended recipient or the agent responsible for delivering the
> message to the intended recipient, you are hereby notified that any
> disclosure, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please notify
> the sender by phone or by replying this message, and then delete this
> message from your system.
>
>
>
> Confidential C
>
> -- Disclaimer 
> Ce message ainsi que les eventuelles pieces jointes constituent une
> correspondance privee et confidentielle a l'attention exclusive du
> destinataire designe ci-dessus. Si vous n'etes pas le destinataire du
> present message ou une personne susceptible de pouvoir le lui delivrer, il
> vous est signifie que toute divulgation, distribution ou copie de cette
> transmission est strictement interdite. Si vous avez recu ce message par
> erreur, nous vous remercions d'en informer l'expediteur par telephone ou de
> lui retourner le present message, puis d'effacer immediatement ce message
> de votre systeme.
>
> *** This e-mail and any attachments is a confidential correspondence
> intended only for use of the individual or entity named above. If you are
> not the intended recipient or the agent responsible for delivering the
> message to the intended recipient, you are hereby notified that any
> disclosure, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please notify
> the sender by phone or by replying this message, and then delete this
> message from your system.
>

Re: Access to Confluence

[Pablo Estrada via dev]

Hello Ait-Zelouai! (did I get that right?:))

Thanks for the help. You can sign up for an account here:
https://cwiki.apache.org/confluence/signup.action

After signing up, you can share you username with me, and I'll grant you
edit privileges.
Best
-P.

On Wed, Jan 11, 2023 at 10:11 AM AIT-ZEOUAY Amrane via dev <
dev@beam.apache.org> wrote:

> Hello,
>
>
>
> I tried cloning the project locally, it couldn’t be built. After sometime,
> I got a hang of it and I wanted to update the document
> https://cwiki.apache.org/confluence/display/BEAM/Using+IntelliJ+IDE to
> help others who want to contribute to apache beam. If it’s not possible, I
> can send you a Markdown document which will contains the missing part of
> the page.
>
>
>
> Best Regards,
>
>
>
> [image: signature_948923254]
>
> [image: signature_2646937704]
>
> Amrane AIT ZEOUAY
>
> Data Engineer
>
> + 33 6 03 05 52 78 <+33603055278>
>
> amrane.ait-zeo...@renault.com
>
>
>
>
>
> Confidential C
>
> -- Disclaimer 
> Ce message ainsi que les eventuelles pieces jointes constituent une
> correspondance privee et confidentielle a l'attention exclusive du
> destinataire designe ci-dessus. Si vous n'etes pas le destinataire du
> present message ou une personne susceptible de pouvoir le lui delivrer, il
> vous est signifie que toute divulgation, distribution ou copie de cette
> transmission est strictement interdite. Si vous avez recu ce message par
> erreur, nous vous remercions d'en informer l'expediteur par telephone ou de
> lui retourner le present message, puis d'effacer immediatement ce message
> de votre systeme.
>
> *** This e-mail and any attachments is a confidential correspondence
> intended only for use of the individual or entity named above. If you are
> not the intended recipient or the agent responsible for delivering the
> message to the intended recipient, you are hereby notified that any
> disclosure, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please notify
> the sender by phone or by replying this message, and then delete this
> message from your system.
>

Re: Gradle

[Pablo Estrada via dev]

The build scan configuration is here:
https://github.com/apache/beam/blob/master/settings.gradle.kts#L25-L44 - we
already upload for all CI builds.

IIUC, all we would need to do is change the server to the node provided by
Infra (
https://docs.gradle.com/enterprise/gradle-plugin/#connecting_to_gradle_enterprise
)

+gmcdon...@apache.org  Gavin, I believe this means
that step 1 is done, and we can move on to step 2?

Thanks!
-P.

On Tue, Jan 3, 2023 at 11:00 AM Kenneth Knowles  wrote:

> Very cool! Thank you!
>
> On Fri, Dec 23, 2022 at 1:33 AM Gavin McDonald 
> wrote:
>
>> Hi All,
>>
>> The ASF Infrastructure Team along with Gradle Inc have created an
>> instance of Gradle Enterprise at ge.apache.org. It is ready to be used,
>> but so far we have not used it other than for testing.
>>
>> We would like for Beam to be among the first to try out the new service.
>>
>> Step 1 would be for us to enable 'build scans' - we can do this in one of
>> two ways, enable it on the ci-beam.apache.org controller in such a way
>> that all builds on all agents attached to that controller will produce
>> build scans and send the results off to ge.apache.org. Or, we can show
>> you how to enable them on a per build basis - let me know what you think
>> and which you prefer.
>>
>
> Currently our builds all do send build scans using their free tier. I do
> not actually see where this is configured in our jobs, so perhaps it is
> default in the jenkins job DSL gradle plugin, or already configured on the
> master? We are also moving more and more builds to GitHub Actions, FWIW.
>
> Kenn
>
>
>> Step 2 - which we can do now or later, let me know please, is to provide
>> you with a Gradle build cache node, connected to ge.apache.org - you
>> have already requested a build cache node via infra ticket
>> https://issues.apache.org/jira/browse/INFRA-20516 - so we can get this
>> done for you.
>>
>> Beam committers can also connect their local builds to ge.apache.org to
>> upload scan results to ge.apache.org.
>>
>> All committers can already login to ge.apache.org using their LDAP
>> credentials.
>>
>> Thanks all, let me know your thoughts.
>>
>> --
>>
>> *Gavin McDonald*
>> Systems Administrator
>> ASF Infrastructure Team
>>
>

Re: [VOTE] Release 2.43.0, release candidate #2

[Pablo Estrada via dev]

+1 (binding)
Tested local tests for existing DF templates.

On Tue, Nov 15, 2022 at 8:17 AM Alexey Romanenko 
wrote:

> +1 (binding)
>
> —
> Alexey
>
> On 15 Nov 2022, at 14:37, Ritesh Ghorse via dev 
> wrote:
>
> +1 (non-binding)
>
> Validated Go SDK quickstart on Direct and Dataflow runner. Also validated
> Dataframe wrapper on Portable and Dataflow runner.
>
>
> On Tue, Nov 15, 2022 at 5:17 AM Anand Inguva via dev 
> wrote:
>
>> +1(non-binding)
>>
>> Validated Python wordcount example on Direct and Dataflow runner. Staging
>> of the Python dependencies works as expected now.
>>
>> Thanks,
>> Anand
>>
>> On Sun, Nov 13, 2022 at 9:52 AM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hi everyone,
>>> Please review and vote on the release candidate #2 for the version
>>> 2.43.0, as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> Reviewers are encouraged to test their own use cases with the release
>>> candidate, and vote +1 if
>>> no issues are found.
>>>
>>> The complete staging area is available for your review, which includes:
>>> * GitHub Release notes [1],
>>> * the official Apache source release to be deployed to dist.apache.org
>>> [2], which is signed with the key with fingerprint
>>> 40C61FBE1761E5DB652A1A780CCD5EB2A718A56E [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "v2.43.0-RC2" [5],
>>> * website pull request listing the release [6], the blog post [6], and
>>> publishing the API reference manual [7].
>>> * Java artifacts were built with Gradle 7.5.1 and openjdk version
>>> 1.8.0_181-google-v7.
>>> * Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2] and PyPI[8].
>>> * Go artifacts and documentation are available at pkg.go.dev [9]
>>> * Validation sheet with a tab for 2.43.0 release to help with validation
>>> [10].
>>> * Docker images published to Docker Hub [11].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> For guidelines on how to try the release in your projects, check out our
>>> blog post at https://beam.apache.org/blog/validate-beam-release/.
>>>
>>> Thanks,
>>> Cham
>>>
>>> [1] https://github.com/apache/beam/milestone/5
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.43.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1288/
>>> [5] https://github.com/apache/beam/tree/v2.43.0-RC2
>>> [6] https://github.com/apache/beam/pull/24044
>>> [7] https://github.com/apache/beam-site/pull/636
>>> [8] https://pypi.org/project/apache-beam/2.43.0rc2/
>>> [9]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.43.0-RC2/go/pkg/beam
>>> [10]
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1310009119
>>> [11] https://hub.docker.com/search?q=apache%2Fbeam&type=image
>>>
>>
>

Re: bhulette stepping back (for now)

[Pablo Estrada via dev]

I promised I wouldn't cry so I won't. Cya!

On Fri, Nov 11, 2022 at 10:46 AM Robin Qiu via dev 
wrote:

> Thanks for your contribution Brian! Hope you enjoy your new team!
>
> Best,
> Robin
>
> On Fri, Nov 11, 2022 at 10:27 AM Kenneth Knowles  wrote:
>
>> Your contributions have been huge. You will be missed! But have a
>> fabulous time with BigQuery. And thank you so much for letting us know [1]
>>
>> Kenn
>>
>> [1] See "stepping down considerately" from
>> https://www.apache.org/foundation/policies/conduct.html
>>
>> On Thu, Nov 10, 2022 at 4:00 PM Brian Hulette 
>> wrote:
>>
>>> Hi dev@beam,
>>>
>>> I just wanted to let the community know that I will be stepping back
>>> from Beam development for now. I'm switching to a different team within
>>> Google next week - I will be working on BigQuery.
>>>
>>> I'm removing myself from automated code review assignments [1], and
>>> won't actively monitor the beam lists anymore. That being said, I'm happy
>>> to contribute to discussions or code reviews when it would be particularly
>>> helpful, e.g. for anything relating to DataFrames/Schemas/SQL. I can always
>>> be reached at bhule...@apache.org, and @TheNeuralBit [2] on GitHub.
>>>
>>> Brian
>>>
>>> [1] https://github.com/apache/beam/pull/24108
>>> [2] https://github.com/TheNeuralBit
>>>
>>

Re: [ANNOUNCE] New committer: Yi Hu

[Pablo Estrada via dev]

+1 thanks Yi : D

On Wed, Nov 9, 2022 at 10:47 AM Danny McCormick via dev 
wrote:

> Congrats Yi! I've really appreciated the ways you've consistently taken
> responsibility for improving our team's infra and working through sharp
> edges in the codebase that others have ignored. This is definitely well
> deserved!
>
> Thanks,
> Danny
>
> On Wed, Nov 9, 2022 at 1:37 PM Anand Inguva via dev 
> wrote:
>
>> Congratulations Yi!
>>
>> On Wed, Nov 9, 2022 at 1:35 PM Ritesh Ghorse via dev 
>> wrote:
>>
>>> Congratulations Yi!
>>>
>>> On Wed, Nov 9, 2022 at 1:34 PM Ahmed Abualsaud via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congrats Yi!

 On Wed, Nov 9, 2022 at 1:33 PM Sachin Agarwal via dev <
 dev@beam.apache.org> wrote:

> Congratulations Yi!
>
> On Wed, Nov 9, 2022 at 10:32 AM Kenneth Knowles 
> wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Yi Hu (y...@apache.org)
>>
>> Yi started contributing to Beam in early 2022. Yi's contributions are
>> very diverse! I/Os, performance tests, Jenkins, support for Schema 
>> logical
>> types. Not only code but a very large amount of code review. Yi is also
>> noted for picking up smaller issues that normally would be left on the
>> backburner and filing issues that he finds rather than ignoring them.
>>
>> Considering their contributions to the project over this timeframe,
>> the Beam PMC trusts Yi with the responsibilities of a Beam committer. [1]
>>
>> Thank you Yi! And we are looking to see more of your contributions!
>>
>> Kenn, on behalf of the Apache Beam PMC
>>
>> [1]
>>
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>

Re: github reviewer help / tips

[Pablo Estrada via dev]

Nice pointer Valentyn.

If you are willing to open the Github page sometimes, it's worth just using
the PR tab: https://github.com/pulls - which should show all PRs that
mention you/have review requests for you from all repos.
-P.

On Tue, Nov 8, 2022 at 10:56 AM Valentyn Tymofieiev via dev <
dev@beam.apache.org> wrote:

> I use Notifier for Github
> 
>  Chrome
> extension.
>
> On Tue, Nov 8, 2022 at 10:29 AM Sachin Agarwal via dev <
> dev@beam.apache.org> wrote:
>
>> Hey folks,
>>
>> I've found myself repeatedly being very untimely in providing reviews on
>> PRs where I've been added as a reviewer.  (Mea culpa and thank you for your
>> understanding to those who have tagged me and emailed me to nudge me along.)
>>
>> Does anyone have any great tips about how to be super on top of things in
>> the Beam repos?  Any Github experts who can get my SLA from three weeks to
>> a day or so would be great.
>>
>> Many thanks in advance -
>>
>> Cheers,
>> Sachin
>>
>

On wrestling with beam/issues/21104

[Pablo Estrada via dev]

Hi all,
I've spent a few days trying to narrow down whatever is going on with
https://github.com/apache/beam/issues/21104.

The bug reproduces frequently on Jenkins and very rarely on my laptop
(sometimes ~20 min, sometimes hours, sometimes I give up after it doesn't
reproduce all night).

Seeing as how it reproduces so frequently in our Jenkins env (see:
https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/), I would like
to do the following:

1. Add debug information to log as much context as possible when it occurs
2. Create a bug blocking the next release to revert the change
3. Revert before the next release

And hopefully this experiment will let us gather more information about the
bug.

Is everyone in favor of this? Is anyone against? Comments?
Thanks!
-P.

Re: [VOTE] Release 2.42.0, release candidate #2

[Pablo Estrada via dev]

+1 (binding)

I've validated local/unit tests for existing dataflow templates. They look
good!
Best
-P.

On Thu, Oct 13, 2022 at 10:41 AM Ning Kang via dev 
wrote:

> +1 Thank you, Robert!
>
> On Thu, Oct 13, 2022 at 12:47 AM Robert Burke  wrote:
>
>> Hi everyone,
>> Please review and vote on the release candidate #2 for the version
>> 2.42.0, as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>> Reviewers are encouraged to test their own use cases with the release
>> candidate, and vote +1 if no issues are found.
>>
>> The complete staging area is available for your review, which includes:
>> * GitHub Release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org [2],
>> which is signed with the key with fingerprint
>> A52F5C83BAE26160120EC25F3D56ACFBFB2975E1 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.42.0-RC2" [5],
>> * website pull request listing the release [6], the blog post [6], and
>> publishing the API reference manual [7].
>> * Java artifacts were built with Gradle 7.5.1 and AdoptOpen JDK 1.8.0_292.
>> * Python artifacts are deployed along with the source release to the
>> dist.apache.org [2] and PyPI [8]
>> * Go Package information and SDK RC [9]
>> * Validation sheet with a tab for 2.42.0 release to help with validation
>> [10].
>> * Docker images published to Docker Hub [11]. (Soon)
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Updates from RC1 include a fix to SpannerIO backlog estimation [12] and a
>> fix to the BigQueryIO interpretation of coders on an internal flatten [13].
>> Otherwise, previous validation should be unaffected.
>>
>> For guidelines on how to try the release in your projects, check out our
>> blog post at https://beam.apache.org/blog/validate-beam-release/.
>>
>> Thanks,
>> Robert Burke
>> 2.42.0 Release Manager
>>
>> [1] https://github.com/apache/beam/milestone/4
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.42.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1286/
>> [5] https://github.com/apache/beam/tree/v2.42.0-RC2
>> [6] https://github.com/apache/beam/pull/23406
>> [7] https://github.com/apache/beam-site/pull/634
>> [8] https://pypi.org/project/apache-beam/2.42.0rc2/
>> [9]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.42.0-RC2/go/pkg/beam
>>
>> [10]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=265602293
>> [11] https://hub.docker.com/search?q=apache%2Fbeam&type=image
>> [12] https://github.com/apache/beam/issues/23494
>> [13] https://github.com/apache/beam/issues/23561
>>
>>

Java compatibility checking library

[Pablo Estrada via dev]

Hi folks!
I learned about this library: https://lvc.github.io/japi-compliance-checker/

It looks like it's able to check API compatibility between versions of
libraries. We could instate checks that use this library to be more aware
of backwards incompatibilities that we may introduce accidentally, and at
least be forced to make a deliberate decision about them.

LMK what you think about adding checks with this utility : )

-P.

Re: [ANNOUNCE] Apache Beam 2.41.0 Released

[Pablo Estrada via dev]

Thank you Kiley!

On Thu, Aug 25, 2022 at 10:55 AM Kiley Sok  wrote:

> The Apache Beam team is pleased to announce the release of version 2.41.0.
>
> Apache Beam is an open source unified programming model to define and
> execute data processing pipelines, including ETL, batch and stream
> (continuous) processing. See https://beam.apache.org
>
> You can download the release here:
>
> https://beam.apache.org/get-started/downloads/
>
> This release includes bug fixes, features, and improvements detailed on
> the Beam blog: https://beam.apache.org/blog/beam-2.41.0/
>
> Thanks to everyone who contributed to this release, and we hope you enjoy
> using Beam 2.41.0.
>
> -- Kiley, on behalf of The Apache Beam team
>

Re: [VOTE] Release 2.41.0, release candidate #1

[Pablo Estrada via dev]

+1 - I validated tests/build with existing Dataflow Templates
Best
-P.

On Fri, Aug 12, 2022 at 9:20 PM Ahmet Altay via dev 
wrote:

> +1 - I validated python quickstarts on direct runner.
>
> Thank you Kiley!
>
>
>
> On Thu, Aug 11, 2022 at 9:56 PM Kiley Sok via dev 
> wrote:
>
>> Hi everyone,
>> Please review and vote on the release candidate #1 for the version
>> 2.41.0, as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> Reviewers are encouraged to test their own use cases with the release
>> candidate, and vote +1 if no issues are found.
>>
>> The complete staging area is available for your review, which includes:
>> * GitHub Release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org
>> [2], which is signed with the key with fingerprint
>> 4D5731CC0AA38097D091EB091E7B28884452AE5D [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.41.0-RC1" [5],
>> * website pull request listing the release [6], the blog post [6], and
>> publishing the API reference manual [7].
>> * Java artifacts were built with Gradle 7.4 and OpenJDK/Oracle JDK
>> 1.8.0_232.
>> * Python artifacts are deployed along with the source release to the
>> dist.apache.org [2] and PyPI[8].
>> * Validation sheet with a tab for 2.41.0 release to help with validation
>> [9].
>> * Docker images published to Docker Hub [10].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> For guidelines on how to try the release in your projects, check out our
>> blog post at https://beam.apache.org/blog/validate-beam-release/.
>>
>> Thanks,
>> Release Manager
>>
>> [1] https://github.com/apache/beam/milestone/3
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.41.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1282/
>> [5] https://github.com/apache/beam/tree/v2.41.0-RC1
>> [6] https://github.com/apache/beam/pull/22706
>> [7] https://github.com/apache/beam-site/pull/633
>> [8] https://pypi.org/project/apache-beam/2.41.0rc1/
>> [9]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=331459080
>> [10] https://hub.docker.com/search?q=apache%2Fbeam&type=image
>>
>

Re: Unable to load class 'org.apache.beam.gradle.BeamModulePlugin'

[Pablo Estrada via dev]

Hi there!
Did you try following the instructions here:
https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch

Or how did you try to load your project into IntelliJ?
Best
-P.

On Mon, Aug 8, 2022 at 8:38 PM 张涛  wrote:

> Hi, I download the source code beam-2.40.0.tar.gz, and my gradle version
> is 7.4.2, JVM is 1.8, like this:
>
> when I used IDEA to open the beam project, the following error occured:
> I'm confused about how to deal with this, I am looking forward for your
> help,  thank you very much!
>

Re: [VOTE] Vendored Dependencies Release

[Pablo Estrada via dev]

+1
Thanks!
-P.

On Mon, Aug 8, 2022 at 9:24 AM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> +1
>
> Thanks,
> Cham
>
> On Fri, Aug 5, 2022 at 1:49 PM Luke Cwik via dev 
> wrote:
>
>> +1
>>
>> I verified the signatures of the artifacts, that the jar doesn't contain
>> classes outside of the org/apache/beam/vendor/grpc/v1p48p1 package and I
>> tested the artifact against our precommits using
>> https://github.com/apache/beam/pull/22595
>>
>> On Fri, Aug 5, 2022 at 1:42 PM Luke Cwik  wrote:
>>
>>> Please review the release of the following artifacts that we vendor:
>>>  * beam-vendor-grpc-1_48_1
>>>
>>> Hi everyone,
>>> Please review and vote on the release candidate #1 for the version 0.1,
>>> as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> The complete staging area is available for your review, which includes:
>>> * the official Apache source release to be deployed to dist.apache.org
>>> [1], which is signed with the key with fingerprint
>>> EAD5DE293F4A03DD2E77565589E68A56E371CCA2 [2],
>>> * all artifacts to be deployed to the Maven Central Repository [3],
>>> * commit hash "db8db0b6ed0fe1e4891f207f0f7f811798e54db1" [4],
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> Thanks,
>>> Release Manager
>>>
>>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
>>> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [3]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1277/
>>> [4]
>>> https://github.com/apache/beam/commit/db8db0b6ed0fe1e4891f207f0f7f811798e54db1
>>>
>>

Re: Join a meeting to help coordinate implementing a Dask Runner for Beam

[Pablo Estrada via dev]

+Brian Hulette  and +Kenneth Knowles  were
curious about this effort.

Alex can you elaborate on what happened with GH actions? We can help you
figure out the CI story.

On Tue, Jun 21, 2022 at 11:06 AM Alex Merose  wrote:

> We had a great meeting last week on this topic! Here is a proposal /
> meeting notes doc:
>
> https://docs.google.com/document/d/1Awj_eNmH-WRSte3bKcCcUlQDiZ5mMKmCO_xV-mHWAak/edit#heading=h.y0pwg4polebc
>
> Tomorrow, another engineer (https://github.com/cisaacstern) and I are
> meeting to create an initial prototype of the Dask runner in the main Beam
> repo. Let us know if you'd like to help out in any way. We'll post updates
> in this mailing list + the above doc.
>
> Best,
> Alex Merose
>
> On 2022/06/08 14:22:41 Ryan Abernathey wrote:
> > Dear Beamer,
> >
> > Thank you for all of your work on this amazing project. I am new to Beam
> > and am quite excited about its potential to help with some data
> processing
> > challenges in my field of climate science.
> >
> > Our community is interested in running Beam on Dask Distributed clusters,
> > which we already know how to deploy. This has been discussed at
> > https://issues.apache.org/jira/browse/BEAM-5336 and
> > https://github.com/apache/beam/issues/18962. It seems technically
> feasible.
> >
> > We are trying to organize a meeting next week to kickstart and coordinate
> > this effort. It would be great if we could entrain some Beam maintainers
> > into this meeting. If you have interest in this topic and are available
> > next week, please share your availability here -
> > https://www.when2meet.com/?15861604-jLnA4
> >
> > Alternatively, if you have any guidance or suggestions you wish to
> provide
> > by email or GitHub discussion, we welcome your input.
> >
> > Thanks again for your open source work.
> >
> > Best,
> > Ryan Abernathey
> >
>