[VOTE] Release 2.25.0, release candidate #1

[Robin Qiu]

Hi everyone,
Please review and vote on the release candidate #1 for the version 2.25.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with fingerprint
AD70476B9D1AF3EFEC2208165952E71AACAF911D [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.25.0-RC1" [5],
* website pull request listing the release [6], publishing the API
reference manual [7], and the blog post [8].
* Java artifacts were built with Maven 3.5.3 and OpenJDK/Oracle JDK 11.0.8.
* Python artifacts are deployed along with the source release to the
dist.apache.org [2].
* Validation sheet with a tab for 2.25.0 release to help with validation
[9].
* Docker images published to Docker Hub [10].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Robin

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12347147
[2] https://dist.apache.org/repos/dist/dev/beam/2.25.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1139/
[5] https://github.com/apache/beam/tree/v2.25.0-RC1
[6] https://github.com/apache/beam/pull/13130
[7] https://github.com/apache/beam-site/pull/608
[8] https://github.com/apache/beam/pull/13131
[9]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1494345946
[10] https://hub.docker.com/search?q=apache%2Fbeam=image

Re: Support for Flink 1.11

[Robert Bradshaw]

Support for Flink 1.11 is
https://issues.apache.org/jira/browse/BEAM-10612 . It has been
implemented and will be included in the next release (Beam 2.25). In
the meantime, you could try building yourself from head.

On Fri, Oct 16, 2020 at 4:39 AM Kishor Joshi  wrote:
>
> Hi Team,
>
> Since the beam has released the version 2.24 but it seems still the highest 
> version that is supported in beam 2.24 (>=2.21) is 1.10.
> So i would like to know what is the plan to release beam with the support of 
> latest flink i.e. 1.11.x.
>
> Thanks & Regards,
> Kishor

Re: [VOTE] JupyterLab Sidepanel extension release v1.0.0 for BEAM-10545 RC #1

[Robert Bradshaw]

Thanks, Ning and Ahmet.

+1 (binding) Approve the release.

On Fri, Oct 16, 2020 at 1:34 PM Ning Kang  wrote:

> Sorry, if you cannot see the missing thread history in the previous
> thread, here is another copy:
>
> On Fri, Oct 16, 2020 at 9:55 AM Robert Bradshaw 
> wrote:
>
>> Thanks.
>>
>> +1 (binding) to this release.
>>
>> On Thu, Oct 15, 2020 at 7:06 PM Ahmet Altay  wrote:
>>
>>> Here you go:
>>> https://dist.apache.org/repos/dist/dev/beam/extensions/jupyterlab-sidepanel/v1.0.0/
>>>
>>> On Thu, Oct 15, 2020 at 5:11 PM Robert Bradshaw 
>>> wrote:
>>>
 If we can stage the sources to dist/dev that sounds good to me.

 On Thu, Oct 15, 2020 at 4:57 PM Ning Kang  wrote:

> +1 to Ahmet's suggestion.
>
> I've taken a look at the process
> 
>  used
> by vendored artifacts and summarized below commands to stage the source
> code to dist/dev
>
> extension=jupyterlab-sidepanel
>
> version=v1.0.0
>
> tag=${extension}-${version}
>
>
> svn co https://dist.apache.org/repos/dist/dev/beam
>
> mkdir -p beam/extensions/${extension}/${version}
>
> pushd beam/extensions/${extension}/${version}
>
> curl -o apache-beam-${tag}-source-release.zip https://
> github.com/apache/beam/archive/${tag}.zip
> 
>
> gpg --armor --detach-sig apache-beam-${tag}-source-release.zip
>
> sha512sum apache-beam-${tag}-source-release.zip > apache-beam-${tag}-
> source-release.zip.sha512
>
> # If sha512sum command is not found, on mac, run brew install
> coreutils;
>
> # on linux, run apt-get install coreutils
>
> popd
>
> pushd beam
>
> # For the first time adding the directory with its contents
>
> svn add extensions
>
> # For future versions, use below to add
>
> # svn add extensions/${extension}/${version}
>
> svn commit
>
> Please feel free to comment on the directory structure.
>
> Ahmet, if everything looks good, could you please help me execute the
> commands with your GPG key to stage the source to dist/dev?
> And once we publish the extension to NPM, we'll move the source from
> dist/dev to dist/release following the same process
> 
>  to
> vendored artifact's releases.
>
> I'll document the release process with release history in the Beam
> repo once the release is done.
>
> Thanks!
>
> On Thu, Oct 15, 2020 at 1:06 PM Ahmet Altay  wrote:
>
>> This is similar to the vendored dependencies release. For that we
>> vote on the artifacts. commit hash, and staged source distribution on
>> dist/dev[1]. And then the same source distribution is promoted to
>> dist/release [2]. We can follow the same process and stage a source
>> distribution to dist.
>>
>> [1]
>> https://lists.apache.org/thread.html/rea4a27c47529a27936ab2c51162c8e532b8b625c4d70c4f7f485c7cd%40%3Cdev.beam.apache.org%3E
>> [2] https://dist.apache.org/repos/dist/release/beam/vendor/
>>
>> On Thu, Oct 15, 2020 at 12:17 PM Robert Bradshaw 
>> wrote:
>>
>>> I'm thinking specifically of
>>>
>>>
>>> https://incubator.apache.org/guides/distribution.html#release_platforms
>>>
>>> In addition to the Apache mirror system incubating projects may
>>> distribute artifacts on other platforms as long as they follow these
>>> general guidelines:
>>> * Source releases must be placed in the Apache mirror system.
>>>
>>>
>>> On Thu, Oct 15, 2020 at 12:05 PM Ning Kang  wrote:
>>>
 Thanks Robert, I didn't know the existence of this document.

 Looks like the only thing potentially missing is the incubation
 disclaimer.
 NPM should be the only channel for distribution. And normally, a
 jupyter user would install extensions through `jupyter` commands. They
 wouldn't even use the `npm` command directly.

 Looking at
 https://incubator.apache.org/guides/branding.html#disclaimers,
 since this extension is part of Beam and we are not incubating 
 something
 new, we might not even need an incubation disclaimer.
 The same applies to pypi: https://pypi.org/project/apache-beam/

 What do you think?


 On Thu, Oct 15, 2020 at 11:05 AM Robert Bradshaw <
 rober...@google.com> wrote:

> The code looks good to me. Do we also need to do a
> corresponding source release in the apache distribution channels, or 
> is a
> vote on a specific github commit sufficient?
>

Re: [VOTE] JupyterLab Sidepanel extension release v1.0.0 for BEAM-10545 RC #1

[Ning Kang]

Sorry, if you cannot see the missing thread history in the previous thread,
here is another copy:

On Fri, Oct 16, 2020 at 9:55 AM Robert Bradshaw  wrote:

> Thanks.
>
> +1 (binding) to this release.
>
> On Thu, Oct 15, 2020 at 7:06 PM Ahmet Altay  wrote:
>
>> Here you go:
>> https://dist.apache.org/repos/dist/dev/beam/extensions/jupyterlab-sidepanel/v1.0.0/
>>
>> On Thu, Oct 15, 2020 at 5:11 PM Robert Bradshaw 
>> wrote:
>>
>>> If we can stage the sources to dist/dev that sounds good to me.
>>>
>>> On Thu, Oct 15, 2020 at 4:57 PM Ning Kang  wrote:
>>>
 +1 to Ahmet's suggestion.

 I've taken a look at the process
 
  used
 by vendored artifacts and summarized below commands to stage the source
 code to dist/dev

 extension=jupyterlab-sidepanel

 version=v1.0.0

 tag=${extension}-${version}


 svn co https://dist.apache.org/repos/dist/dev/beam

 mkdir -p beam/extensions/${extension}/${version}

 pushd beam/extensions/${extension}/${version}

 curl -o apache-beam-${tag}-source-release.zip https://
 github.com/apache/beam/archive/${tag}.zip
 

 gpg --armor --detach-sig apache-beam-${tag}-source-release.zip

 sha512sum apache-beam-${tag}-source-release.zip > apache-beam-${tag}-
 source-release.zip.sha512

 # If sha512sum command is not found, on mac, run brew install
 coreutils;

 # on linux, run apt-get install coreutils

 popd

 pushd beam

 # For the first time adding the directory with its contents

 svn add extensions

 # For future versions, use below to add

 # svn add extensions/${extension}/${version}

 svn commit

 Please feel free to comment on the directory structure.

 Ahmet, if everything looks good, could you please help me execute the
 commands with your GPG key to stage the source to dist/dev?
 And once we publish the extension to NPM, we'll move the source from
 dist/dev to dist/release following the same process
 
  to
 vendored artifact's releases.

 I'll document the release process with release history in the Beam repo
 once the release is done.

 Thanks!

 On Thu, Oct 15, 2020 at 1:06 PM Ahmet Altay  wrote:

> This is similar to the vendored dependencies release. For that we vote
> on the artifacts. commit hash, and staged source distribution on
> dist/dev[1]. And then the same source distribution is promoted to
> dist/release [2]. We can follow the same process and stage a source
> distribution to dist.
>
> [1]
> https://lists.apache.org/thread.html/rea4a27c47529a27936ab2c51162c8e532b8b625c4d70c4f7f485c7cd%40%3Cdev.beam.apache.org%3E
> [2] https://dist.apache.org/repos/dist/release/beam/vendor/
>
> On Thu, Oct 15, 2020 at 12:17 PM Robert Bradshaw 
> wrote:
>
>> I'm thinking specifically of
>>
>>
>> https://incubator.apache.org/guides/distribution.html#release_platforms
>>
>> In addition to the Apache mirror system incubating projects may
>> distribute artifacts on other platforms as long as they follow these
>> general guidelines:
>> * Source releases must be placed in the Apache mirror system.
>>
>>
>> On Thu, Oct 15, 2020 at 12:05 PM Ning Kang  wrote:
>>
>>> Thanks Robert, I didn't know the existence of this document.
>>>
>>> Looks like the only thing potentially missing is the incubation
>>> disclaimer.
>>> NPM should be the only channel for distribution. And normally, a
>>> jupyter user would install extensions through `jupyter` commands. They
>>> wouldn't even use the `npm` command directly.
>>>
>>> Looking at
>>> https://incubator.apache.org/guides/branding.html#disclaimers,
>>> since this extension is part of Beam and we are not incubating something
>>> new, we might not even need an incubation disclaimer.
>>> The same applies to pypi: https://pypi.org/project/apache-beam/
>>>
>>> What do you think?
>>>
>>>
>>> On Thu, Oct 15, 2020 at 11:05 AM Robert Bradshaw <
>>> rober...@google.com> wrote:
>>>
 The code looks good to me. Do we also need to do a
 corresponding source release in the apache distribution channels, or 
 is a
 vote on a specific github commit sufficient?

 (Looking at
 https://incubator.apache.org/guides/distribution.html#npm)

>>>
On Fri, Oct 16, 2020 at 1:29 PM Ning Kang  wrote:

> Thanks Ahmet and Robert! Please see the quote attached for the missing
> history of discussion and vote from 

Re: [VOTE] JupyterLab Sidepanel extension release v1.0.0 for BEAM-10545 RC #1

[Ning Kang]

Thanks Ahmet and Robert! Please see the quote attached for the missing
history of discussion and vote from Robert, sorry for the confusion.
Based on Robert's suggestion, we updated the original vote thread with a
new link to "the official Apache source release to be deployed to
dist.apache.org [4]".
The whole vote thread is copied with the update as below. No source or
binary change. the vote will be open throughout the weekend:

Please review the release of the following jupyter labextension (TypeScript
node package) for running Beam notebooks in JupyterLab:
* apache-beam-jupyterlab-sidepanel

Hi everyone,
Please review and vote on the release candidate #1 for the version 1.0.0,
as follows:
[ ] +1, Approve the release
[ ] -1. Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* the assets (only the
`sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel`
sub directory) to be published to npmjs.com [1]
* commit hash "b7ae7bb1dc28a7c8f26e9f48682e781a74e2d3c4" [2]
* NPM package will be signed by NPM once published; the pgp machinery [3]
* the official Apache source release to be deployed to dist.apache.org [4]

Additional details:
* to install the package before it being published, install it locally by
cloning the Beam repo or downloading the assets:

git checkout jupyterlab-sidepanel-v1.0.0 -b some-branch # if cloning the
repo, do this step

pushd sdks/python/apache_beam/runners/interactive/extensions/apache-beam-
jupyterlab-sidepanel

jlpm

jlpm build

jupyter labextension link .
* screenshots of the extension [5]
* a publish dry run:

npm notice === Tarball Details ===

npm notice name:  apache-beam-jupyterlab-sidepanel

npm notice version:   1.0.0

npm notice package size:  19.8 kB

npm notice unpacked size: 101.9 kB

npm notice shasum:7f896de0d6e587aab2bef348a6e94f95f75f280f

npm notice integrity: sha512-hdkn2Ni2S0roY[...]ShMK2/MAbQvyQ==

npm notice total files:   51

npm notice

+ apache-beam-jupyterlab-sidepanel@1.0.0

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks!

[1] https://github.com/apache/beam/releases/tag/jupyterlab-sidepanel-v1.0.0
[2]
https://github.com/apache/beam/commit/b7ae7bb1dc28a7c8f26e9f48682e781a74e2d3c4
[3] https://blog.npmjs.org/post/172999548390/new-pgp-machinery
[4]
https://dist.apache.org/repos/dist/dev/beam/extensions/jupyterlab-sidepanel/v1.0.0/

[5]
https://docs.google.com/document/d/1aKK8TzSrl8WiG0K4v9xZEfLMCinuGqRlMOyb7xOhgy4/edit#heading=h.he7se5yxfo7

On Fri, Oct 9, 2020 at 1:58 PM Ning Kang  wrote:

> To Pablo,
>
> The public key in use by NPM can be found in this blog "[3]
> https://blog.npmjs.org/post/172999548390/new-pgp-machinery;. A direct
> link: https://keybase.io/npmregistry/pgp_keys.asc
> Quoted from the blog:
>
>> We’ve also chosen to use Keybase
>> 
>>  to
>> publicize our PGP key and give you confidence that the npm registry you
>> install from is the same registry that’s signing packages. Our account on
>> Keybase is npmregistry
>> 
>> .
>
> Keybase can be found here: https://keybase.io/
>
> Thanks!
> Ning.
>
> On Fri, Oct 9, 2020 at 1:53 PM Pablo Estrada  wrote:
>
>> +1
>> I installed the extension and reviewed it as well.
>>
>> I have a question: You mention that NPM will sign the package. What key
>> will it use? We may need to upload your pgp key to the Beam list of keys?
>> Thanks Ning!
>> -P.
>>
>> On Tue, Oct 6, 2020 at 2:57 PM Ning Kang  wrote:
>>
>>> Please review the release of the following jupyter labextension
>>> (TypeScript node package) for running Beam notebooks in JupyterLab:
>>> * apache-beam-jupyterlab-sidepanel
>>>
>>> Hi everyone,
>>> Please review and vote on the release candidate #1 for the version
>>> 1.0.0, as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1. Do not approve the release (please provide specific comments)
>>>
>>> The complete staging area is available for your review, which includes:
>>> * the assets (only the
>>> `sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel`
>>> sub directory) to be published to npmjs.com [1]
>>> * commit hash "b7ae7bb1dc28a7c8f26e9f48682e781a74e2d3c4" [2]
>>> * package will be signed by NPM once published; the pgp machinery [3]
>>>
>>> 

Re: [Proposal] Website Revamp project

[Alexey Romanenko]

Welcome to Beam! 

Regards,
Alexey

> On 15 Oct 2020, at 16:06, Nam Bui  wrote:
> 
> Hi everyone,
> 
> I'm Nam. I used to work on Apache Beam website migration. From now on, I will 
> be responsible for website development. Thus, I will implement new designs 
> and some of the functionalities on our website. Great to be working with you!
> 
> Best regards,
> Nam
> 
> 
> 
> On Thu, Oct 15, 2020 at 8:42 PM Kasia Zając  > wrote:
> Hi all, 
> 
> I am Kasia and I will be responsible for the UX side of things in the 
> project. I will be preparing the wireframes and also asking you for feedback 
> later in the process. Happy to join the community at least for a short amount 
> of time :) 
> 
> Have a good day,
>   
> Kasia Zając
> Product Owner
> +48 664 115 440 
> ka...@utilodesign.com 
>  
> www.utilodesign.com 
>  
>    
>   
> 
> On Thu, Oct 15, 2020 at 2:12 PM Agnieszka Sell  > wrote:
> Hi Everyone,
> 
> I'm Agnieszka Sell, Project Manager in the Beam website revamp project.
> 
> I'm looking forward to getting your feedback on design and development 
> progress we'll be sharing with you in upcoming weeks.
> 
> Best,
> 
> Agnieszka 
> 
> On Wed, Oct 14, 2020 at 7:06 PM Gris Cuevas  > wrote:
> Hi Everyone, 
> 
> We're ready to start work on the revamp of the website, we'll use the PRD 
> shared in this thread previously. 
> 
> Polidea will be the team working on this revamp and we'll be bringing designs 
> and proposals to the community for review as the project progresses. 
> 
> Thank you! 
> Gris
> 
> On 2020/09/09 18:14:18, Gris Cuevas  > wrote: 
> > Hi Beam community, 
> > 
> > In a previous thread [1] I mentioned that I was going to work on product 
> > requirements document (PRD) for a project to address some of the requests 
> > and ideas collected by Aizhamal, Rose and David in previous efforts. 
> > 
> > The PRD is ready [2], and I'd like to get your feedback before moving 
> > forward into implementation. Please add you comments by Sunday Septermber 
> > 13, 2020. 
> > 
> > We're looking to start work on this project in around 2 weeks. 
> > 
> > Thank you! 
> > Gris
> > 
> > [1] 
> > https://lists.apache.org/thread.html/r1a4cea1e8b53bef73c49f75df13956024d8d78bc81b36e54ef71f8a9%40%3Cdev.beam.apache.org%3E
> >  
> > 
> > 
> > [2] https://s.apache.org/beam-site-revamp 
> > 
> > 
> 
> 
> -- 
>   
> Agnieszka Sell
> Polidea  | Project Manager
> M: +48 504 901 334 
> E: agnieszka.s...@polidea.com 
>  
> Check out our projects! 
>    
>  
>    
>   
> 
> Unique Tech
> Check out our projects! 

Re: [VOTE] JupyterLab Sidepanel extension release v1.0.0 for BEAM-10545 RC #1

[Ning Kang]

Thanks Ahmet and Robert! Please see the quote attached for the missing
history of discussion and vote from Robert, sorry for the confusion.

On Fri, Oct 16, 2020 at 9:55 AM Robert Bradshaw  wrote:

> Thanks.
>
> +1 (binding) to this release.
>
> On Thu, Oct 15, 2020 at 7:06 PM Ahmet Altay  wrote:
>
>> Here you go:
>> https://dist.apache.org/repos/dist/dev/beam/extensions/jupyterlab-sidepanel/v1.0.0/
>>
>> On Thu, Oct 15, 2020 at 5:11 PM Robert Bradshaw 
>> wrote:
>>
>>> If we can stage the sources to dist/dev that sounds good to me.
>>>
>>> On Thu, Oct 15, 2020 at 4:57 PM Ning Kang  wrote:
>>>
 +1 to Ahmet's suggestion.

 I've taken a look at the process
 
 used by vendored artifacts and summarized below commands to stage the
 source code to dist/dev

 extension=jupyterlab-sidepanel

 version=v1.0.0

 tag=${extension}-${version}


 svn co https://dist.apache.org/repos/dist/dev/beam

 mkdir -p beam/extensions/${extension}/${version}

 pushd beam/extensions/${extension}/${version}

 curl -o apache-beam-${tag}-source-release.zip https://
 github.com/apache/beam/archive/${tag}.zip
 

 gpg --armor --detach-sig apache-beam-${tag}-source-release.zip

 sha512sum apache-beam-${tag}-source-release.zip > apache-beam-${tag}-
 source-release.zip.sha512

 # If sha512sum command is not found, on mac, run brew install
 coreutils;

 # on linux, run apt-get install coreutils

 popd

 pushd beam

 # For the first time adding the directory with its contents

 svn add extensions

 # For future versions, use below to add

 # svn add extensions/${extension}/${version}

 svn commit

 Please feel free to comment on the directory structure.

 Ahmet, if everything looks good, could you please help me execute the
 commands with your GPG key to stage the source to dist/dev?
 And once we publish the extension to NPM, we'll move the source from
 dist/dev to dist/release following the same process
 
 to vendored artifact's releases.

 I'll document the release process with release history in the Beam repo
 once the release is done.

 Thanks!

 On Thu, Oct 15, 2020 at 1:06 PM Ahmet Altay  wrote:

> This is similar to the vendored dependencies release. For that we vote
> on the artifacts. commit hash, and staged source distribution on
> dist/dev[1]. And then the same source distribution is promoted to
> dist/release [2]. We can follow the same process and stage a source
> distribution to dist.
>
> [1]
> https://lists.apache.org/thread.html/rea4a27c47529a27936ab2c51162c8e532b8b625c4d70c4f7f485c7cd%40%3Cdev.beam.apache.org%3E
> [2] https://dist.apache.org/repos/dist/release/beam/vendor/
>
> On Thu, Oct 15, 2020 at 12:17 PM Robert Bradshaw 
> wrote:
>
>> I'm thinking specifically of
>>
>>
>> https://incubator.apache.org/guides/distribution.html#release_platforms
>>
>> In addition to the Apache mirror system incubating projects may
>> distribute artifacts on other platforms as long as they follow these
>> general guidelines:
>> * Source releases must be placed in the Apache mirror system.
>>
>>
>> On Thu, Oct 15, 2020 at 12:05 PM Ning Kang  wrote:
>>
>>> Thanks Robert, I didn't know the existence of this document.
>>>
>>> Looks like the only thing potentially missing is the incubation
>>> disclaimer.
>>> NPM should be the only channel for distribution. And normally, a
>>> jupyter user would install extensions through `jupyter` commands. They
>>> wouldn't even use the `npm` command directly.
>>>
>>> Looking at
>>> https://incubator.apache.org/guides/branding.html#disclaimers,
>>> since this extension is part of Beam and we are not incubating something
>>> new, we might not even need an incubation disclaimer.
>>> The same applies to pypi: https://pypi.org/project/apache-beam/
>>>
>>> What do you think?
>>>
>>>
>>> On Thu, Oct 15, 2020 at 11:05 AM Robert Bradshaw <
>>> rober...@google.com> wrote:
>>>
 The code looks good to me. Do we also need to do a
 corresponding source release in the apache distribution channels, or 
 is a
 vote on a specific github commit sufficient?

 (Looking at
 https://incubator.apache.org/guides/distribution.html#npm)

>>>
Based on Robert's suggestion, we updated the original vote thread with a
new link to "the official Apache source release to be 

Re: [Proposal] Website Revamp project

[Milka Toikkanen]

Hi Everyone,

I’m Milka and I’m a UI & motion designer. I will be applying the designs on
Kasia’s wireframes and take care of the look and feel, as well as
accessible UI solutions. Looking forward to working with you!



Best,


-- 

Milka Toikkanen
UI & Motion Designer

+48 577 926 829 <+48577926829>
mi...@utilodesign.com
[image: utilo] 

www.utilodesign.com
[image: Behance]  [image: dribbble]
 [image: Instagram]
 [image: Linkedin]

Re: Contributor Permission for Beam Jira

[Ahmet Altay]

Welcome. I added you as a contributor to Jira.

On Fri, Oct 16, 2020 at 10:11 AM Aliraza Nagamia 
wrote:

> Sorry I forgot to mention my jira account handle: alirazan
>
> On Fri, Oct 16, 2020 at 10:36 AM Aliraza Nagamia 
> wrote:
>
>> Hello,
>>
>> I am working on a project related to Google Cloud Platform's Dicom api. I
>> would like to contribute code for a Dicom IO connector. Could I please have
>> permission to add/assign tickets on the Beam Jira?
>>
>> Thank you,
>> Aliraza
>>
>

Re: Contributor Permission for Beam Jira

[Aliraza Nagamia]

Sorry I forgot to mention my jira account handle: alirazan

On Fri, Oct 16, 2020 at 10:36 AM Aliraza Nagamia 
wrote:

> Hello,
>
> I am working on a project related to Google Cloud Platform's Dicom api. I
> would like to contribute code for a Dicom IO connector. Could I please have
> permission to add/assign tickets on the Beam Jira?
>
> Thank you,
> Aliraza
>

Contributor Permission for Beam Jira

[Aliraza Nagamia]

Hello,

I am working on a project related to Google Cloud Platform's Dicom api. I
would like to contribute code for a Dicom IO connector. Could I please have
permission to add/assign tickets on the Beam Jira?

Thank you,
Aliraza

Re: Requesting Jira contributor access for Website revamp project team

[Alexey Romanenko]

Hi Gris,

It’s done. 

Regards,
Alexey

> On 16 Oct 2020, at 16:57, Gris Cuevas  wrote:
> 
> Hi folks, could someone in the PMC provide "Contributor" roles to the 
> following folks who will be working on the website revamp? 
> 
> Usernames: nam.bui, Milka, AgnieszkaSell, kasiazjc
> 

Requesting Jira contributor access for Website revamp project team

[Gris Cuevas]

Hi folks, could someone in the PMC provide "Contributor" roles to the following 
folks who will be working on the website revamp? 

Usernames: nam.bui, Milka, AgnieszkaSell, kasiazjc

Re: [Proposal] Add a new Beam example to ingest data from Kafka to Pub/Sub

[Ilya Kozyrev]

Thanks a lot for your comments!

I see your point regarding the location of such templates.
I moved the templates folder under the examples folder. Currently it looks like 
/examples/templates/java/kafka-to-pubsub.

Does it look better?

Thank you,
Ilya
On 15 Oct 2020, 20:30 +0300, Kyle Weaver , wrote:
I agree with Kenn. Since any pipeline can be made into a template, it doesn't 
really make sense to have a separate "templates" directory. Based on a quick 
skim of your PR, the only thing that's specific to Dataflow templates is the 
instructions.

Gradle does not make subprojects inherit dependencies from their parents. So 
you could move your subproject under the `examples/java` directory and it 
should still build exactly the same.

On Thu, Oct 15, 2020 at 7:34 AM Ilya Kozyrev 
mailto:ilya.kozy...@akvelon.com>> wrote:
Hi!

Thanks for joining the discussion a lot!

By creating new templates folder in the Beam we try to make templates generic. 
As it seems to us, another important reason for adding a new folder is Gradle 
build. We will configure Gradle build directly for templates, without a 
necessity to build all examples, and without unused dependencies.

Creating subfolders for different runners makes a sense, however, the proposed 
template could be running on the different runners, and GCP here is an optional 
approach than a requirement.

I suggest adding a subfolder for different languages, under the /templates 
e.g.: /templates/python, /templates/java, etc. If we will need more differences 
between templates we can add new subfolders for specific cases in the future.

Thank you,
Ilya
On 15 Oct 2020, 07:30 +0300, Reza Ardeshir Rokni 
mailto:raro...@gmail.com>>, wrote:
Just a thought, but what if in the future there were templates for other 
runners?

Then having a template folder would fit nicely no? We could even have a runner 
specific subfolder and maybe even a shared area for things that could be used 
by all templates for all runners?

On Thu, 15 Oct 2020 at 11:47, Kenneth Knowles 
mailto:k...@apache.org>> wrote:
Hi Ilya,

I have added you to the "Contributors" role on Jira so you can be assigned 
tickets, and given you the ticket you filed since you are already solving it. 
Thanks!

I have a very high level thought: Since Dataflow's "Flex Templates" feature is 
just any pipeline, perhaps the main pipeline can be more of an "example" and 
fit into the `examples/` folder? Then the containerization and Google-specific* 
JSON could be alongside. In this way, users of other runners could possibly use 
or learn from it even if they are not interested in GCP. I understand this is 
not your primary goal, considering the contribution. I just want to open this 
for discussion.

Kenn

*In fact, the JSON is very generic. It is not really "Google specific" in 
concept, just in practice.

On Wed, Oct 14, 2020 at 12:14 PM Ilya Kozyrev 
mailto:ilya.kozy...@akvelon.com>> wrote:
Hi Beam Community,

There was no feedback on the proposal, and I would like to submit PR for this 
proposal.

I created a JIRA improvement 
to track this proposal and now submitting  
PR in the Beam repository related to 
the proposal that I sent before. We suggest adding /template folder to the 
repository root to help discover templates by developers. This will provide 
structure for future templates development for Beam.

Could someone kindle help with reviewing the 
PR ?

Thank you,
Ilya

On 7 Oct 2020, at 21:23, Ilya Kozyrev 
mailto:ilya.kozy...@akvelon.com>> wrote:

Hi Beam Community,

I have a proposal to add Apache Beam example that is a template to ingest data 
from Apache Kafka to Google Cloud Pub/Sub. More detailed information about the 
proposed template can be found in 
README
 file, and a prototype was built with a 
team. I'd like to ask for your feedback before moving forward with finishing 
the template.

I did not see a folder that provides easily discoverable templates to a 
developer.  I would like to propose adding a "templates" folder where other 
Apache Beam templates may be added in the future. E.g., 
beam/templates/java/kafka-to-pubsub could be used for the Kafka to Pub/Sub 
template.

Please share your feedback/comments about this proposal in the thread.

Thank you,
Ilya

Support for Flink 1.11

[Kishor Joshi]

Hi Team,
Since the beam has released the version 2.24 but it seems still the highest 
version that is supported in beam 2.24 (>=2.21) is 1.10.So i would like to know 
what is the plan to release beam with the support of latest flink i.e. 1.11.x.
Thanks & Regards,Kishor

Re: Throttling stream outputs per trigger?

[Maximilian Michels]

the downstream consumer has these requirements. 


Blocking should normally be avoided at all cost, but if the downstream 
operator has the requirement to only emit a fixed number of messages per 
second, it should enforce this, i.e. block once the maximum number of 
messages for a time period have been reached. This will automatically 
lead to backpressure in Runners like Flink or Dataflow.


-Max

On 07.10.20 18:30, Luke Cwik wrote:
SplittableDoFns apply to both batch and streaming pipelines. They are 
allowed to produce an unbounded amount of data and can either self 
checkpoint saying they want to resume later or the runner will ask them 
to checkpoint via a split call.


There hasn't been anything concrete on backpressure, there has been work 
done about exposing signals[1] related to IO that a runner can then use 
intelligently but throttling isn't one of them yet.


1: 
https://lists.apache.org/thread.html/r7c1bf68bd126f3421019e238363415604505f82aeb28ccaf8b834d0d%40%3Cdev.beam.apache.org%3E 



On Tue, Oct 6, 2020 at 3:51 PM Vincent Marquez 
mailto:vincent.marq...@gmail.com>> wrote:


Thanks for the response.  Is my understanding correct that
SplittableDoFns are only applicable to Batch pipelines?  I'm
wondering if there's any proposals to address backpressure needs?
/~Vincent/


On Tue, Oct 6, 2020 at 1:37 PM Luke Cwik mailto:lc...@google.com>> wrote:

There is no general back pressure mechanism within Apache Beam
(runners should be intelligent about this but there is currently
no way to say I'm being throttled so runners don't know that
throwing more CPUs at a problem won't make it go faster). Y

You can control how quickly you ingest data for runners that
support splittable DoFns with SDK initiated checkpoints with
resume delays. A splittable DoFn is able to return
resume().withDelay(Duration.seconds(10)) from
the @ProcessElement method. See Watch[1] for an example.

The 2.25.0 release enables more splittable DoFn features on more
runners. I'm working on a blog (initial draft[2], still mostly
empty) to update the old blog from 2017.

1:

https://github.com/apache/beam/blob/9c239ac93b40e911f03bec5da3c58a07fdceb245/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Watch.java#L908


2:

https://docs.google.com/document/d/1kpn0RxqZaoacUPVSMYhhnfmlo8fGT-p50fEblaFr2HE/edit#




On Tue, Oct 6, 2020 at 10:39 AM Vincent Marquez
mailto:vincent.marq...@gmail.com>>
wrote:

Hmm, I'm not sure how that will help, I understand how to
batch up the data, but it is the triggering part that I
don't see how to do.  For example, in Spark Structured
Streaming, you can set a time trigger which happens at a
fixed interval all the way up to the source, so the source
can throttle how much data to read even.

Here is my use case more thoroughly explained:

I have a Kafka topic (with multiple partitions) that I'm
reading from, and I need to aggregate batches of up to 500
before sending a single batch off in an RPC call.  However,
the vendor specified a rate limit, so if there are more than
500 unread messages in the topic, I must wait 1 second
before issuing another RPC call. When searching on Stack
Overflow I found this answer:
https://stackoverflow.com/a/57275557/25658
 that makes it
seem challenging, but I wasn't sure if things had changed
since then or you had better ideas.

/~Vincent/


On Thu, Oct 1, 2020 at 2:57 PM Luke Cwik mailto:lc...@google.com>> wrote:

Look at the GroupIntoBatches[1] transform. It will
buffer "batches" of size X for you.

1:

https://beam.apache.org/documentation/transforms/java/aggregation/groupintobatches/



On Thu, Oct 1, 2020 at 2:51 PM Vincent Marquez
mailto:vincent.marq...@gmail.com>> wrote:

the downstream consumer has these requirements.

/~Vincent/


On Thu, Oct 1, 2020 at 2:29 PM Luke Cwik
mailto:lc...@google.com>> wrote:

Why do you want to only emit X?