Beam Summit 2024: CfP is open!

[Teyza Ponce]

[image: Beam Summit Virtual] 

*Dear Beam Community,*

We are glad to announce that the dates for Beam Summit 2024 are confirmed!
Join us next September 4-5 at Sunnyvale, CA. Submit your talk proposal, and
watch the recordings of the previous edition in 2023.
️ Save the date 
[image: Join our speakers] 


 Beam Summit 2024 is coming soon, and we can’t wait to hear from you! The
call for papers is open now. *Submit your proposal here*
 to share your insights and
stories on how or where you are using Apache Beam 



Watch the previous recordings


Beam Summit 2023 was a success! If you missed it, take a look at our *Youtube
channel * and dive into the
recordings we have prepared for you. If you'd like to experience in person
more content like this around Apache Beam, we look forward to seeing you at
Beam Summit 2024!


*Sponsors and partners are welcome!*

Be one of the organizations that support the Beam community. If you can
sponsor the event or partner with us, we have many opportunities for you.
Contact us for more information: *cont...@beamsummit.org
 *


*See you soon in Sunnyvale!*
[image: Tw]  [image: Yt]
 [image: In]

Re: 2.56.0 release

[Danny McCormick via dev]

I just cut the release branch for the 2.56.0 release. There are still
several open issues for the milestone, some of which will require
cherry-picks [1], so there is still some time if you have urgent changes
that you would like to make the cut. I'll now move into stabilizing the
release branch [2].

Thanks,
Danny

[1] https://github.com/apache/beam/milestone/20
[2] https://github.com/apache/beam/tree/release-2.56.0

On Mon, Apr 15, 2024 at 4:09 PM Ahmet Altay via dev 
wrote:

> Thank you Danny!
>
> On Fri, Apr 5, 2024 at 3:20 AM Maciej Szwaja via dev 
> wrote:
>
>> Hi,
>>
>> a nice round version number :)
>>
>> Cheers,
>> Maciej
>>
>> On Thu, Apr 4, 2024 at 3:56 PM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hey everyone,
>>>
>>> On top of running the current 2.55.1 patch release, I'd like to
>>> volunteer as release manager for the 2.56.0 release, which will be cut on
>>> April 17. Please add any release blockers to the 2.56.0 milestone [1], I
>>> will be trying to push on those as the cut date approaches.
>>>
>>> Thanks,
>>> Danny
>>>
>>> [1] https://github.com/apache/beam/milestone/20
>>>
>>

Re: release process: typescript SDK?

[Danny McCormick via dev]

Probably the easiest way for this to happen is for @Robert Bradshaw
 to get the token set up as a secret (should be quick)
and then Austin to take the workflow forward.

In the past to get secrets added, Infra has asked that I (a) email
r...@apache.org with the secret name and secret contents, and (b) opened a
JIRA to externally track progress -
https://issues.apache.org/jira/browse/INFRA-25009

On Wed, Apr 17, 2024 at 11:24 AM Austin Bennett  wrote:

> I don't mind doing, esp. if nobody is eager to handle/prioritize the push
> artifact in near-term.  If I'm to do, let's connect off-list for
> token/creds.
>
> Furthermore, I agree that getting RCs as part of the overall
> release/validation process would be a nice addition.
>
> On Tue, Apr 16, 2024 at 2:43 PM Robert Bradshaw via dev <
> dev@beam.apache.org> wrote:
>
>> Correct, I've just been pushing these manually, and lately there haven't
>> been many changes to push. I'm all for getting these set up as part of the
>> standard release process.
>>
>> On Tue, Apr 16, 2024 at 1:22 PM Danny McCormick <
>> dannymccorm...@google.com> wrote:
>>
>>> I've never published npm artifacts before, but I imagine the hardest
>>> part is getting the credentials set up, then it is probably very easy to
>>> set up a GitHub Actions workflow to publish
>>> .
>>> Who has done these releases in the past/has credentials for the npm
>>> package? Maybe @Robert Bradshaw ? We will need a
>>> token set up as a secret to automate this.
>>>
>>> I'll also note that we don't do any typescript validation today, and it
>>> would be nice to publish RCs as part of this
>>>
>>> On Tue, Apr 16, 2024 at 4:11 PM Austin Bennett 
>>> wrote:
>>>
 Hi Beam Devs,

 Calling out it looks like our release process for apache-beam for
 typescript/npm is broken, seemingly the last published release was 2.49.0
 about 9 months ago.  The other languages look like they are publishing to
 expected locations.

 https://www.npmjs.com/package/apache-beam

 I noticed this since I was digging into security concerns raised by
 GitHub's dependabot across our repos [ ex:
 https://github.com/apache/beam-starter-typescript/security/dependabot ], 
 and
 towards getting our repos tidied.

 This leads me to believe we may want two distinct things:
 * update our release docs/process/scripts to ensure that we
 generate/publish all artifacts to relevant repositories.
 * Arrive at a process to more straightforwardly attend to security
 updates [ maybe we want these sent to dev list, or another distribution? ]

 From a very quick search, it did not look like we have scripts to push
 to npm.  That should be verified more thoroughly -- i haven't done a
 release before, so relevant scripts could be hiding elsewhere.

 Cheers,
 Austin


 NOTE:  everything with our main Beam repo specifically looks OK.  Some
 things discovered were on the other/supplementary repos, though I believe
 those are still worthwhile to attend to and support.

>>>

Re: release process: typescript SDK?

[Austin Bennett]

I don't mind doing, esp. if nobody is eager to handle/prioritize the push
artifact in near-term.  If I'm to do, let's connect off-list for
token/creds.

Furthermore, I agree that getting RCs as part of the overall
release/validation process would be a nice addition.

On Tue, Apr 16, 2024 at 2:43 PM Robert Bradshaw via dev 
wrote:

> Correct, I've just been pushing these manually, and lately there haven't
> been many changes to push. I'm all for getting these set up as part of the
> standard release process.
>
> On Tue, Apr 16, 2024 at 1:22 PM Danny McCormick 
> wrote:
>
>> I've never published npm artifacts before, but I imagine the hardest part
>> is getting the credentials set up, then it is probably very easy to set up
>> a GitHub Actions workflow to publish
>> .
>> Who has done these releases in the past/has credentials for the npm
>> package? Maybe @Robert Bradshaw ? We will need a
>> token set up as a secret to automate this.
>>
>> I'll also note that we don't do any typescript validation today, and it
>> would be nice to publish RCs as part of this
>>
>> On Tue, Apr 16, 2024 at 4:11 PM Austin Bennett  wrote:
>>
>>> Hi Beam Devs,
>>>
>>> Calling out it looks like our release process for apache-beam for
>>> typescript/npm is broken, seemingly the last published release was 2.49.0
>>> about 9 months ago.  The other languages look like they are publishing to
>>> expected locations.
>>>
>>> https://www.npmjs.com/package/apache-beam
>>>
>>> I noticed this since I was digging into security concerns raised by
>>> GitHub's dependabot across our repos [ ex:
>>> https://github.com/apache/beam-starter-typescript/security/dependabot ], and
>>> towards getting our repos tidied.
>>>
>>> This leads me to believe we may want two distinct things:
>>> * update our release docs/process/scripts to ensure that we
>>> generate/publish all artifacts to relevant repositories.
>>> * Arrive at a process to more straightforwardly attend to security
>>> updates [ maybe we want these sent to dev list, or another distribution? ]
>>>
>>> From a very quick search, it did not look like we have scripts to push
>>> to npm.  That should be verified more thoroughly -- i haven't done a
>>> release before, so relevant scripts could be hiding elsewhere.
>>>
>>> Cheers,
>>> Austin
>>>
>>>
>>> NOTE:  everything with our main Beam repo specifically looks OK.  Some
>>> things discovered were on the other/supplementary repos, though I believe
>>> those are still worthwhile to attend to and support.
>>>
>>

Beam High Priority Issue Report (63)

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P1 Issues:

https://github.com/apache/beam/issues/31006 The Python tests job is flaky
https://github.com/apache/beam/issues/30966 The PreCommit Python Docs job is 
flaky
https://github.com/apache/beam/issues/30955 [Failing Test]: setuptools 69.3.0 
breaks sdist
https://github.com/apache/beam/issues/30941 [Failing Test]: Java IO Precommit 
failing due to KafkaIOTranslation being out of sync
https://github.com/apache/beam/issues/30927 [Failing Test]: 
BigtableIOWriteTest::test_bigtable_write is about 50% flaky
https://github.com/apache/beam/issues/30904 The PostCommit XVR PythonUsingJava 
Dataflow job is flaky
https://github.com/apache/beam/issues/30899 The PostCommit XVR JavaUsingPython 
Dataflow job is flaky
https://github.com/apache/beam/issues/30760 The PostCommit Python Arm job is 
flaky
https://github.com/apache/beam/issues/30757 [Bug]: Beam Playground scio 
examples cannot run
https://github.com/apache/beam/issues/30737 [Failing Test]: Playground 
PreCommit failing goLint
https://github.com/apache/beam/issues/30644 The Inference Python Benchmarks 
Dataflow job is flaky
https://github.com/apache/beam/issues/30612 The Playground CI Nightly job is 
flaky
https://github.com/apache/beam/issues/30606 The PostCommit Java Nexmark 
Dataflow job is flaky
https://github.com/apache/beam/issues/30593 The PostCommit XVR 
PythonUsingJavaSQL Dataflow job is flaky
https://github.com/apache/beam/issues/30530 The LoadTests Java GBK Smoke job is 
flaky
https://github.com/apache/beam/issues/30529 The PostCommit Java Sickbay job is 
flaky
https://github.com/apache/beam/issues/30527 The PostCommit Java IO Performance 
Tests job is flaky
https://github.com/apache/beam/issues/30526 The PerformanceTests xlang KafkaIO 
Python job is flaky
https://github.com/apache/beam/issues/30525 The PostCommit Python 
ValidatesContainer Dataflow With RC job is flaky
https://github.com/apache/beam/issues/30521 The LoadTests Go Combine Flink 
Batch job is flaky
https://github.com/apache/beam/issues/30520 The LoadTests Python Combine Flink 
Streaming job is flaky
https://github.com/apache/beam/issues/30519 The PostCommit XVR GoUsingJava 
Dataflow job is flaky
https://github.com/apache/beam/issues/30517 The PostCommit XVR Direct job is 
flaky
https://github.com/apache/beam/issues/30513 The PostCommit Python job is flaky
https://github.com/apache/beam/issues/30511 The LoadTests Python Smoke job is 
flaky
https://github.com/apache/beam/issues/30507 The LoadTests Go GBK Flink Batch 
job is flaky
https://github.com/apache/beam/issues/30506 The TypeScript Tests job is flaky
https://github.com/apache/beam/issues/30503 The PostCommit Java ValidatesRunner 
Flink Java11 job is flaky
https://github.com/apache/beam/issues/30502 The LoadTests Go CoGBK Flink Batch 
job is flaky
https://github.com/apache/beam/issues/29971 [Bug]: FixedWindows not working for 
large Kafka topic
https://github.com/apache/beam/issues/29926 [Bug]: FileIO: lack of timeouts may 
cause the pipeline to get stuck indefinitely
https://github.com/apache/beam/issues/29099 [Bug]: FnAPI Java SDK Harness 
doesn't update user counters in OnTimer callback functions
https://github.com/apache/beam/issues/28760 [Bug]: EFO Kinesis IO reader 
provided by apache beam does not pick the event time for watermarking
https://github.com/apache/beam/issues/28383 [Failing Test]: 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest.testMaxThreadMetric
https://github.com/apache/beam/issues/28326 Bug: 
apache_beam.io.gcp.pubsublite.ReadFromPubSubLite not working
https://github.com/apache/beam/issues/27892 [Bug]: ignoreUnknownValues not 
working when using CreateDisposition.CREATE_IF_NEEDED 
https://github.com/apache/beam/issues/27616 [Bug]: Unable to use 
applyRowMutations() in bigquery IO apache beam java
https://github.com/apache/beam/issues/27486 [Bug]: Read from datastore with 
inequality filters
https://github.com/apache/beam/issues/27314 [Failing Test]: 
bigquery.StorageApiSinkCreateIfNeededIT.testCreateManyTables[1]
https://github.com/apache/beam/issues/27238 [Bug]: Window trigger has lag when 
using Kafka and GroupByKey on Dataflow Runner
https://github.com/apache/beam/issues/26911 [Bug]: UNNEST ARRAY with a nested 
ROW (described below)
https://github.com/apache/beam/issues/26343 [Bug]: 
apache_beam.io.gcp.bigquery_read_it_test.ReadAllBQTests.test_read_queries is 
flaky
https://github.com/apache/beam/issues/26329 [Bug]: BigQuerySourceBase does not 
propagate a Coder to AvroSource
https://github.com/apache/beam/issues/26041 [Bug]: Unable to create 
exactly-once Flink pipeline with stream source and file sink
https://github.com/apache/beam/issues/24776 [Bug]: Race condition in Python SDK 
Harness ProcessBundleProgress
https://github.com/apache/beam/issues/24313