Jenkins down

[Yi Hu via dev]

Dear Beam developers,

There is currently an outage for Beam repo's Jenkins test server (
https://github.com/apache/beam/issues/27142 and
https://issues.apache.org/jira/browse/INFRA-24703) some tests shows success
status but actually not run anything. Until it get resolved I propose a
code freeze to avoid untested code get merged.

Best,
Yi

-- 

Yi Hu, (he/him/his)

Software Engineer

[PROPOSAL] Preparing for 2.49.0 Release

[Yi Hu via dev]

Hey Beam community,

The next release (2.49.0) branch cut is scheduled on June 28th, 2023,
according to
the release calendar [1].

I volunteer to perform this release. My plan is to cut the branch on that
date, and cherrypick release-blocking fixes afterwards, if any.

Please help me make sure the release goes smoothly by:
- Making sure that any unresolved release blocking issues for 2.49.0 should
have their "Milestone" marked as "2.49.0 Release" as soon as possible.
- Reviewing the current release blockers [2] and remove the Milestone if
they don't meet the criteria at [3].

Let me know if you have any comments/objections/questions.

Thanks,

Yi

[1]
https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
[2] https://github.com/apache/beam/milestone/13
[3] https://beam.apache.org/contribute/release-blocking/

-- 

Yi Hu, (he/him/his)

Software Engineer

Re: Asgarde: Error Handling for Beam?

[Kerry Donny-Clark via dev]

This looks like an excellent contribution. I can easily understand the
motivation, and I think Beam would benefit from a higher level abstraction
for error handling.
Kerry

On Wed, Jun 14, 2023, 6:31 PM Austin Bennett  wrote:

> Hi Beam Devs,
>
> @Mazlum  was
> suggested to consider donating Asgarde
>  to Beam for Java/Kotlin error
> handling to Beam [ see:
> https://2022.beamsummit.org/sessions/error-handling-asgarde/ for last
> year's Beam Summit talk ], he is also the author of Pasgard
> e [ for Python ] and Milgard [ for
> a simplified Kotlin API ].
>
> Would Asgarde be a good contribution, something the Beam community would
> be willing to accept?  I imagine we might want it to live at
> github.com/apache/beam-asgarde ?  Or perhaps there is a good place in
> github.com/apache/beam ??
>
> Especially once/if officially part of Beam, I imagine we'd add follow-up
> items like getting onto the website/docs, and related.
>
> Cheers,
> Austin
>
>
> P.S.  This might warrant separate/additional conversations for his other
> libraries, but let's focus any discussion on Asgarde for now?
>

Beam High Priority Issue Report (37)

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P1 Issues:

https://github.com/apache/beam/issues/27019 [Failing Test]: Azure Integration 
test is failing in python 3.7 PostCommit
https://github.com/apache/beam/issues/26981 [Bug]: Getting an error related to 
SchemaCoder after upgrading to 2.48
https://github.com/apache/beam/issues/26969 [Failing Test]: Python PostCommit 
is failing due to exceeded rate limits
https://github.com/apache/beam/issues/26911 [Bug]: UNNEST ARRAY with a nested 
ROW (described below)
https://github.com/apache/beam/issues/26547 [Failing Test]: 
beam_PostCommit_Java_DataflowV2
https://github.com/apache/beam/issues/26354 [Bug]: BigQueryIO direct read not 
reading all rows when set --setEnableBundling=true
https://github.com/apache/beam/issues/26343 [Bug]: 
apache_beam.io.gcp.bigquery_read_it_test.ReadAllBQTests.test_read_queries is 
flaky
https://github.com/apache/beam/issues/26329 [Bug]: BigQuerySourceBase does not 
propagate a Coder to AvroSource
https://github.com/apache/beam/issues/26272 [Failing Test]: Python 3.7 
postcommit is red
https://github.com/apache/beam/issues/26041 [Bug]: Unable to create 
exactly-once Flink pipeline with stream source and file sink
https://github.com/apache/beam/issues/25975 [Bug]: Reducing parallelism in 
FlinkRunner leads to a data loss
https://github.com/apache/beam/issues/24776 [Bug]: Race condition in Python SDK 
Harness ProcessBundleProgress
https://github.com/apache/beam/issues/24389 [Failing Test]: 
HadoopFormatIOElasticTest.classMethod ExceptionInInitializerError 
ContainerFetchException
https://github.com/apache/beam/issues/24313 [Flaky]: 
apache_beam/runners/portability/portable_runner_test.py::PortableRunnerTestWithSubprocesses::test_pardo_state_with_custom_key_coder
https://github.com/apache/beam/issues/23944  beam_PreCommit_Python_Cron 
regularily failing - test_pardo_large_input flaky
https://github.com/apache/beam/issues/23709 [Flake]: Spark batch flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
https://github.com/apache/beam/issues/22913 [Bug]: 
beam_PostCommit_Java_ValidatesRunner_Flink is flakes in 
org.apache.beam.sdk.transforms.GroupByKeyTest$BasicTests.testAfterProcessingTimeContinuationTriggerUsingState
https://github.com/apache/beam/issues/22605 [Bug]: Beam Python failure for 
dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest.test_metrics_it
https://github.com/apache/beam/issues/21714 
PulsarIOTest.testReadFromSimpleTopic is very flaky
https://github.com/apache/beam/issues/21708 beam_PostCommit_Java_DataflowV2, 
testBigQueryStorageWrite30MProto failing consistently
https://github.com/apache/beam/issues/21706 Flaky timeout in github Python unit 
test action 
StatefulDoFnOnDirectRunnerTest.test_dynamic_timer_clear_then_set_timer
https://github.com/apache/beam/issues/21643 FnRunnerTest with non-trivial 
(order 1000 elements) numpy input flakes in non-cython environment
https://github.com/apache/beam/issues/21476 WriteToBigQuery Dynamic table 
destinations returns wrong tableId
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: 
Connection refused
https://github.com/apache/beam/issues/21424 Java VR (Dataflow, V2, Streaming) 
failing: ParDoTest$TimestampTests/OnWindowExpirationTests
https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not 
follow spec
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit 
data at GC time
https://github.com/apache/beam/issues/21121 
apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it
 flakey
https://github.com/apache/beam/issues/21104 Flaky: 
apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers
https://github.com/apache/beam/issues/20976 
apache_beam.runners.portability.flink_runner_test.FlinkRunnerTestOptimized.test_flink_metrics
 is flaky
https://github.com/apache/beam/issues/20108 Python direct runner doesn't emit 
empty pane when it should
https://github.com/apache/beam/issues/19814 Flink streaming flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
https://github.com/apache/beam/issues/19465 Explore possibilities to lower 
in-use IP address quota footprint.


P1 Issues with no update in the last week:

https://github.com/apache/beam/issues/26902 [Bug]: Images built not saved in 
the local image store
https://github.com/apache/beam/issues/26723 [Failing Test]: Tour of Beam 
Frontend Test suite is perma-red on master
https://github.com/apache/beam/issues/23525 [Bug]: Default PubsubMessage coder 
will drop message id and orderingKey