Re: Contribution of Asgarde: Error Handling for Beam?

[Reuven Lax via dev]

Let's be careful about whether these tests are included in our presubmits.
Contrib code with flaky tests has been a major pain point in the past.

On Sat, Sep 2, 2023 at 12:02 PM Austin Bennett  wrote:

> Wanting us to not miss this. @Mazlum TOSUN  is
> happy to donate Asgarde to our project.
>
> It looks like he'd need a SGA and CCLA [ 1 ] on file; anything else?
>
> I recalled the donation of Euphoria [ 2 ] , so I looked at those threads [
> 3 ]  for insights into the process.  It didn't look like there was a needed
> VOTE, so mostly a matter of ensuring necessary signatures, and ideally some
> sort of consensus [ or non-opposition ] to the donation.
>
>
> [ 1 ] https://www.apache.org/licenses/contributor-agreements.html
> [ 2 ] https://beam.apache.org/documentation/sdks/java/euphoria/
> [ 3 ] https://lists.apache.org/thread/xzlx4rm2tvc36mmwvhyvtdvsw7bnjscp
>
>
>
> On Thu, Jun 15, 2023 at 7:05 AM Kerry Donny-Clark via dev <
> dev@beam.apache.org> wrote:
>
>> This looks like an excellent contribution. I can easily understand the
>> motivation, and I think Beam would benefit from a higher level abstraction
>> for error handling.
>> Kerry
>>
>> On Wed, Jun 14, 2023, 6:31 PM Austin Bennett  wrote:
>>
>>> Hi Beam Devs,
>>>
>>> @Mazlum  was
>>> suggested to consider donating Asgarde
>>>  to Beam for Java/Kotlin error
>>> handling to Beam [ see:
>>> https://2022.beamsummit.org/sessions/error-handling-asgarde/ for last
>>> year's Beam Summit talk ], he is also the author of Pasgard
>>> e [ for Python ] and Milgard [
>>> for a simplified Kotlin API ].
>>>
>>> Would Asgarde be a good contribution, something the Beam community would
>>> be willing to accept?  I imagine we might want it to live at
>>> github.com/apache/beam-asgarde ?  Or perhaps there is a good place in
>>> github.com/apache/beam ??
>>>
>>> Especially once/if officially part of Beam, I imagine we'd add follow-up
>>> items like getting onto the website/docs, and related.
>>>
>>> Cheers,
>>> Austin
>>>
>>>
>>> P.S.  This might warrant separate/additional conversations for his other
>>> libraries, but let's focus any discussion on Asgarde for now?
>>>
>>

Beam High Priority Issue Report (41)

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P1 Issues:

https://github.com/apache/beam/issues/28168 [Bug]: BigQuery Storage Write API 
does not write with no complaint
https://github.com/apache/beam/issues/28142 [Bug]: [Go SDK] Memory seems to be 
leaking on 2.49.0 with Dataflow
https://github.com/apache/beam/issues/28104 [Failing Test]: gradlew 
:sdks:python:test-suites:tox:pycommon:docs is failing
https://github.com/apache/beam/issues/27892 [Bug]: ignoreUnknownValues not 
working when using CreateDisposition.CREATE_IF_NEEDED 
https://github.com/apache/beam/issues/27648 [Bug]: Python SDFs (e.g. 
PeriodicImpulse) running in Flink and polling using tracker.defer_remainder 
have checkpoint size growing indefinitely 
https://github.com/apache/beam/issues/27616 [Bug]: Unable to use 
applyRowMutations() in bigquery IO apache beam java
https://github.com/apache/beam/issues/27486 [Bug]: Read from datastore with 
inequality filters
https://github.com/apache/beam/issues/27314 [Failing Test]: 
bigquery.StorageApiSinkCreateIfNeededIT.testCreateManyTables[1]
https://github.com/apache/beam/issues/27238 [Bug]: Window trigger has lag when 
using Kafka and GroupByKey on Dataflow Runner
https://github.com/apache/beam/issues/26981 [Bug]: Getting an error related to 
SchemaCoder after upgrading to 2.48
https://github.com/apache/beam/issues/26969 [Failing Test]: Python PostCommit 
is failing due to exceeded rate limits
https://github.com/apache/beam/issues/26911 [Bug]: UNNEST ARRAY with a nested 
ROW (described below)
https://github.com/apache/beam/issues/26354 [Bug]: BigQueryIO direct read not 
reading all rows when set --setEnableBundling=true
https://github.com/apache/beam/issues/26343 [Bug]: 
apache_beam.io.gcp.bigquery_read_it_test.ReadAllBQTests.test_read_queries is 
flaky
https://github.com/apache/beam/issues/26329 [Bug]: BigQuerySourceBase does not 
propagate a Coder to AvroSource
https://github.com/apache/beam/issues/26041 [Bug]: Unable to create 
exactly-once Flink pipeline with stream source and file sink
https://github.com/apache/beam/issues/25975 [Bug]: Reducing parallelism in 
FlinkRunner leads to a data loss
https://github.com/apache/beam/issues/24776 [Bug]: Race condition in Python SDK 
Harness ProcessBundleProgress
https://github.com/apache/beam/issues/24389 [Failing Test]: 
HadoopFormatIOElasticTest.classMethod ExceptionInInitializerError 
ContainerFetchException
https://github.com/apache/beam/issues/24313 [Flaky]: 
apache_beam/runners/portability/portable_runner_test.py::PortableRunnerTestWithSubprocesses::test_pardo_state_with_custom_key_coder
https://github.com/apache/beam/issues/23944  beam_PreCommit_Python_Cron 
regularily failing - test_pardo_large_input flaky
https://github.com/apache/beam/issues/23709 [Flake]: Spark batch flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
https://github.com/apache/beam/issues/23525 [Bug]: Default PubsubMessage coder 
will drop message id and orderingKey
https://github.com/apache/beam/issues/22913 [Bug]: 
beam_PostCommit_Java_ValidatesRunner_Flink is flakes in 
org.apache.beam.sdk.transforms.GroupByKeyTest$BasicTests.testAfterProcessingTimeContinuationTriggerUsingState
https://github.com/apache/beam/issues/22605 [Bug]: Beam Python failure for 
dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest.test_metrics_it
https://github.com/apache/beam/issues/21714 
PulsarIOTest.testReadFromSimpleTopic is very flaky
https://github.com/apache/beam/issues/21708 beam_PostCommit_Java_DataflowV2, 
testBigQueryStorageWrite30MProto failing consistently
https://github.com/apache/beam/issues/21706 Flaky timeout in github Python unit 
test action 
StatefulDoFnOnDirectRunnerTest.test_dynamic_timer_clear_then_set_timer
https://github.com/apache/beam/issues/21643 FnRunnerTest with non-trivial 
(order 1000 elements) numpy input flakes in non-cython environment
https://github.com/apache/beam/issues/21476 WriteToBigQuery Dynamic table 
destinations returns wrong tableId
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: 
Connection refused
https://github.com/apache/beam/issues/21424 Java VR (Dataflow, V2, Streaming) 
failing: ParDoTest$TimestampTests/OnWindowExpirationTests
https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not 
follow spec
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit 
data at GC time
https://github.com/apache/beam/issues/21121 
apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it
 flakey
https://github.com/apache/beam/issues/21104 Flaky: 
apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers
https://github.com/apache/bea