Re: Fun with WebAssembly transforms

[Chamikara Jayalath via dev]

+1 and this is exactly what I suggested as well. Python Dataframe,
RunInference, Python Map are available via x-lang for Java already [1] and
all three need/use simple UDFs to customize operation. There is some logic
that needs to be added to use Python transforms from Go SDK. As you
suggested there are many Java x-lang transforms that can use simple UDF
support as well. Either language combination should work to implement a
first proof of concept for WASI support while also addressing an existing
limitation.

Thanks,
Cham

[1]
https://github.com/apache/beam/tree/master/sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/extensions/python/transforms

On Wed, Jul 13, 2022 at 8:26 PM Kenneth Knowles  wrote:

> I agree with Luke. Targeting little helper UDFs that go along with IOs are
> actually a major feature gap for xlang - like timestamp extractors that
> have to parse particular data formats. This could be a very useful place to
> try out the design options. I think we can simplify the problem by
> insisting that they are pure functions that do not access state or side
> inputs.
>
> On Wed, Jul 13, 2022 at 7:52 PM Luke Cwik via dev 
> wrote:
>
>> I think an easier target would be to support things like
>> DynamicDestinations for Java IO connectors that are exposed as XLang for
>> Go/Python .
>>
>> This is because Go/Python  have good
>> transpiling support to WebAssembly and we already exposed several Java IO
>> XLang connectors already so its about plumbing one more thing through for
>> these IO connectors.
>>
>> What interface should we expect for UDFs / UDAFs and should they be
>> purpose oriented or should we do something like we did for portability
>> where we have a graph of transforms that we feed arbitrary data in/out
>> from. The latter would have the benefit of allowing the runner to embed the
>> language execution directly within the runner and would pay the Wasm
>> communication tax instead of the gRPC communication tax. If we do the
>> former we still have the same issues where we have to be able to have a
>> type system to pass information between the host system and the transpiled
>> WebAssembly code that wraps the users UDF/UDAF and what if the UDF wants
>> access to side inputs or user state ...
>>
>> On Wed, Jul 13, 2022 at 4:09 PM Chamikara Jayalath 
>> wrote:
>>
>>>
>>>
>>> On Wed, Jul 13, 2022 at 9:31 AM Luke Cwik  wrote:
>>>
 First we'll want to choose whether we want to target Wasm, WASI or Wagi.

>>>
>>> These terms are defined here
>>> 
>>> if anybody is confused as I am :)
>>>
>>>
 WASI adds a lot of simple things like access to a clock, random number
 generator, ... that would expand the scope of what transpiled code can do.
 It is debatable whether we'll want the power to run the transpiled code as
 a microservice. Using UDFs for XLang and UDFs and UDAFs for SQL as our
 expected use cases seem to make WASI the best choice. The issue is in the
 details as there is a hodgepodge of what language runtimes support and what
 are the limits of transpiling from a language to WebAssembly.

>>>
>>> Agree that WASI seems like a good target since it gives access to
>>> additional system resources/tooling.
>>>
>>>

 Assuming WASI then it breaks down to these two aspects:
 1) Does the host language have a runtime?
 Java: https://github.com/wasmerio/wasmer-java
 Python: https://github.com/wasmerio/wasmer-python
 Go: https://github.com/wasmerio/wasmer-go

 2) How good is compilation from source language to WebAssembly
 ?
 Java (very limited):
 Issues with garbage collection and the need to transpile/replace much
 of the VM's capabilities plus the large standard library that everyone uses
 causes a lot of challenges.
 JWebAssembly can do simple things like basic classes, strings, method
 calls. Should be able to compile trivial lambdas to Wasm. There are other
 choices but to my knowledge all are very limited.

>>>
>>> That's unfortunate. But hopefully Java support will be implemented soon ?
>>>
>>>

 Python  (quite good):
 Features CPython Emscripten browser CPython Emscripten node Pyodide
 subprocess (fork, exec) no no no
 threads no YES WIP
 file system no (only MEMFS) YES (Node raw FS) YES (IDB, Node, …)
 shared extension modules WIP WIP YES
 PyPI packages no no YES
 sockets ? ? ?
 urllib, asyncio no no WebAPI fetch / WebSocket
 signals no WIP YES

 Go (excellent): Native support in go compiler

>>>
>>> Great. Could executing Go UDFs in Python x-lang transforms (for example,
>>> Dataframe, RunInference, Python Map) be a 

Re: Fun with WebAssembly transforms

[Kenneth Knowles]

I agree with Luke. Targeting little helper UDFs that go along with IOs are
actually a major feature gap for xlang - like timestamp extractors that
have to parse particular data formats. This could be a very useful place to
try out the design options. I think we can simplify the problem by
insisting that they are pure functions that do not access state or side
inputs.

On Wed, Jul 13, 2022 at 7:52 PM Luke Cwik via dev 
wrote:

> I think an easier target would be to support things like
> DynamicDestinations for Java IO connectors that are exposed as XLang for
> Go/Python .
>
> This is because Go/Python  have good
> transpiling support to WebAssembly and we already exposed several Java IO
> XLang connectors already so its about plumbing one more thing through for
> these IO connectors.
>
> What interface should we expect for UDFs / UDAFs and should they be
> purpose oriented or should we do something like we did for portability
> where we have a graph of transforms that we feed arbitrary data in/out
> from. The latter would have the benefit of allowing the runner to embed the
> language execution directly within the runner and would pay the Wasm
> communication tax instead of the gRPC communication tax. If we do the
> former we still have the same issues where we have to be able to have a
> type system to pass information between the host system and the transpiled
> WebAssembly code that wraps the users UDF/UDAF and what if the UDF wants
> access to side inputs or user state ...
>
> On Wed, Jul 13, 2022 at 4:09 PM Chamikara Jayalath 
> wrote:
>
>>
>>
>> On Wed, Jul 13, 2022 at 9:31 AM Luke Cwik  wrote:
>>
>>> First we'll want to choose whether we want to target Wasm, WASI or Wagi.
>>>
>>
>> These terms are defined here
>> 
>> if anybody is confused as I am :)
>>
>>
>>> WASI adds a lot of simple things like access to a clock, random number
>>> generator, ... that would expand the scope of what transpiled code can do.
>>> It is debatable whether we'll want the power to run the transpiled code as
>>> a microservice. Using UDFs for XLang and UDFs and UDAFs for SQL as our
>>> expected use cases seem to make WASI the best choice. The issue is in the
>>> details as there is a hodgepodge of what language runtimes support and what
>>> are the limits of transpiling from a language to WebAssembly.
>>>
>>
>> Agree that WASI seems like a good target since it gives access to
>> additional system resources/tooling.
>>
>>
>>>
>>> Assuming WASI then it breaks down to these two aspects:
>>> 1) Does the host language have a runtime?
>>> Java: https://github.com/wasmerio/wasmer-java
>>> Python: https://github.com/wasmerio/wasmer-python
>>> Go: https://github.com/wasmerio/wasmer-go
>>>
>>> 2) How good is compilation from source language to WebAssembly
>>> ?
>>> Java (very limited):
>>> Issues with garbage collection and the need to transpile/replace much of
>>> the VM's capabilities plus the large standard library that everyone uses
>>> causes a lot of challenges.
>>> JWebAssembly can do simple things like basic classes, strings, method
>>> calls. Should be able to compile trivial lambdas to Wasm. There are other
>>> choices but to my knowledge all are very limited.
>>>
>>
>> That's unfortunate. But hopefully Java support will be implemented soon ?
>>
>>
>>>
>>> Python  (quite good):
>>> Features CPython Emscripten browser CPython Emscripten node Pyodide
>>> subprocess (fork, exec) no no no
>>> threads no YES WIP
>>> file system no (only MEMFS) YES (Node raw FS) YES (IDB, Node, …)
>>> shared extension modules WIP WIP YES
>>> PyPI packages no no YES
>>> sockets ? ? ?
>>> urllib, asyncio no no WebAPI fetch / WebSocket
>>> signals no WIP YES
>>>
>>> Go (excellent): Native support in go compiler
>>>
>>
>> Great. Could executing Go UDFs in Python x-lang transforms (for example,
>> Dataframe, RunInference, Python Map) be a good first target ?
>>
>> Thanks,
>> Cham
>>
>>
>>>
>>> On Tue, Jul 12, 2022 at 5:51 PM Chamikara Jayalath via dev <
>>> dev@beam.apache.org> wrote:
>>>


 On Wed, Jun 29, 2022 at 9:31 AM Luke Cwik  wrote:

> I have had interest in integrating Wasm within Beam as well as I have
> had a lot of interest in improving language portability.
>
> Wasm has a lot of benefits over using docker containers to provide a
> place for code to execute. From experience implementing working on the
> Beam's portability layer and internal Flume knowledge:
> * encoding and decoding data is expensive, anything which ensures that
> in-memory representations for data being transferred from the host to the
> guest and back without transcoding/re-interpreting will be a big win.
> * reducing 

Re: Fun with WebAssembly transforms

[Luke Cwik via dev]

I think an easier target would be to support things like
DynamicDestinations for Java IO connectors that are exposed as XLang for
Go/Python.

This is because Go/Python have good transpiling support to WebAssembly and
we already exposed several Java IO XLang connectors already so its about
plumbing one more thing through for these IO connectors.

What interface should we expect for UDFs / UDAFs and should they be purpose
oriented or should we do something like we did for portability where we
have a graph of transforms that we feed arbitrary data in/out from. The
latter would have the benefit of allowing the runner to embed the language
execution directly within the runner and would pay the Wasm communication
tax instead of the gRPC communication tax. If we do the former we still
have the same issues where we have to be able to have a type system to pass
information between the host system and the transpiled WebAssembly code
that wraps the users UDF/UDAF and what if the UDF wants access to side
inputs or user state ...

On Wed, Jul 13, 2022 at 4:09 PM Chamikara Jayalath 
wrote:

>
>
> On Wed, Jul 13, 2022 at 9:31 AM Luke Cwik  wrote:
>
>> First we'll want to choose whether we want to target Wasm, WASI or Wagi.
>>
>
> These terms are defined here
> 
> if anybody is confused as I am :)
>
>
>> WASI adds a lot of simple things like access to a clock, random number
>> generator, ... that would expand the scope of what transpiled code can do.
>> It is debatable whether we'll want the power to run the transpiled code as
>> a microservice. Using UDFs for XLang and UDFs and UDAFs for SQL as our
>> expected use cases seem to make WASI the best choice. The issue is in the
>> details as there is a hodgepodge of what language runtimes support and what
>> are the limits of transpiling from a language to WebAssembly.
>>
>
> Agree that WASI seems like a good target since it gives access to
> additional system resources/tooling.
>
>
>>
>> Assuming WASI then it breaks down to these two aspects:
>> 1) Does the host language have a runtime?
>> Java: https://github.com/wasmerio/wasmer-java
>> Python: https://github.com/wasmerio/wasmer-python
>> Go: https://github.com/wasmerio/wasmer-go
>>
>> 2) How good is compilation from source language to WebAssembly
>> ?
>> Java (very limited):
>> Issues with garbage collection and the need to transpile/replace much of
>> the VM's capabilities plus the large standard library that everyone uses
>> causes a lot of challenges.
>> JWebAssembly can do simple things like basic classes, strings, method
>> calls. Should be able to compile trivial lambdas to Wasm. There are other
>> choices but to my knowledge all are very limited.
>>
>
> That's unfortunate. But hopefully Java support will be implemented soon ?
>
>
>>
>> Python  (quite good):
>> Features CPython Emscripten browser CPython Emscripten node Pyodide
>> subprocess (fork, exec) no no no
>> threads no YES WIP
>> file system no (only MEMFS) YES (Node raw FS) YES (IDB, Node, …)
>> shared extension modules WIP WIP YES
>> PyPI packages no no YES
>> sockets ? ? ?
>> urllib, asyncio no no WebAPI fetch / WebSocket
>> signals no WIP YES
>>
>> Go (excellent): Native support in go compiler
>>
>
> Great. Could executing Go UDFs in Python x-lang transforms (for example,
> Dataframe, RunInference, Python Map) be a good first target ?
>
> Thanks,
> Cham
>
>
>>
>> On Tue, Jul 12, 2022 at 5:51 PM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>>
>>>
>>> On Wed, Jun 29, 2022 at 9:31 AM Luke Cwik  wrote:
>>>
 I have had interest in integrating Wasm within Beam as well as I have
 had a lot of interest in improving language portability.

 Wasm has a lot of benefits over using docker containers to provide a
 place for code to execute. From experience implementing working on the
 Beam's portability layer and internal Flume knowledge:
 * encoding and decoding data is expensive, anything which ensures that
 in-memory representations for data being transferred from the host to the
 guest and back without transcoding/re-interpreting will be a big win.
 * reducing the amount of times we need to pass data between guest and
 host and back is important
   * fusing transforms reduces the number of data passing points
   * batching (row or columnar) data reduces the amount of times we need
 to pass data at each data passing point
 * there are enough complicated use cases (state & timers, large
 iterables, side inputs) where handling the trivial map/flatmap usecase will
 provide little value since it will prevent fusion

 I have been meaning to work on a prototype where we replace the current
 gRPC + docker path with one in which we use Wasm to 

Re: Fun with WebAssembly transforms

[Chamikara Jayalath via dev]

On Wed, Jul 13, 2022 at 9:31 AM Luke Cwik  wrote:

> First we'll want to choose whether we want to target Wasm, WASI or Wagi.
>

These terms are defined here

if anybody is confused as I am :)


> WASI adds a lot of simple things like access to a clock, random number
> generator, ... that would expand the scope of what transpiled code can do.
> It is debatable whether we'll want the power to run the transpiled code as
> a microservice. Using UDFs for XLang and UDFs and UDAFs for SQL as our
> expected use cases seem to make WASI the best choice. The issue is in the
> details as there is a hodgepodge of what language runtimes support and what
> are the limits of transpiling from a language to WebAssembly.
>

Agree that WASI seems like a good target since it gives access to
additional system resources/tooling.


>
> Assuming WASI then it breaks down to these two aspects:
> 1) Does the host language have a runtime?
> Java: https://github.com/wasmerio/wasmer-java
> Python: https://github.com/wasmerio/wasmer-python
> Go: https://github.com/wasmerio/wasmer-go
>
> 2) How good is compilation from source language to WebAssembly
> ?
> Java (very limited):
> Issues with garbage collection and the need to transpile/replace much of
> the VM's capabilities plus the large standard library that everyone uses
> causes a lot of challenges.
> JWebAssembly can do simple things like basic classes, strings, method
> calls. Should be able to compile trivial lambdas to Wasm. There are other
> choices but to my knowledge all are very limited.
>

That's unfortunate. But hopefully Java support will be implemented soon ?


>
> Python  (quite good):
> Features CPython Emscripten browser CPython Emscripten node Pyodide
> subprocess (fork, exec) no no no
> threads no YES WIP
> file system no (only MEMFS) YES (Node raw FS) YES (IDB, Node, …)
> shared extension modules WIP WIP YES
> PyPI packages no no YES
> sockets ? ? ?
> urllib, asyncio no no WebAPI fetch / WebSocket
> signals no WIP YES
>
> Go (excellent): Native support in go compiler
>

Great. Could executing Go UDFs in Python x-lang transforms (for example,
Dataframe, RunInference, Python Map) be a good first target ?

Thanks,
Cham


>
> On Tue, Jul 12, 2022 at 5:51 PM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>>
>>
>> On Wed, Jun 29, 2022 at 9:31 AM Luke Cwik  wrote:
>>
>>> I have had interest in integrating Wasm within Beam as well as I have
>>> had a lot of interest in improving language portability.
>>>
>>> Wasm has a lot of benefits over using docker containers to provide a
>>> place for code to execute. From experience implementing working on the
>>> Beam's portability layer and internal Flume knowledge:
>>> * encoding and decoding data is expensive, anything which ensures that
>>> in-memory representations for data being transferred from the host to the
>>> guest and back without transcoding/re-interpreting will be a big win.
>>> * reducing the amount of times we need to pass data between guest and
>>> host and back is important
>>>   * fusing transforms reduces the number of data passing points
>>>   * batching (row or columnar) data reduces the amount of times we need
>>> to pass data at each data passing point
>>> * there are enough complicated use cases (state & timers, large
>>> iterables, side inputs) where handling the trivial map/flatmap usecase will
>>> provide little value since it will prevent fusion
>>>
>>> I have been meaning to work on a prototype where we replace the current
>>> gRPC + docker path with one in which we use Wasm to execute a fused graph
>>> re-using large parts of the existing code base written to support
>>> portability.
>>>
>>
>> This sounds very interesting. Probably using Wasm to implement proper UDF
>> support for x-lang (for example, executing Python timestamp/watermark
>> functions provided through the Kafka Python x-lang wrapper on the Java
>> Kafka transform) will be a good first target ? My main question for this at
>> this point is whether Wasm has adequate support for existing SDKs that use
>> x-lang to implement this in a useful way.
>>
>> Thanks,
>> Cham
>>
>>
>>>
>>>
>>> On Fri, Jun 17, 2022 at 2:19 PM Brian Hulette 
>>> wrote:
>>>
 Re: Arrow - it's long been my dream to use Arrow for interchange in
 Beam [1]. I'm trying to move us in that direction with
 https://s.apache.org/batched-dofns (arrow is discussed briefly in the
 Future Work section). This gives the Python SDK a concept of batches of
 logical elements. My goal is Beam schemas + batches of logical elements ->
 Arrow RecordBatches.

 The Batched DoFn infrastructure is stable as of the 2.40.0 release cut
 and I'm currently working on adding what I'm calling a "BatchConverter" [2]
 

[PROPOSAL] Preparing for 2.41.0 Release

[Kiley Sok via dev]

Hi all,

The next (2.41.0) release branch cut is scheduled for July 27th, according
to
the release calendar [1].

I would like to volunteer myself to do this release. My plan is to cut the
branch on that date, and cherrypick release-blocking fixes afterwards, if
any.

Please help me make sure the release goes smoothly by:
- Making sure that any unresolved release blocking issues for 2.41.0 should
have their "Milestone" marked as "2.41.0" as soon as possible.
- Reviewing the current release blockers [2] and remove the Milestone if
they don't meet the criteria at [3].

Let me know if you have any comments/objections/questions.

Thanks,
Kiley

[1]
https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
[2] https://github.com/apache/beam/milestone/3
[3] https://beam.apache.org/contribute/release-blocking/

Re: Fun with WebAssembly transforms

[Luke Cwik via dev]

First we'll want to choose whether we want to target Wasm, WASI or Wagi.
WASI adds a lot of simple things like access to a clock, random number
generator, ... that would expand the scope of what transpiled code can do.
It is debatable whether we'll want the power to run the transpiled code as
a microservice. Using UDFs for XLang and UDFs and UDAFs for SQL as our
expected use cases seem to make WASI the best choice. The issue is in the
details as there is a hodgepodge of what language runtimes support and what
are the limits of transpiling from a language to WebAssembly.

Assuming WASI then it breaks down to these two aspects:
1) Does the host language have a runtime?
Java: https://github.com/wasmerio/wasmer-java
Python: https://github.com/wasmerio/wasmer-python
Go: https://github.com/wasmerio/wasmer-go

2) How good is compilation from source language to WebAssembly
?
Java (very limited):
Issues with garbage collection and the need to transpile/replace much of
the VM's capabilities plus the large standard library that everyone uses
causes a lot of challenges.
JWebAssembly can do simple things like basic classes, strings, method
calls. Should be able to compile trivial lambdas to Wasm. There are other
choices but to my knowledge all are very limited.

Python  (quite good):
Features CPython Emscripten browser CPython Emscripten node Pyodide
subprocess (fork, exec) no no no
threads no YES WIP
file system no (only MEMFS) YES (Node raw FS) YES (IDB, Node, …)
shared extension modules WIP WIP YES
PyPI packages no no YES
sockets ? ? ?
urllib, asyncio no no WebAPI fetch / WebSocket
signals no WIP YES

Go (excellent): Native support in go compiler

On Tue, Jul 12, 2022 at 5:51 PM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

>
>
> On Wed, Jun 29, 2022 at 9:31 AM Luke Cwik  wrote:
>
>> I have had interest in integrating Wasm within Beam as well as I have had
>> a lot of interest in improving language portability.
>>
>> Wasm has a lot of benefits over using docker containers to provide a
>> place for code to execute. From experience implementing working on the
>> Beam's portability layer and internal Flume knowledge:
>> * encoding and decoding data is expensive, anything which ensures that
>> in-memory representations for data being transferred from the host to the
>> guest and back without transcoding/re-interpreting will be a big win.
>> * reducing the amount of times we need to pass data between guest and
>> host and back is important
>>   * fusing transforms reduces the number of data passing points
>>   * batching (row or columnar) data reduces the amount of times we need
>> to pass data at each data passing point
>> * there are enough complicated use cases (state & timers, large
>> iterables, side inputs) where handling the trivial map/flatmap usecase will
>> provide little value since it will prevent fusion
>>
>> I have been meaning to work on a prototype where we replace the current
>> gRPC + docker path with one in which we use Wasm to execute a fused graph
>> re-using large parts of the existing code base written to support
>> portability.
>>
>
> This sounds very interesting. Probably using Wasm to implement proper UDF
> support for x-lang (for example, executing Python timestamp/watermark
> functions provided through the Kafka Python x-lang wrapper on the Java
> Kafka transform) will be a good first target ? My main question for this at
> this point is whether Wasm has adequate support for existing SDKs that use
> x-lang to implement this in a useful way.
>
> Thanks,
> Cham
>
>
>>
>>
>> On Fri, Jun 17, 2022 at 2:19 PM Brian Hulette 
>> wrote:
>>
>>> Re: Arrow - it's long been my dream to use Arrow for interchange in Beam
>>> [1]. I'm trying to move us in that direction with
>>> https://s.apache.org/batched-dofns (arrow is discussed briefly in the
>>> Future Work section). This gives the Python SDK a concept of batches of
>>> logical elements. My goal is Beam schemas + batches of logical elements ->
>>> Arrow RecordBatches.
>>>
>>> The Batched DoFn infrastructure is stable as of the 2.40.0 release cut
>>> and I'm currently working on adding what I'm calling a "BatchConverter" [2]
>>> for Beam Rows -> Arrow RecordBatch. Once that's done it could be
>>> interesting to experiment with a "WasmDoFn" that uses Arrow for interchange.
>>>
>>> Brian
>>>
>>> [1]
>>> https://docs.google.com/presentation/d/1D9vigwYTCuAuz_CO8nex3GK3h873acmQJE5Ui8TFsDY/edit#slide=id.g608e662464_0_160
>>> [2]
>>> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/batch.py
>>>
>>>
>>> On Thu, Jun 16, 2022 at 10:55 AM Sean Jensen-Grey 
>>> wrote:
>>>
 Interesting.

 Robert, I was just served an ad for Redpanda when I searched for
 "golang wasm" :)

 The storage and execution grid systems are all embracing wasm in some
 way.

 https://redpanda.com/
 https://www.fluvio.io/

[CDAP IO] CDAP connector code reviews

[Elizaveta Lomteva]

Hi community,


Our team is working on the new CdapIO connector implementation and we prepared 
PR with HasOffset interface [1] for 
the SparkReceiverIO package.

We would appreciate it very much if you could review this PR and leave comments.

Thank you for your attention to it,
Elizaveta

[1] HasOffset interface was implemented 
https://github.com/apache/beam/pull/22193

Beam High Priority Issue Report

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P0 Issues:

https://github.com/apache/beam/issues/22204 [Bug]: Python builds fail with 
google-api-core dep version cannot be satisfied


Unassigned P1 Issues:

https://github.com/apache/beam/issues/22217 Trying to connect to Kerberos 
enabled HDFS cluster from Beam facing issues Want to Read from HDFS file and 
write can someone ping a sample working code  
https://github.com/apache/beam/issues/22188 BigQuery Storage API sink sometimes 
gets stuck outputting to an invalid timestamp
https://github.com/apache/beam/issues/21935 [Bug]: Reject illformed GBK Coders
https://github.com/apache/beam/issues/21794 Dataflow runner creates a new timer 
whenever the output timestamp is change
https://github.com/apache/beam/issues/21713 404s in BigQueryIO don't get output 
to Failed Inserts PCollection
https://github.com/apache/beam/issues/21704 beam_PostCommit_Java_DataflowV2 
failures parent bug
https://github.com/apache/beam/issues/21703 pubsublite.ReadWriteIT failing in 
beam_PostCommit_Java_DataflowV1 and V2
https://github.com/apache/beam/issues/21702 SpannerWriteIT failing in beam 
PostCommit Java V1
https://github.com/apache/beam/issues/21701 beam_PostCommit_Java_DataflowV1 
failing with a variety of flakes and errors
https://github.com/apache/beam/issues/21700 
--dataflowServiceOptions=use_runner_v2 is broken
https://github.com/apache/beam/issues/21696 Flink Tests failure :  
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.beam.runners.core.construction.SerializablePipelineOptions 
https://github.com/apache/beam/issues/21695 DataflowPipelineResult does not 
raise exception for unsuccessful states.
https://github.com/apache/beam/issues/21694 BigQuery Storage API insert with 
writeResult retry and write to error table
https://github.com/apache/beam/issues/21480 flake: 
FlinkRunnerTest.testEnsureStdoutStdErrIsRestored
https://github.com/apache/beam/issues/21473 PVR_Spark2_Streaming perma-red
https://github.com/apache/beam/issues/21472 Dataflow streaming tests failing 
new AfterSynchronizedProcessingTime test
https://github.com/apache/beam/issues/21471 Flakes: Failed to load cache entry
https://github.com/apache/beam/issues/21470 Test flake: test_split_half_sdf
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: 
Connection refused
https://github.com/apache/beam/issues/21468 
beam_PostCommit_Python_Examples_Dataflow failing
https://github.com/apache/beam/issues/21467 GBK and CoGBK streaming Java load 
tests failing
https://github.com/apache/beam/issues/21465 Kafka commit offset drop data on 
failure for runners that have non-checkpointing shuffle
https://github.com/apache/beam/issues/21463 NPE in Flink Portable 
ValidatesRunner streaming suite
https://github.com/apache/beam/issues/21462 Flake in 
org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadObject: Address already in use
https://github.com/apache/beam/issues/21271 pubsublite.ReadWriteIT flaky in 
beam_PostCommit_Java_DataflowV2  
https://github.com/apache/beam/issues/21270 
org.apache.beam.sdk.transforms.CombineTest$WindowingTests.testWindowedCombineGloballyAsSingletonView
 flaky on Dataflow Runner V2
https://github.com/apache/beam/issues/21268 Race between member variable being 
accessed due to leaking uninitialized state via OutboundObserverFactory
https://github.com/apache/beam/issues/21267 WriteToBigQuery submits a duplicate 
BQ load job if a 503 error code is returned from googleapi
https://github.com/apache/beam/issues/21266 
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
 is flaky in Java ValidatesRunner Flink suite.
https://github.com/apache/beam/issues/21265 
apache_beam.runners.portability.fn_api_runner.translations_test.TranslationsTest.test_run_packable_combine_globally
 'apache_beam.coders.coder_impl._AbstractIterable' object is not reversible
https://github.com/apache/beam/issues/21264 beam_PostCommit_Python36 - 
CrossLanguageSpannerIOTest - flakey failing
https://github.com/apache/beam/issues/21263 (Broken Pipe induced) Bricked 
Dataflow Pipeline 
https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not 
follow spec
https://github.com/apache/beam/issues/21261 
org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
 is flaky
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit 
data at GC time
https://github.com/apache/beam/issues/21257 Either Create or DirectRunner fails 
to produce all elements to the following transform
https://github.com/apache/beam/issues/21123 Multiple jobs running on Flink 
session cluster reuse the persistent Python environment.
https://github.com/apache/beam/issues/21121