This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from e68a79c03f4 Validate circular reference for yaml (#33208)
add f4764174d12 Added support for SparkRunner streaming stateful
processing (#33267)
No new revisions were added by this update.
Summary of changes:
...beam_PostCommit_Java_ValidatesRunner_Spark.json | 1 +
...a_ValidatesRunner_SparkStructuredStreaming.json | 3 +-
...stCommit_Java_ValidatesRunner_Spark_Java11.json | 1 +
CHANGES.md | 3 +-
runners/spark/spark_runner.gradle | 2 +-
.../spark/coders/SparkRunnerKryoRegistrator.java | 2 +-
.../SparkGroupAlsoByWindowViaWindowSet.java | 44 +---
.../spark/stateful/SparkStateInternals.java | 18 +-
.../spark/stateful/SparkTimerInternals.java | 5 +-
.../runners/spark/stateful/StateAndTimers.java} | 26 +-
.../spark/translation/DoFnRunnerWithMetrics.java | 4 +-
.../spark/translation/SparkInputDataProcessor.java | 2 +-
.../spark/translation/SparkProcessContext.java | 4 +-
.../spark/translation/TranslationUtils.java | 50 ++++
.../translation/streaming/ParDoStateUpdateFn.java | 266 +++++++++++++++++++++
.../streaming/StatefulStreamingParDoEvaluator.java | 246 +++++++++++++++++++
.../streaming/StreamingTransformTranslator.java | 22 +-
.../translation/streaming/CreateStreamTest.java | 2 +-
.../StatefulStreamingParDoEvaluatorTest.java | 226 +++++++++++++++++
19 files changed, 860 insertions(+), 67 deletions(-)
copy
runners/{google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ActiveMessageMetadata.java
=>
spark/src/main/java/org/apache/beam/runners/spark/stateful/StateAndTimers.java}
(54%)
create mode 100644
runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/ParDoStateUpdateFn.java
create mode 100644
runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StatefulStreamingParDoEvaluator.java
create mode 100644
runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/StatefulStreamingParDoEvaluatorTest.java
