This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/go_modules/sdks/github.com/testcontainers/testcontainers-go-0.30.0
in repository https://gitbox.apache.org/repos/asf/beam.git
discard 0409ced3202 Bump github.com/testcontainers/testcontainers-go in /sdks
add 0437a8dcfcc Initial Iceberg connector
add fd08eb49f31 Fix up IcebergIO Write path
add a7a6515cf3f Add IcebergIO GitHub Action workflow
add 819e54c972c Merge pull request #30797: Initial Iceberg Sink
add e3ce3bca115 Update Python Dependencies (#30816)
add 54c2a7277b4 [Flink] Speed up file write in batch mode by using larger
bundle size (#30802)
add 76710e8e734 Implement LockFreeHistogram and use it for
PerWorkerHistograms (#30769)
add fcc37a0cc2c [yaml] remove Combine from yaml_experimental_features
(#30897)
add 282ff87726d Add another dependency restriction to unblock tests.
(#30907)
add 3f4b2561c58 [bug30870]: make consumer polling timeout configurable for
KafkaIO.Read (#30877)
add 06588745182 [yaml] disable javascript mapping for python >=3.12
(#30843)
add 89aeb187f2c Correct per-entry HashMap overhead in WindmillStateCache
(#30672)
add e6b7a6607aa Remove Bruno from reviewer pool (#30917)
add c2188643d27 [java-extension-sql] Beam Sql doesn't honor aliases
because of `BeamAggregateProjectMergeRule` optimization (#30902)
add b018c25167e Add snappy to 3.11 containers (#30918)
add c1761abc512 Adding support for high priority queries to xlang
transforms writing … (#30869)
add 48d80128f62 [yaml] add use-case example (#30896)
add 8ca15eff0cf Load and Stress Tests: Add workflows, improvements and
fixes (#30848)
add cc719214e44 [yaml] Add Beam YAML Blog (#30898)
add f021bb08991 Fix cron in github workflow file (#30929)
add ec5155ed26b Fix playground typo breaking rendering (#30930)
add 701670624b3 Remove comma that breaks formatting (#30931)
add 1dc3346705a Add gradle target and github workflow for cross-langauge
yaml tests. (#30874)
add 0e7b047c636 [yaml] Fix Beam YAML blog links (#30932)
add 153c7db71d5 Fixup iceberg GitHub Action workflow
add 983f4e8a499 Merge pull request #30935: Fixup iceberg GitHub Action
workflow
add 067ab8d70d6 Bump transformers from 4.36.0 to 4.38.0 in
/sdks/python/apache_beam/examples/inference/runinference_metrics (#30924)
add e1a3eafb6fe Add PubSubIO Stress test (#30886)
add a6903628e77 Update sqlalchemy requirement from >=1.3,<2.0 to
>=1.3,<3.0 in /sdks/python (#30919)
add 67b3e111ea9 Upgrade grpcio-tools. (#30912)
add e33dec69c7c Add config validation to kafka read schema transform
(#30625)
add cd368cbd402 Bump tar (#30928)
add 0d5d5670f10 Fix broken lint and improve lint rules. (#30940)
add 325c0b51ce2 Support FLOAT32 type in Spanner (#30893)
add 5b3dbf849ce Add important category (#30920)
add 883c339372d Add a cross language yaml test.
add b05a9c7593a Add parquet, avro, and text tests.
add 0c85f0d5bbb Add java mapping test.
add 7ed6b6d5b5a Add tests for windowing and combining in java and Python.
add 99df52cffea Reword docstring.
add dac2bd87b2c Merge pull request #30880 [YAML] Adds several tests
exercising the cross-language capabilities.
add 179103f7a64 [BEAM-30950] Disable failing test for now.
add ce364e128b2 Merge pull request #30951 [BEAM-30950] Disable failing
test for now.
add be7d84d1096 Update java-dependency-guide contributor-doc reflecting
gcpbomupgrader script (#30942)
add 65032cecfd8 Increase the timeout value for LeaderBoard example on
DataflowRunner (#30949)
add 1a0591ea797 Bump transformers (#30922)
add 631f6e7a832 Fix iceberg dependencies
add 36f32280a1b Refactor iceberg/beam conversions for clarity
add a169e6a2a1c Re-add Iceberg bounded source; test splitting
add 96dc16aedb7 Merge pull request #30805: Re-add iceberg bounded source;
test splitting
add 72a45e4fbd4 Fix sdist (#30956)
add a764632db32 Document how to trigger test suites. (#30953)
add cd253fd8922 Revert setuptools upperbound since buggy releases are
yanked. (#30958)
add cfe7be6a609 [runners-flink] Use ImpulseSourceFunction for streaming
Impulse (#30903)
add f071e9e3415 Merge pull request #30905: Do not reemit data from impulse
(#30903)
add f8e4330b6ba change grafana creds to use readonly account for
beammetrics cloudsql db (#30968)
add 285b20de1ce Restore pip logs also for requirements.txt case (#30936)
add a00e947f781 Upload PerWorkerMetrics every 30 second instead of every
10 seconds (#30795)
add 4b808b0d4f5 Fix build wheels due to setuptools changed naming (#30972)
add d680bf0a57d [runners-flink] Fix watermark emission for empty splits
(#29816)
add e119cd49cf9 Merge pull request #30969: [runners-flink] Fix watermark
emission for empty splits (#29816)
add 9fa45df9b06 Move logic to build a Labeled Metric Name to a util file.
(#30796)
add 2f3893259fa [runners-flink] Finalize checkpoint marks after successful
checkpoint (#29902)
add f41f3648c3e Merge pull request #30971: #29902 finalize checkpoints
after checkpoint
add eb7ad46fcb7 remove CLOUDSDK env variable , not needed here (#30970)
add fc7b955218c Reapply "[BEAM-30531] Automatically execute unbounded
pipelines in streaming mode. (#30533)" (#30706)
add 042284afd4e Add experiment for disabling auto-streaming.
add c165f8af134 Correct release notes.
add 63ebda005d0 Merge pull request #30959 Automatically execute unbounded
pipelines in streaming mode.
add fb6bfc3ef11 Skip generating docs for yaml/integration_tests.py.
(#30986)
add a44c4f19ce3 [#30870]: support consumer polling timeout in KafkaIO
expansion service (#30915)
add e59d313c3b1 Add a link to Grafana statistics for flaky workflows
(#30934)
add db585b7b462 replace clock.milliseconds with stopwatch (#30678)
add 1a26eada0b0 [Python] Allow lambda function in bigtable handler to
build a custom row key (#30974)
add a6f3ddf898c Improvements to GetWorkTimingInfosTracker when there is
clock skew between the worker and service. (#30990)
add 995d1bf8a5a Update the upper bound for google-auth-httplib2 to
'google-auth-httplib2>=0.1.0,<0.3.0' (#30962)
add 277b6c3dbe5 Update python transform catalog (#30788)
add f6322dae132 Fix ConcurrentModification exception possible in
DataflowExecutionStateSampler (#30993)
add 2c36cc0bf49 Move dependency updates to right after branch cut (#30939)
add 1e89f834c84 Folow up of recent GHA breakages (#30996)
add 1d8fdfa83ca Remove caching of global window side input param when
invoking. (#30991)
add ae597ccbf10 Add mechanism for skipping batching if your data is
prebatched (#30961)
add ec31847d934 Skip calling Compute in RemoveSafeDeltaCounterCell if
value exists (#30913)
add a62dfa732d4 [Flink] finalize checkpoint marks in the new Flink source
implementation (#30849) (#30987)
add 6bca71070e9 Add some metrics for CoGBK profiling. (#30979)
add 6366bd4c890 Disable profiling in cythonized code by default. (#30938)
add 8092932a2e1 [#30941]fix upgrade test due to missed config
ConsumerPollingTimeout (#30998)
add 9947260deba fix naming changes caused by setuptools (#31012)
add a26c5055ccc Downgrade google-api-core to mitigate #30927 (#31004)
add 27643432ea9 [yaml] backtick generated aliases on sql mappings (#30895)
add 6624d952bbd Cache parsed metrics in StreamingStepMetricsContainer
(#31005)
add e48a9351d2f add Yaml_Xlang_Direct PreCommit to README (#31009)
add deee0e9cc8c [flink] add support for Flink 1.17 (#29939)
add d70c25328ee Merge pull request #30197: Flink 1.17
add 38a8b5f7047 Create YAML Join Transform (#30734)
add a207e4e8bbe Revert "Docker - Remove Deprecated Types (#30826)" (#31010)
add 05b7146656a Add /job/cancel endpoint to prism web server. (#30825)
add 1e8a16d9eae Use base 2 exponential histograms in BigQuerySinkMetrics
(#31000)
add 7159beac55f Minor updates to StreamingInsertsMetrics (#31003)
add 5c6bc39a44d Moving to 2.57.0-SNAPSHOT on master branch.
add e4baee698d8 [runners-flink] remove 1.12 and 1.13 runners
add 3cfc160b512 Merge pull request #30988: Remove flink 1.12 1.13
add b134e72a068 [changes] Update CHANGES.md (#29939) (#31017)
add 3822f551dfc remove CLOUDSDK (#31016)
add d6eb576d037 Allow IFrames to youtube. (#31021)
add e3077b7460c add new user info to metrics Readme (#30989)
add 4f964bf05d5 Fix workflow param value for Grafana link (#31011)
add bcb40cf4e4a Change caching of global window inputs to be guarded by
experiment (#31013)
add 2eb1a756258 [Python] Clean doc related to write data in bigquery.py
(#30887)
add 04ff4bdd7fc Support BQ clustering with value provider (#30460)
add 61153bbda6a Update documentation of @SchemaFieldNumber (#30273)
(#30277)
add bb0b63cb940 Bump TPCDS test Flink version (#31041)
add 70e067e1fde fix url for content security (#31043)
add bb310e7e907 Change type for UnboundedReaderMaxReadTimeSec (#31037)
add b69e8c615af Updates Python Dev container used by Dataflow (#31029)
add 76c77cd28ae Fix typo in count_unique_words() (#31023)
add 3e52e3554a0 Add code change guide contributor-doc (#30879)
add d05196dea8a Upgrade the version of GRPC to pick up a fix for #30867
(#31044)
add fd1003034ff Add ScaleToGaussian TFTransform (#31026)
add 98b1f03125c Better and earlier error for missing service loader
metadata. (#31042)
add 40ec29ad289 fix website embedding issue (#31019)
add 31f1c7968d4 fix url for content security
add 93aa62c2900 Merge pull request #31058 from svetakvsundhar/broken_links
add 49da7a34d99 Bump golang.org/x/net from 0.22.0 to 0.23.0 in /sdks
(#31054)
add 516ac97659f fix examples on website (#31066)
add 1a5dc1c7fab Replace Source type for BigTable and Kafka stress tests
(#30999)
add 8dcbf967a1a [#30083][prism] Stabilize additional teststream cases.
(#31046)
add 2472695eb04 filter out outdated workflow runs (#31071)
add f8b81211049 Fix test failure due to locale sensitivity. (#31069)
add 718af0d7652 Don't use isolated builds when building an sdist for a
pipeline package supplied in --setup_file option. (#31070)
add 5a86b3fc922 Bump golang.org/x/net from 0.17.0 to 0.23.0 in
/learning/katas/go (#31050)
add 3dbbad98c80 Bump golang.org/x/net from 0.17.0 to 0.23.0 in
/.test-infra/mock-apis (#31051)
add a0dad088980 Bump golang.org/x/net from 0.22.0 to 0.24.0 in /sdks
(#31065)
add a840b8c9482 Bump github.com/testcontainers/testcontainers-go in /sdks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (0409ced3202)
\
N -- N -- N
refs/heads/dependabot/go_modules/sdks/github.com/testcontainers/testcontainers-go-0.30.0
(a840b8c9482)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 1 +
.github/REVIEWERS.yml | 2 -
...les_Direct.json => beam_PostCommit_Python.json} | 1 +
...tCommit_Python_ValidatesContainer_Dataflow.json | 3 +
...am_PostCommit_Python_ValidatesRunner_Flink.json | 0
...mit_Java_Kafka_IO_Direct.yml => IO_Iceberg.yml} | 35 +-
.github/workflows/README.md | 9 +-
...espace.yml => beam_LoadTests_Java_PubsubIO.yml} | 45 +-
.../workflows/beam_PostCommit_Java_Tpcds_Flink.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 4 +-
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 2 -
..._PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml | 2 -
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 2 -
.../beam_PreCommit_Java_Kafka_IO_Direct.yml | 2 +-
.github/workflows/beam_PreCommit_Whitespace.yml | 4 +-
...ow.yml => beam_PreCommit_Yaml_Xlang_Direct.yml} | 43 +-
...ce.yml => beam_StressTests_Java_BigQueryIO.yml} | 45 +-
...ce.yml => beam_StressTests_Java_BigTableIO.yml} | 45 +-
...Flink.yml => beam_StressTests_Java_KafkaIO.yml} | 79 +-
...pace.yml => beam_StressTests_Java_PubSubIO.yml} | 45 +-
...ace.yml => beam_StressTests_Java_SpannerIO.yml} | 45 +-
.github/workflows/build_wheels.yml | 6 +-
.github/workflows/dask_runner_tests.yml | 2 +-
.github/workflows/python_tests.yml | 2 +-
.github/workflows/update_python_dependencies.yml | 12 +-
.../base/v0.33.2/kafka-persistent.yaml | 14 +-
.../gke-internal-load-balanced/kustomization.yaml | 8 +-
.test-infra/metrics/README.md | 7 +
.../GA-Post-Commits_status_dashboard.json | 1007 +++++++++++++++++++-
.../grafana/provisioning/alerting/flaky_test.yaml | 2 +-
.../metrics/kubernetes/beamgrafana-deploy.yaml | 4 +-
.../github/github_runs_prefetcher/code/config.yaml | 96 +-
.../github/github_runs_prefetcher/code/main.py | 40 +-
.test-infra/mock-apis/go.mod | 6 +-
.test-infra/mock-apis/go.sum | 12 +-
.test-infra/tools/flaky_test_detection.py | 26 +-
CHANGES.md | 9 +-
contributor-docs/code-change-guide.md | 525 ++++++++++
contributor-docs/java-dependency-upgrades.md | 44 +-
contributor-docs/release-guide.md | 20 +-
.../get-started/try-apache-beam-yaml.ipynb | 6 +-
gradle.properties | 6 +-
it/google-cloud-platform/build.gradle | 86 +-
.../org/apache/beam/it/gcp/IOLoadTestBase.java | 14 +-
.../java/org/apache/beam/it/gcp/LoadTestBase.java | 32 +-
.../it/gcp/bigtable/BigtableResourceManager.java | 2 +-
.../apache/beam/it/gcp/bigquery/BigQueryIOST.java | 144 +--
.../apache/beam/it/gcp/bigtable/BigTableIOST.java | 51 +-
.../pubsub/{PubSubIOLT.java => PubSubIOST.java} | 182 ++--
.../pubsub/{PubSubIOLT.java => PubsubIOLT.java} | 144 ++-
.../apache/beam/it/gcp/spanner/SpannerIOST.java | 6 +-
it/kafka/build.gradle | 12 +-
.../java/org/apache/beam/it/kafka/KafkaIOST.java | 49 +-
learning/katas/go/go.mod | 2 +-
learning/katas/go/go.sum | 20 +-
.../Core Transforms/GroupByKey/GroupByKey/task.py | 2 +
playground/infrastructure/requirements.txt | 2 +-
release/build.gradle.kts | 2 +-
.../src/main/groovy/MobileGamingCommands.groovy | 2 +-
runners/flink/1.12/build.gradle | 33 -
.../streaming/AbstractStreamOperatorCompat.java | 94 --
.../io/source/compat/SplitEnumeratorCompat.java | 27 -
.../beam/runners/flink/RemoteMiniClusterImpl.java | 68 --
.../runners/flink/streaming/StreamSources.java | 72 --
.../streaming/io/source/SourceTestCompat.java | 75 --
runners/flink/1.13/build.gradle | 33 -
runners/flink/1.13/job-server/build.gradle | 31 -
runners/flink/1.14/build.gradle | 13 +-
.../translation/types/CoderTypeSerializer.java | 10 +-
.../streaming/ProcessingTimeCallbackCompat.java | 0
.../io/source/compat/SplitEnumeratorCompat.java | 0
.../streaming/io/source/compat/package-info.java | 0
.../beam/runners/flink/MiniClusterCompat.java | 0
.../translation/types/CoderTypeSerializerTest.java | 2 +-
runners/flink/1.15/build.gradle | 13 +-
runners/flink/1.16/build.gradle | 13 +-
.../job-server-container => 1.17}/build.gradle | 7 +-
.../job-server-container/build.gradle | 0
.../flink/{1.12 => 1.17}/job-server/build.gradle | 2 +-
.../translation/types/CoderTypeSerializer.java | 31 +-
runners/flink/flink_runner.gradle | 35 +-
.../beam/runners/flink/FlinkPipelineOptions.java | 42 +-
.../flink/FlinkStreamingTransformTranslators.java | 27 +-
.../runners/flink/FlinkTransformOverrides.java | 23 +-
.../types/UnversionedTypeSerializerSnapshot.java | 86 ++
.../wrappers/streaming/io/source/FlinkSource.java | 18 +-
.../streaming/io/source/FlinkSourceReaderBase.java | 68 +-
.../streaming/io/source/FlinkSourceSplit.java | 37 +-
.../io/source/FlinkSourceSplitEnumerator.java | 6 +-
.../io/source/bounded/FlinkBoundedSource.java | 1 +
.../source/bounded/FlinkBoundedSourceReader.java | 39 +-
.../io/source/impulse/BeamImpulseSource.java | 34 +-
.../unbounded/FlinkUnboundedSourceReader.java | 90 +-
.../runners/flink/FlinkPipelineOptionsTest.java | 12 +-
.../UnversionedTypeSerializerSnapshotTest.java | 52 +
.../bounded/FlinkBoundedSourceReaderTest.java | 1 +
.../unbounded/FlinkUnboundedSourceReaderTest.java | 46 +
.../options/DataflowPipelineDebugOptions.java | 9 +-
.../options/DataflowStreamingPipelineOptions.java | 10 +
.../dataflow/worker/ActiveMessageMetadata.java | 7 +-
.../worker/DataflowBatchWorkerHarness.java | 3 +
.../dataflow/worker/DataflowExecutionContext.java | 9 +-
.../worker/DataflowExecutionStateSampler.java | 14 +-
.../runners/dataflow/worker/LockFreeHistogram.java | 223 +++++
.../MetricsToPerStepNamespaceMetricsConverter.java | 74 +-
.../worker/RemoveSafeDeltaCounterCell.java | 15 +-
.../dataflow/worker/StreamingDataflowWorker.java | 14 +-
.../worker/StreamingStepMetricsContainer.java | 40 +-
.../dataflow/worker/WorkerCustomSources.java | 4 +-
.../runners/dataflow/worker/streaming/Work.java | 2 +-
.../harness/StreamingWorkerStatusReporter.java | 72 +-
.../client/grpc/GetWorkTimingInfosTracker.java | 25 +-
.../worker/windmill/state/WindmillStateCache.java | 3 +-
.../worker/DataflowExecutionContextTest.java | 7 +-
.../worker/DataflowExecutionStateSamplerTest.java | 8 +-
.../dataflow/worker/LockFreeHistogramTest.java | 185 ++++
...ricsToPerStepNamespaceMetricsConverterTest.java | 120 ++-
.../worker/StreamingStepMetricsContainerTest.java | 54 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 7 +-
.../harness/StreamingWorkerStatusReporterTest.java | 10 +-
.../client/grpc/GrpcWindmillServerTest.java | 51 +-
.../windmill/state/WindmillStateCacheTest.java | 12 +-
.../windmill/state/WindmillStateInternalsTest.java | 18 +-
sdks/go.mod | 6 +-
sdks/go.sum | 12 +-
sdks/go/examples/stringsplit/stringsplit.go | 2 +-
sdks/go/pkg/beam/core/core.go | 2 +-
.../prism/internal/engine/elementmanager.go | 26 +-
.../runners/prism/internal/engine/engine_test.go | 5 +
.../runners/prism/internal/engine/teststream.go | 49 +-
.../beam/runners/prism/internal/environments.go | 4 +-
sdks/go/pkg/beam/runners/prism/internal/web/web.go | 57 ++
.../beam/runners/prism/internal/worker/bundle.go | 5 +-
sdks/go/test/integration/primitives/teststream.go | 74 ++
.../test/integration/primitives/teststream_test.go | 20 +
sdks/go/test/integration/primitives/timers.go | 2 +-
.../main/java/org/apache/beam/sdk/Pipeline.java | 2 +
.../beam/sdk/metrics/LabeledMetricNameUtils.java | 122 +++
.../sdk/schemas/annotations/SchemaFieldNumber.java | 7 +-
.../beam/sdk/transforms/join/CoGbkResult.java | 15 +
.../sdk/util/construction/CoderTranslation.java | 17 +
.../sdk/metrics/LabeledMetricNameUtilsTest.java | 96 ++
.../extensions/sql/impl/planner/BeamRuleSets.java | 3 +-
.../beam/sdk/extensions/sql/BeamSqlAliasTest | 94 ++
.../rule/BeamAggregateProjectMergeRuleTest.java | 3 +
.../java/org/apache/beam/fn/harness/FnHarness.java | 2 +
.../org/apache/beam/fn/harness/CachesTest.java | 5 +-
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 22 +
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 31 +-
.../sdk/io/gcp/bigquery/BigQueryIOTranslation.java | 18 +-
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 12 +-
.../sdk/io/gcp/bigquery/BigQuerySinkMetrics.java | 134 +--
.../gcp/bigquery/DynamicDestinationsHelpers.java | 8 +
.../io/gcp/bigquery/StreamingInsertsMetrics.java | 63 +-
.../sdk/io/gcp/spanner/MutationSizeEstimator.java | 4 +
.../beam/sdk/io/gcp/spanner/MutationUtils.java | 9 +-
.../beam/sdk/io/gcp/spanner/SpannerSchema.java | 6 +
.../io/gcp/spanner/SpannerTransformRegistrar.java | 8 +
.../beam/sdk/io/gcp/spanner/StructUtils.java | 16 +-
.../sdk/io/gcp/bigquery/BigQueryClusteringIT.java | 5 +-
.../sdk/io/gcp/bigquery/BigQueryHelpersTest.java | 11 +
.../io/gcp/bigquery/BigQueryIOTranslationTest.java | 10 +-
.../io/gcp/bigquery/BigQuerySinkMetricsTest.java | 124 +--
.../BigQueryTimePartitioningClusteringIT.java | 1 +
.../gcp/bigquery/StreamingInsertsMetricsTest.java | 15 +-
.../io/gcp/spanner/MutationSizeEstimatorTest.java | 16 +-
.../beam/sdk/io/gcp/spanner/MutationUtilsTest.java | 22 +-
.../beam/sdk/io/gcp/spanner/StructUtilsTest.java | 21 +-
sdks/java/io/iceberg/build.gradle | 91 ++
.../beam/io/iceberg/AppendFilesToTables.java | 102 ++
.../apache/beam/io/iceberg/AssignDestinations.java | 65 ++
.../beam/io/iceberg/DynamicDestinations.java | 23 +-
.../apache/beam/io/iceberg/FileWriteResult.java | 203 ++++
.../beam/io/iceberg/IcebergCatalogConfig.java | 233 +++++
.../apache/beam/io/iceberg/IcebergDestination.java | 65 ++
.../java/org/apache/beam/io/iceberg/IcebergIO.java | 93 ++
.../apache/beam/io/iceberg/IcebergScanConfig.java | 166 ++++
.../beam/io/iceberg/IcebergTableCreateConfig.java | 30 +-
.../apache/beam/io/iceberg/IcebergWriteResult.java | 65 ++
.../io/iceberg/OneTableDynamicDestinations.java | 65 ++
.../apache/beam/io/iceberg/PropertyBuilder.java | 27 +-
.../org/apache/beam/io/iceberg/RecordWriter.java | 98 ++
.../org/apache/beam/io/iceberg/ScanSource.java | 127 +++
.../org/apache/beam/io/iceberg/ScanTaskReader.java | 186 ++++
.../org/apache/beam/io/iceberg/ScanTaskSource.java | 87 ++
.../beam/io/iceberg/SchemaAndRowConversions.java | 270 ++++++
.../beam/io/iceberg/WriteGroupedRowsToFiles.java | 103 ++
.../beam/io/iceberg/WriteToDestinations.java | 123 +++
.../beam/io/iceberg/WriteUngroupedRowsToFiles.java | 350 +++++++
.../org/apache/beam/io/iceberg}/package-info.java | 4 +-
.../beam/io/iceberg/FileWriteResultTest.java | 151 +++
.../apache/beam/io/iceberg/IcebergIOReadTest.java | 120 +++
.../apache/beam/io/iceberg/IcebergIOWriteTest.java | 293 ++++++
.../org/apache/beam/io/iceberg/ScanSourceTest.java | 180 ++++
.../io/iceberg/SchemaAndRowConversionsTest.java | 226 +++++
.../apache/beam/io/iceberg/TestDataWarehouse.java | 146 +++
.../org/apache/beam/io/iceberg/TestFixtures.java | 101 ++
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 48 +-
.../KafkaIOReadImplementationCompatibility.java | 6 +
.../KafkaReadSchemaTransformConfiguration.java | 36 +-
.../io/kafka/KafkaReadSchemaTransformProvider.java | 2 +
.../beam/sdk/io/kafka/ReadFromKafkaDoFn.java | 19 +-
.../beam/sdk/io/kafka/KafkaIOExternalTest.java | 5 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 12 +
.../KafkaReadSchemaTransformProviderTest.java | 4 +-
.../beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java | 13 +
.../sdk/io/kafka/upgrade/KafkaIOTranslation.java | 21 +-
.../io/kafka/upgrade/KafkaIOTranslationTest.java | 1 +
sdks/java/testing/nexmark/build.gradle | 2 +-
sdks/java/testing/tpcds/build.gradle | 2 +-
sdks/python/apache_beam/coders/coder_impl.pxd | 1 -
.../online_clustering/clustering_pipeline/setup.py | 2 +-
.../inference/runinference_metrics/setup.py | 2 +-
.../aggregation/approximatequantiles.py} | 64 +-
.../aggregation/approximatequantiles_test.py} | 37 +-
.../transforms/aggregation/approximateunique.py} | 64 +-
.../aggregation/approximateunique_test.py | 54 ++
.../transforms/aggregation/batchelements.py | 64 ++
.../transforms/aggregation/batchelements_test.py | 65 ++
.../snippets/transforms/aggregation/tolist.py} | 42 +-
.../snippets/transforms/aggregation/tolist_test.py | 60 ++
.../io/external/xlang_jdbcio_it_test.py | 16 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 6 +-
sdks/python/apache_beam/io/gcp/spanner.py | 11 +
sdks/python/apache_beam/io/kafka.py | 11 +-
.../python/apache_beam/metrics/monitoring_infos.py | 1 -
sdks/python/apache_beam/ml/inference/base.py | 100 +-
sdks/python/apache_beam/ml/inference/base_test.py | 8 +
sdks/python/apache_beam/ml/transforms/tft.py | 41 +-
sdks/python/apache_beam/ml/transforms/tft_test.py | 126 +++
sdks/python/apache_beam/runners/common.pxd | 4 +-
sdks/python/apache_beam/runners/common.py | 104 +-
.../runners/dataflow/dataflow_runner.py | 27 +
.../runners/dataflow/dataflow_runner_test.py | 61 ++
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../apache-beam-jupyterlab-sidepanel/yarn.lock | 13 +-
.../apache_beam/runners/portability/stager.py | 1 +
.../apache_beam/runners/worker/opcounters.py | 1 -
.../apache_beam/runners/worker/operations.py | 1 -
.../runners/worker/statesampler_fast.pyx | 1 -
.../python/apache_beam/testing/fast_test_utils.pxd | 1 -
.../cy_dataflow_distribution_counter.pxd | 1 -
.../cy_dataflow_distribution_counter.pyx | 1 -
.../transforms/enrichment_handlers/bigtable.py | 26 +-
.../enrichment_handlers/bigtable_it_test.py | 28 +
.../enrichment_handlers/bigtable_test.py | 41 +
sdks/python/apache_beam/utils/counters.pxd | 1 -
sdks/python/apache_beam/utils/windowed_value.py | 1 -
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/examples/README.md | 9 -
.../apache_beam/yaml/examples/simple_filter.yaml | 49 +
.../yaml/examples/simple_filter_and_combine.yaml | 64 ++
.../yaml/examples/{ => testing}/__init__.py | 0
.../yaml/examples/{ => testing}/examples_test.py | 82 +-
.../aggregation/combine_count_minimal.yaml | 3 -
.../aggregation/combine_max_minimal.yaml | 3 -
.../aggregation/combine_mean_minimal.yaml | 3 -
.../aggregation/combine_min_minimal.yaml | 3 -
.../aggregation/combine_multiple_aggregations.yaml | 3 -
.../transforms/aggregation/combine_sum.yaml | 3 -
.../aggregation/combine_sum_minimal.yaml | 3 -
.../transforms/aggregation/group_into_batches.yaml | 3 -
.../aggregation/top_largest_per_key.yaml | 3 -
.../aggregation/top_smallest_per_key.yaml | 3 -
.../yaml/examples/wordcount_minimal.yaml | 5 +-
sdks/python/apache_beam/yaml/integration_tests.py | 27 +-
.../apache_beam/yaml/programming_guide_test.py | 15 +-
sdks/python/apache_beam/yaml/readme_test.py | 33 +-
sdks/python/apache_beam/yaml/tests/avro.yaml | 47 +
sdks/python/apache_beam/yaml/tests/java-map.yaml | 74 ++
sdks/python/apache_beam/yaml/tests/join.yaml | 186 ++++
sdks/python/apache_beam/yaml/tests/parquet.yaml | 47 +
sdks/python/apache_beam/yaml/tests/sql.yaml | 59 ++
sdks/python/apache_beam/yaml/tests/text.yaml | 47 +
sdks/python/apache_beam/yaml/tests/windowing.yaml | 122 +++
sdks/python/apache_beam/yaml/yaml_combine.py | 4 -
sdks/python/apache_beam/yaml/yaml_combine_test.py | 21 +-
sdks/python/apache_beam/yaml/yaml_join.py | 281 ++++++
sdks/python/apache_beam/yaml/yaml_join_test.py | 216 +++++
sdks/python/apache_beam/yaml/yaml_mapping.py | 32 +-
sdks/python/apache_beam/yaml/yaml_provider.py | 8 +-
sdks/python/apache_beam/yaml/yaml_transform.py | 4 +-
.../python/apache_beam/yaml/yaml_transform_test.py | 3 +-
sdks/python/apache_beam/yaml/yaml_udf_test.py | 89 +-
sdks/python/build.gradle | 19 +
.../container/base_image_requirements_manual.txt | 4 +-
sdks/python/container/piputil.go | 4 +-
.../container/py310/base_image_requirements.txt | 59 +-
.../container/py311/base_image_requirements.txt | 61 +-
.../container/py38/base_image_requirements.txt | 55 +-
.../container/py39/base_image_requirements.txt | 59 +-
sdks/python/pyproject.toml | 2 +-
sdks/python/scripts/generate_pydoc.sh | 1 +
sdks/python/setup.py | 9 +-
sdks/python/tox.ini | 3 +-
sdks/typescript/package.json | 2 +-
settings.gradle.kts | 14 +-
.../www/site/content/en/blog/beam-yaml-release.md | 205 ++++
.../content/en/documentation/dsls/sql/shell.md | 6 +-
.../content/en/documentation/runners/dataflow.md | 5 +-
.../content/en/documentation/sdks/yaml-combine.md | 2 -
.../site/content/en/documentation/sdks/yaml-udf.md | 173 +++-
.../python/aggregation/approximatequantiles.md | 9 +-
.../python/aggregation/approximateunique.md | 9 +-
.../{approximatequantiles.md => batchelements.md} | 14 +-
.../transforms/python/aggregation/distinct.md | 2 +-
.../transforms/python/aggregation/groupbykey.md | 14 +-
.../{approximatequantiles.md => tolist.md} | 13 +-
.../en/documentation/transforms/python/overview.md | 9 +-
.../site/content/en/get-started/quickstart-java.md | 2 +-
.../get-started/resources/videos-and-podcasts.md | 8 +-
website/www/site/data/authors.yml | 3 +
.../partials/section-menu/en/documentation.html | 3 +
.../shortcodes/flink_java_pipeline_options.html | 23 +-
.../shortcodes/flink_python_pipeline_options.html | 23 +-
website/www/site/static/.htaccess | 2 +-
316 files changed, 11329 insertions(+), 2443 deletions(-)
rename .github/trigger_files/{beam_PostCommit_Python_Examples_Direct.json =>
beam_PostCommit_Python.json} (97%)
create mode 100644
.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
copy
learning/tour-of-beam/learning-content/io/text-io/text-io-local-write/go-example/myfile.txt
=> .github/trigger_files/beam_PostCommit_Python_ValidatesRunner_Flink.json
(100%)
copy .github/workflows/{beam_PreCommit_Java_Kafka_IO_Direct.yml =>
IO_Iceberg.yml} (79%)
copy .github/workflows/{beam_PreCommit_Whitespace.yml =>
beam_LoadTests_Java_PubsubIO.yml} (65%)
copy .github/workflows/{beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml =>
beam_PreCommit_Yaml_Xlang_Direct.yml} (75%)
copy .github/workflows/{beam_PreCommit_Whitespace.yml =>
beam_StressTests_Java_BigQueryIO.yml} (65%)
copy .github/workflows/{beam_PreCommit_Whitespace.yml =>
beam_StressTests_Java_BigTableIO.yml} (65%)
copy .github/workflows/{beam_PostCommit_Java_Tpcds_Flink.yml =>
beam_StressTests_Java_KafkaIO.yml} (50%)
copy .github/workflows/{beam_PreCommit_Whitespace.yml =>
beam_StressTests_Java_PubSubIO.yml} (65%)
copy .github/workflows/{beam_PreCommit_Whitespace.yml =>
beam_StressTests_Java_SpannerIO.yml} (65%)
create mode 100644 contributor-docs/code-change-guide.md
copy
it/google-cloud-platform/src/test/java/org/apache/beam/it/gcp/pubsub/{PubSubIOLT.java
=> PubSubIOST.java} (75%)
rename
it/google-cloud-platform/src/test/java/org/apache/beam/it/gcp/pubsub/{PubSubIOLT.java
=> PubsubIOLT.java} (82%)
delete mode 100644 runners/flink/1.12/build.gradle
delete mode 100644
runners/flink/1.12/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/AbstractStreamOperatorCompat.java
delete mode 100644
runners/flink/1.12/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/compat/SplitEnumeratorCompat.java
delete mode 100644
runners/flink/1.12/src/test/java/org/apache/beam/runners/flink/RemoteMiniClusterImpl.java
delete mode 100644
runners/flink/1.12/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
delete mode 100644
runners/flink/1.12/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/SourceTestCompat.java
delete mode 100644 runners/flink/1.13/build.gradle
delete mode 100644 runners/flink/1.13/job-server/build.gradle
copy runners/flink/{ =>
1.14}/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
(94%)
rename runners/flink/{1.12 =>
1.14}/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ProcessingTimeCallbackCompat.java
(100%)
rename runners/flink/{1.13 =>
1.14}/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/compat/SplitEnumeratorCompat.java
(100%)
copy runners/flink/{1.12 =>
1.14}/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/compat/package-info.java
(100%)
rename runners/flink/{1.12 =>
1.14}/src/test/java/org/apache/beam/runners/flink/MiniClusterCompat.java (100%)
rename runners/flink/{ =>
1.14}/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java
(97%)
rename runners/flink/{1.13/job-server-container => 1.17}/build.gradle (87%)
rename runners/flink/{1.12 => 1.17}/job-server-container/build.gradle (100%)
rename runners/flink/{1.12 => 1.17}/job-server/build.gradle (95%)
rename runners/flink/{ =>
1.17}/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
(82%)
create mode 100644
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/UnversionedTypeSerializerSnapshot.java
create mode 100644
runners/flink/src/test/java/org/apache/beam/runners/flink/translation/types/UnversionedTypeSerializerSnapshotTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/LockFreeHistogram.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LockFreeHistogramTest.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/LabeledMetricNameUtils.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/LabeledMetricNameUtilsTest.java
create mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlAliasTest
create mode 100644 sdks/java/io/iceberg/build.gradle
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/AppendFilesToTables.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/AssignDestinations.java
rename
runners/flink/1.12/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/compat/FlinkSourceCompat.java
=>
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/DynamicDestinations.java
(61%)
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/FileWriteResult.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergCatalogConfig.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergDestination.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIO.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergScanConfig.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ActiveMessageMetadata.java
=>
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergTableCreateConfig.java
(50%)
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergWriteResult.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/OneTableDynamicDestinations.java
rename
runners/flink/1.12/src/test/java/org/apache/beam/runners/flink/metrics/MetricGroupWrapper.java
=>
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/PropertyBuilder.java
(56%)
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/RecordWriter.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/ScanSource.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/ScanTaskReader.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/ScanTaskSource.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/SchemaAndRowConversions.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/WriteGroupedRowsToFiles.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/WriteToDestinations.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/WriteUngroupedRowsToFiles.java
rename
{runners/flink/1.12/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/compat
=>
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg}/package-info.java
(82%)
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/FileWriteResultTest.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/IcebergIOReadTest.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/IcebergIOWriteTest.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/ScanSourceTest.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/SchemaAndRowConversionsTest.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/TestDataWarehouse.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/TestFixtures.java
copy
sdks/python/apache_beam/{yaml/examples/transforms/aggregation/combine_min_minimal.yaml
=> examples/snippets/transforms/aggregation/approximatequantiles.py} (51%)
copy sdks/python/apache_beam/{utils/counters.pxd =>
examples/snippets/transforms/aggregation/approximatequantiles_test.py} (51%)
copy
sdks/python/apache_beam/{yaml/examples/transforms/aggregation/combine_min_minimal.yaml
=> examples/snippets/transforms/aggregation/approximateunique.py} (51%)
create mode 100644
sdks/python/apache_beam/examples/snippets/transforms/aggregation/approximateunique_test.py
create mode 100644
sdks/python/apache_beam/examples/snippets/transforms/aggregation/batchelements.py
create mode 100644
sdks/python/apache_beam/examples/snippets/transforms/aggregation/batchelements_test.py
copy sdks/python/apache_beam/{utils/counters.pxd =>
examples/snippets/transforms/aggregation/tolist.py} (52%)
create mode 100644
sdks/python/apache_beam/examples/snippets/transforms/aggregation/tolist_test.py
create mode 100644
sdks/python/apache_beam/transforms/enrichment_handlers/bigtable_test.py
create mode 100644 sdks/python/apache_beam/yaml/examples/simple_filter.yaml
create mode 100644
sdks/python/apache_beam/yaml/examples/simple_filter_and_combine.yaml
rename sdks/python/apache_beam/yaml/examples/{ => testing}/__init__.py (100%)
rename sdks/python/apache_beam/yaml/examples/{ => testing}/examples_test.py
(63%)
create mode 100644 sdks/python/apache_beam/yaml/tests/avro.yaml
create mode 100644 sdks/python/apache_beam/yaml/tests/java-map.yaml
create mode 100644 sdks/python/apache_beam/yaml/tests/join.yaml
create mode 100644 sdks/python/apache_beam/yaml/tests/parquet.yaml
create mode 100644 sdks/python/apache_beam/yaml/tests/sql.yaml
create mode 100644 sdks/python/apache_beam/yaml/tests/text.yaml
create mode 100644 sdks/python/apache_beam/yaml/tests/windowing.yaml
create mode 100644 sdks/python/apache_beam/yaml/yaml_join.py
create mode 100644 sdks/python/apache_beam/yaml/yaml_join_test.py
create mode 100644 website/www/site/content/en/blog/beam-yaml-release.md
copy
website/www/site/content/en/documentation/transforms/python/aggregation/{approximatequantiles.md
=> batchelements.md} (57%)
copy
website/www/site/content/en/documentation/transforms/python/aggregation/{approximatequantiles.md
=> tolist.md} (65%)
