This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly-refs/heads/master in repository https://gitbox.apache.org/repos/asf/beam.git
from e596abf7660 Merge pull request #17262: [BEAM-14244] Use the supplied output timestamp for processing time timers rather than the input watermark add 9578d82e5f9 [BEAM-13015] Lookup the container for the step once when registering instead of on each execution. (#17281) add af006afe9c0 [BEAM-14175] Log read loop abort at debug rather than error (#17183) add dce3fabf1fa [BEAM-11745] Fix author list rendering (#17308) add 8c1fdaaaf76 [BEAM-14144] Record JFR profiles when GC thrashing is detected (#17151) add 416fc9ba752 [BEAM-14104] Support shard aware aggregation in Kinesis writer. add ad4561ea5d8 Merge pull request #17113 from mosche/BEAM-14104-ShardAwareAggregation add a88b2a109e1 Factors enable_prime flag in when checking use_unified_worker conditions (#17271) add 9e4c288627c [BEAM-11104] Add ProcessContinuation type to Go SDK (#17265) add e44d8a72d4a BEAM-13939: Restructure Protos to fix namespace conflicts (#16961) add 81d3de2007f [BEAM-14270] Mark {Snowflake/BigQuery}Services as @Internal (#17309) add 26f6d86b556 [BEAM-13901] Add unit tests for graphx/cogbk.go add f6cfa2739f4 [BEAM-13992] [Website] update Contribute/Code Contribution Guide page add 468994f37ed [BEAM-13992] [Website] change text, transfer tag a add 3951a9941c1 [BEAM-13992] [Website] change code tags add 9642881a6d7 [BEAM-13992] [Website] change text add 3137b6ef960 [BEAM-13992] [Website] change text and links, add empty lines add 7a4df7de59e Merge branch 'master' into BEAM-13992-update-contribution-guide-page add cf9ae549e03 [BEAM-13992] [Website] change links, change text add f82be4394c9 [BEAM-13992] [Website] change links, add text, add dots add da76ae13d10 [BEAM-13992] [Website] change links, change text add dc824115352 [BEAM-13992] [Website] change text, delete whitespace add 33cc5cda880 Merge branch 'master' into BEAM-13992-update-contribution-guide-page add a63c3973647 [BEAM-13992] [Website] update text add acc7dc6bbb3 Merge branch 'master' into BEAM-13992-update-contribution-guide-page add 24c20ff8709 [BEAM-13991] [Website] change links, add contribute file add 882b9afc580 [BEAM-13991] [Website] add content, add styles add 661d34a21d8 Merge branch 'master' into BEAM-13991-update-contribute-to-beam-page add 6d1dab25a3d [BEAM-13991] [Website] add images, add styles, delete spaces add 1f2f1c17f93 [BEAM-13991] [Website] change url and aliases, delete bullet points add 30af0b9aff6 [BEAM-13991] [Website] add empty line add b77d513845c Merge branch 'master' into BEAM-13991-update-contribute-to-beam-page add a6143af4cd5 [BEAM-13991] [Website] change styles, change quotes add e8a155f07a4 [BEAM-13991] [Website] change link color add 8ec18a2403a Merge branch 'master' into BEAM-13991-update-contribute-to-beam-page add 7788ba26832 Merge branch 'master' into BEAM-13991-update-contribute-to-beam-page add 3f0540ce075 [BEAM-13991] [Website] change text add 63ee894027c Merge remote-tracking branch 'origin/BEAM-13991-update-contribute-to-beam-page' into BEAM-13991-update-contribute-to-beam-page add 1978c9040b7 [BEAM-13991] [Website] added changes from PR 13992, changed get-starting contributing add d27119c63c0 Merge branch 'master' into BEAM-13991-update-contribute-to-beam-page add 0035f37e4e0 Merge pull request #17136 from akvelon/BEAM-13991-update-contribute-to-beam-page add f9e317cebec Sets 'sdk_harness_container_images' property for all Dataflow jobs - Python SDK add 238d7b99b9e Merge pull request #16974: [BEAM-14009] Sets 'sdk_harness_container_images' property for all Dataflow jobs - Python SDK add 5343f1a606d Sets 'sdk_harness_container_images' property for all Dataflow jobs - Java SDK add 25d14a7f12a Merge pull request #16981: [BEAM-14009] Sets 'sdk_harness_container_images' property for all Dataflow jobs - Java SDK add 443924a76d0 [BEAM-14259, BEAM-14266] Remove unused function, replace use of ptypes (#17301) add 397b1809957 [BEAM-14274] Fix staticcheck warnings in pipelinex (#17311) add f42d4375e18 [BEAM-13857] Switched Go IT script to using Go flags for expansion services (#17161) add b4b0025acd6 [BEAM-10529] add java and generic components of nullable xlang tests add 798fd532832 [BEAM-10529] fix test case add 87435496681 [BEAM-10529] add coders and typehints to support nullable xlang coders add 1ce66059a97 [BEAM-10529] update external builder to support nullable coder add 04868d6a76a Merge remote-tracking branch 'origin/master' into feature/BEAM-10529-nullable-Xlang-Coder add a6a38d0ccc9 [BEAM-10529] clean up coders.py add e3e02235a5a [BEAM-10529] add coder translation test add 30f06d69dfd [BEAM-10529] add additional check to typecoder to not accidentally misidentify coders as nullable add 24dd8767553 [BEAM-10529] add test to retrieve nullable coder from typehint add 3e93b56c886 [BEAM-10529] improve comments, clean up python add e642e7c437a [BEAM-10529] run spotless add f55074c7aed Merge remote-tracking branch 'origin/feature/BEAM-10529-nullable-Xlang-Coder' into feature/BEAM-10529-nullable-Xlang-Coder add 4dd1c548139 [BEAM-10529] add go nullable coder add 43c84e62b5b [BEAM-10529] cleanup extra println add 48b6f34fe70 Merge remote-tracking branch 'origin/feature/BEAM-10529-nullable-Xlang-Coder' into feature/BEAM-10529-nullable-Xlang-Coder add 5be0e94b5e5 Merge remote-tracking branch 'origin/master' into feature/BEAM-10529-enable-nullable-for-kafka add e9afde092c1 Merge pull request #17268: [BEAM-10529] enable nullable for kafkaIO xlang No new revisions were added by this update. Summary of changes: .gitignore | 3 +- build.gradle.kts | 2 +- .../org/apache/beam/gradle/BeamModulePlugin.groovy | 7 +- .../beam/model/fn_execution/v1}/beam_fn_api.proto | 6 +- .../fn_execution/v1}/beam_provision_api.proto | 4 +- .../interactive/v1}/beam_interactive_api.proto | 2 +- .../job_management/v1}/beam_artifact_api.proto | 2 +- .../job_management/v1}/beam_expansion_api.proto | 2 +- .../model/job_management/v1}/beam_job_api.proto | 6 +- .../beam/model/pipeline/v1}/beam_runner_api.proto | 2 +- .../apache/beam/model/pipeline/v1}/endpoints.proto | 0 .../model/pipeline/v1}/external_transforms.proto | 4 +- .../apache/beam/model/pipeline/v1}/metrics.proto | 2 +- .../apache/beam/model/pipeline/v1}/schema.proto | 0 .../model/pipeline/v1}/standard_window_fns.proto | 2 +- .../beam/runners/dataflow/DataflowRunner.java | 39 +- .../options/DataflowPipelineDebugOptions.java | 19 + .../beam/runners/dataflow/DataflowRunnerTest.java | 19 +- .../dataflow/worker/WorkItemStatusClient.java | 14 +- .../dataflow/worker/status/JfrzServlet.java | 69 + .../dataflow/worker/status/WorkerStatusPages.java | 6 +- .../runners/dataflow/worker/util/JfrInterop.java | 84 + .../dataflow/worker/util/MemoryMonitor.java | 168 +- .../worker/util/common/worker/ReadOperation.java | 10 +- .../dataflow/worker/util/MemoryMonitorTest.java | 84 +- sdks/go/pkg/beam/core/funcx/fn.go | 32 +- sdks/go/pkg/beam/core/funcx/fn_test.go | 7 + sdks/go/pkg/beam/core/runtime/exec/translate.go | 57 +- sdks/go/pkg/beam/core/runtime/graphx/cogbk_test.go | 125 + sdks/go/pkg/beam/core/runtime/graphx/v1/gen.go | 2 +- sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go | 664 ++-- sdks/go/pkg/beam/core/runtime/harness/gen.go | 2 +- .../go/pkg/beam/core/runtime/harness/session.proto | 2 +- .../core/runtime/harness/session/session.pb.go | 245 +- sdks/go/pkg/beam/core/runtime/pipelinex/replace.go | 5 +- sdks/go/pkg/beam/core/runtime/pipelinex/util.go | 2 +- .../pkg/beam/core/runtime/pipelinex/util_test.go | 15 - sdks/go/pkg/beam/core/sdf/continuation.go | 62 + .../graphx/v1/gen.go => sdf/continuation_test.go} | 25 +- sdks/go/pkg/beam/model/PROTOBUF.md | 10 +- .../beam/model/fnexecution_v1/beam_fn_api.pb.go | 2540 +++++++------ .../model/fnexecution_v1/beam_fn_api_grpc.pb.go | 17 +- .../model/fnexecution_v1/beam_provision_api.pb.go | 258 +- .../fnexecution_v1/beam_provision_api_grpc.pb.go | 7 +- sdks/go/pkg/beam/model/gen.go | 4 +- .../model/jobmanagement_v1/beam_artifact_api.pb.go | 665 ++-- .../jobmanagement_v1/beam_artifact_api_grpc.pb.go | 13 +- .../jobmanagement_v1/beam_expansion_api.pb.go | 167 +- .../jobmanagement_v1/beam_expansion_api_grpc.pb.go | 7 +- .../beam/model/jobmanagement_v1/beam_job_api.pb.go | 841 ++--- .../model/jobmanagement_v1/beam_job_api_grpc.pb.go | 7 +- .../beam/model/pipeline_v1/beam_runner_api.pb.go | 3935 +++++++++++--------- .../model/pipeline_v1/beam_runner_api_grpc.pb.go | 7 +- sdks/go/pkg/beam/model/pipeline_v1/endpoints.pb.go | 116 +- .../model/pipeline_v1/external_transforms.pb.go | 467 ++- sdks/go/pkg/beam/model/pipeline_v1/metrics.pb.go | 1049 +++--- sdks/go/pkg/beam/model/pipeline_v1/schema.pb.go | 632 ++-- .../model/pipeline_v1/standard_window_fns.pb.go | 224 +- sdks/go/pkg/beam/transforms/stats/util_gen.go | 1 + sdks/go/pkg/beam/transforms/stats/util_gen.tmpl | 2 +- sdks/go/scripts/genproto.sh | 159 + sdks/go/test/build.gradle | 57 +- sdks/go/test/integration/expansions.go | 7 +- sdks/go/test/integration/flags.go | 16 - .../integration/io/xlang/debezium/debezium_test.go | 23 +- .../go/test/integration/io/xlang/jdbc/jdbc_test.go | 29 +- .../test/integration/io/xlang/kafka/kafka_test.go | 26 +- sdks/go/test/integration/xlang/xlang_test.go | 38 +- .../harness/data/PTransformFunctionRegistry.java | 2 +- .../apache/beam/sdk/io/aws2/common/ClientPool.java | 123 - .../apache/beam/sdk/io/aws2/common/ObjectPool.java | 151 + .../sdk/io/aws2/common/RetryConfiguration.java | 2 +- .../apache/beam/sdk/io/aws2/kinesis/KinesisIO.java | 270 +- .../sdk/io/aws2/kinesis/KinesisPartitioner.java | 28 +- .../sdk/io/aws2/kinesis/RecordsAggregator.java | 4 - .../{ClientPoolTest.java => ObjectPoolTest.java} | 99 +- .../sdk/io/aws2/kinesis/KinesisIOWriteTest.java | 243 +- .../sdk/io/aws2/kinesis/PutRecordsHelpers.java | 8 + .../sdk/io/aws2/kinesis/testing/KinesisIOIT.java | 5 +- .../beam/sdk/io/gcp/bigquery/BigQueryServices.java | 2 + .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 17 +- .../io/snowflake/services/SnowflakeServices.java | 2 + sdks/python/.yapfignore | 2 +- sdks/python/apache_beam/io/gcp/bigquery.py | 4 +- .../python/apache_beam/portability/api/__init__.py | 21 - sdks/python/apache_beam/portability/common_urns.py | 43 +- .../runners/dataflow/internal/apiclient.py | 60 +- .../runners/dataflow/internal/apiclient_test.py | 140 +- .../runners/interactive/caching/streaming_cache.py | 20 +- .../interactive/caching/streaming_cache_test.py | 246 +- .../interactive/options/capture_control_test.py | 16 +- .../interactive/options/capture_limiters.py | 11 +- .../interactive/options/capture_limiters_test.py | 6 +- .../runners/interactive/recording_manager.py | 4 +- .../runners/interactive/recording_manager_test.py | 5 +- .../interactive/testing/test_cache_manager.py | 31 +- .../apache_beam/runners/interactive/utils.py | 8 +- .../apache_beam/runners/interactive/utils_test.py | 8 +- .../runners/portability/expansion_service_test.py | 4 +- .../runners/portability/fn_api_runner/execution.py | 2 +- .../portability/fn_api_runner/worker_handlers.py | 2 +- sdks/python/apache_beam/testing/test_stream.py | 34 +- .../apache_beam/testing/test_stream_service.py | 4 +- .../testing/test_stream_service_test.py | 14 +- .../python/apache_beam/testing/test_stream_test.py | 161 +- sdks/python/apache_beam/transforms/external.py | 10 +- .../python/apache_beam/transforms/external_test.py | 34 +- .../transforms/fully_qualified_named_transform.py | 7 +- sdks/python/gen_protos.py | 506 ++- sdks/python/mypy.ini | 2 +- sdks/python/scripts/run_pylint.sh | 6 +- sdks/python/scripts/run_pylint_2to3.sh | 6 +- sdks/python/setup.py | 41 +- website/www/site/assets/scss/_case_study.scss | 2 +- website/www/site/assets/scss/_contribute.sass | 41 + website/www/site/assets/scss/_traits.scss | 2 +- website/www/site/assets/scss/main.scss | 1 + website/www/site/content/en/contribute/_index.md | 578 ++- .../en/contribute/get-started-contributing.md | 390 ++ website/www/site/layouts/blog/single.html | 6 +- .../partials/section-menu/en/contribute.html | 2 +- .../talat_uyarer.png => talat_uyarer_photo.png} | Bin 122 files changed, 9761 insertions(+), 6800 deletions(-) rename model/fn-execution/src/main/proto/{ => org/apache/beam/model/fn_execution/v1}/beam_fn_api.proto (99%) rename model/fn-execution/src/main/proto/{ => org/apache/beam/model/fn_execution/v1}/beam_provision_api.proto (96%) rename model/interactive/src/main/proto/{ => org/apache/beam/model/interactive/v1}/beam_interactive_api.proto (96%) rename model/job-management/src/main/proto/{ => org/apache/beam/model/job_management/v1}/beam_artifact_api.proto (99%) rename model/job-management/src/main/proto/{ => org/apache/beam/model/job_management/v1}/beam_expansion_api.proto (97%) rename model/job-management/src/main/proto/{ => org/apache/beam/model/job_management/v1}/beam_job_api.proto (98%) rename model/pipeline/src/main/proto/{ => org/apache/beam/model/pipeline/v1}/beam_runner_api.proto (99%) rename model/pipeline/src/main/proto/{ => org/apache/beam/model/pipeline/v1}/endpoints.proto (100%) rename model/pipeline/src/main/proto/{ => org/apache/beam/model/pipeline/v1}/external_transforms.proto (97%) rename model/pipeline/src/main/proto/{ => org/apache/beam/model/pipeline/v1}/metrics.proto (99%) rename model/pipeline/src/main/proto/{ => org/apache/beam/model/pipeline/v1}/schema.proto (100%) rename model/pipeline/src/main/proto/{ => org/apache/beam/model/pipeline/v1}/standard_window_fns.proto (98%) create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/JfrzServlet.java create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/JfrInterop.java create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/cogbk_test.go create mode 100644 sdks/go/pkg/beam/core/sdf/continuation.go copy sdks/go/pkg/beam/core/{runtime/graphx/v1/gen.go => sdf/continuation_test.go} (62%) create mode 100755 sdks/go/scripts/genproto.sh delete mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/ClientPool.java create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/ObjectPool.java rename sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/common/{ClientPoolTest.java => ObjectPoolTest.java} (63%) delete mode 100644 sdks/python/apache_beam/portability/api/__init__.py create mode 100644 website/www/site/assets/scss/_contribute.sass create mode 100644 website/www/site/content/en/contribute/get-started-contributing.md copy website/www/site/static/images/{case-study/paloalto/talat_uyarer.png => talat_uyarer_photo.png} (100%)