[beam] branch nightly-refs/heads/master updated (d08cef6 -> 4c979cc)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from d08cef6 Merge pull request #17055: [BEAM-14087] Mapped
JOB_STATE_RESOURCE_CLEANING_UP to State.RUNNING
add 9096f96 [BEAM-11934] Add enable_file_dynamic_sharding to allow
DataflowRunner users to opt-in to dynamic sharding logic (#17095)
add 280099a [BEAM-12777] Create symlink for `current` directory (#17105)
add 5f7bb55 [BEAM-14020] Adding SchemaTransform, SchemaTransformProvider,
TypedSchemaTransformProvider, and PCollectionRowTuple (#16958)
add 256bdd4 [BEAM-13015] Modify metrics to begin and reset to a non-dirty
state. (#16983)
add 62a6610 [BEAM-14112] Avoid storing a generator in
_CustomBigQuerySource (#17100)
add 6da5fa6 Populate environment capabilities in v1beta3 protos. (#17042)
add 0510cec [BEAM-14038] Auto-startup for Python expansion service.
(#17035)
add b88bed5 [BEAM-12976] Test a whole pipeline using projection pushdown
in BQ IO.
add eb5fbd3 [BEAM-12976] Enable projection pushdown for Java pipelines on
Dataflow, Flink, and Spark.
add 3e472e2 Merge pull request #17107 from ibzib/enable-optimizer
add d699086 [BEAM-14123] Fix typo in hdfsIntegrationTest task name.
add b36a557 Merge pull request #17114 from ibzib/hdfs-typo
add 6e196b2 [BEAM-13893] improved coverage of jobopts package (#17003)
add 48ed687 [BEAM-10212] Clean-up comments, remove rawtypes usage.
add 4c979cc Merge pull request #16954 from lukecwik/beam10212
No new revisions were added by this update.
Summary of changes:
CHANGES.md | 1 +
.../src/main/scripts/build_release_candidate.sh| 8 +-
.../beam/runners/core/metrics/CounterCell.java | 2 +-
.../beam/runners/core/metrics/DirtyState.java | 10 +-
.../runners/core/metrics/DistributionCell.java | 2 +-
.../beam/runners/core/metrics/GaugeCell.java | 2 +-
.../beam/runners/core/metrics/HistogramCell.java | 2 +-
.../beam/runners/core/metrics/CounterCellTest.java | 3 +-
.../beam/runners/core/metrics/DirtyStateTest.java | 8 +-
.../runners/core/metrics/DistributionCellTest.java | 3 +-
.../beam/runners/core/metrics/GaugeCellTest.java | 3 +-
.../runners/core/metrics/HistogramCellTest.java| 109 +
.../core/metrics/MetricsContainerImplTest.java | 38 ++-
.../org/apache/beam/runners/flink/FlinkRunner.java | 6 +
.../beam/runners/dataflow/DataflowRunner.java | 18 +-
.../org/apache/beam/runners/spark/SparkRunner.java | 6 +
sdks/go/pkg/beam/options/jobopts/options.go| 4 +-
sdks/go/pkg/beam/options/jobopts/options_test.go | 131 ++
.../SchemaTransform.java} | 23 +-
.../SchemaTransformProvider.java} | 43 ++--
.../transforms/TypedSchemaTransformProvider.java | 93 +++
.../org/apache/beam/sdk/util/HistogramData.java| 28 +++
.../beam/sdk/values/PCollectionRowTuple.java | 271 +
.../TypedSchemaTransformProviderTest.java | 126 ++
.../beam/sdk/values/PCollectionRowTupleTest.java | 177 ++
.../java/extensions/{arrow => python}/build.gradle | 14 +-
.../extensions/python/ExternalPythonTransform.java | 110 +
.../beam/sdk/extensions/python/PythonService.java | 132 ++
.../beam/sdk/extensions/python}/package-info.java | 4 +-
.../sdk/extensions/python/bootstrap_beam_venv.py | 115 +
.../python/ExternalPythonTransformTest.java} | 32 +--
.../apache/beam/fn/harness/FnApiDoFnRunner.java| 8 -
.../io/gcp/bigquery/BigQueryIOStorageReadIT.java | 59 -
sdks/python/apache_beam/io/gcp/bigquery.py | 38 ++-
.../apache_beam/io/gcp/bigquery_read_it_test.py| 12 +
.../runners/dataflow/internal/apiclient.py | 9 +-
sdks/python/test-suites/direct/common.gradle | 2 +-
settings.gradle.kts| 1 +
38 files changed, 1533 insertions(+), 120 deletions(-)
create mode 100644
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/HistogramCellTest.java
create mode 100644 sdks/go/pkg/beam/options/jobopts/options_test.go
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/{io/GenericDlqProvider.java
=> transforms/SchemaTransform.java} (57%)
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/{io/SchemaIOProvider.java
=> transforms/SchemaTransformProvider.java} (55%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/TypedSchemaTransformProvider.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionRowTuple.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/TypedSchemaTransformProviderTest.java
create mode
[beam] branch master updated: Merge pull request #16977 from [BEAM-12164] Added integration test for transaction boundaries and transaction ID ordering.
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 31ed331 Merge pull request #16977 from [BEAM-12164] Added integration test for transaction boundaries and transaction ID ordering. 31ed331 is described below commit 31ed3311711d64dc5588fea0e555122c4f2cbb4c Author: nancyxu123 AuthorDate: Fri Mar 18 04:25:45 2022 + Merge pull request #16977 from [BEAM-12164] Added integration test for transaction boundaries and transaction ID ordering. * Added integration test for transaction boundaries and transaction ID ordering. Made small fixes in ordered by key integration test. * [BEAM-9150] Fix beam_PostRelease_Python_Candidate (python RC validation scripts) (#16955) * Use default context output rather than outputWithTimestamp for ElasticsearchIO * Palo Alto case study - fix link * [BEAM-12777] Removed current docs version redirect * Merge pull request #16850: [BEAM-11205] Upgrade Libraries BOM dependencies to 24.3.0 * Update GCP Libraries BOM version to 24.3.0 * Update associated dependencies * Merge pull request #16484 from [BEAM-13633] [Playground] Implement method to get a default example for each SDKs * Implement method to get a default example for each SDKs * Add error handling * Added saving of precompiled objects catalog to cache at the server startup * Added caching of the catalog only in case of unspecified SDK * Update regarding comments * Update regarding comments * Simplified logging regarding comment * Get defaultExamplePath from the corresponding config * Refactoring code * Add the `link` field to response * Remove gjson; Resolve conflicts; * Refactoring code * Getting default precompiled object from cache * Refactoring code * Added saving of precompiled objects catalog to cache at the server startup * Added caching of the catalog only in case of unspecified SDK * Update regarding comments * Update regarding comments * Simplified logging regarding comment * Updates regarding comments * Update for environment_service_test.go * Get default example from catalog * GetCatalogFromCacheOrStorage method * Update licenses * Update licenses; Resolve conflicts; * [BEAM-13633][Playground] Change saving default precompiled objects to the cache * [BEAM-13633][Playground] Change logic of saving and receiving info about default precompiled objects * [BEAM-13633][Playground] Separate for each sdk * [BEAM-13633][Playground] regenerate proto files * Add code of the default example to response * Revert "Add code of the default example to response" This reverts commit da6baa0aaa272190d4a035568a5e4db0b093dfd9. * Refactoring code * Refactoring code; Add test; * Edit commentaries * Refactoring code * Add bucket name to methods Co-authored-by: Artur Khanin Co-authored-by: AydarZaynutdinov Co-authored-by: Pavel Avilov * Add 2022 events blog post (#16975) * Clean up Go formatter suggestions (#16973) * [BEAM-14012] Add go fmt to Github Actions (#16978) * [BEAM-13911] Add basic tests to Go direct runner. (#16979) * [BEAM-13960] Add support for more types when converting from between row and proto (#16875) * Adding schema support. * Addressing feedback. * Bump org.mongodb:mongo-java-driver to 3.12.10 * [BEAM-13973] Link Dataproc Flink master URLs to the InteractiveRunner when FlinkRunner is used (#16904) * [BEAM-13925] Turn pr bot on for go prs (#16984) * [BEAM-13964] Bump kotlin to 1.6.x (#16882) * [BEAM-13964] Bump kotlin to 1.6.x * [BEAM-13964] Bump kotlin to 1.6.x * [BEAM-13964] fix warnings in Kotlin compilation * Skipping flaky sad-path tests for Spanner changestreams * Merge pull request #16906: [BEAM-13974] Handle idle Storage Api streams * Merge pull request #16562 from [BEAM-13051][D] Enable pylint warnings (no-name-in-module/no-value-for-parameter) * [BEAM-13051] Pylint no-name-in-module and no-value-for-parameter warnings enabled * [BEAM-13051] Fixed no-value-for-parameter warning for missing default values * [BEAM-13051] Fixed parameters warnings * [BEAM-13925] A couple small pr-bot bug fixes (#16996) * [BEAM-14029] Add getter, setter for target maven repo (#16995) * [BEAM-13903] Improve coverage of metricsx package (#16994) *
[beam] branch master updated: [BEAM-10212] Clean-up comments, remove rawtypes usage.
This is an automated email from the ASF dual-hosted git repository.
altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 48ed687 [BEAM-10212] Clean-up comments, remove rawtypes usage.
new 4c979cc Merge pull request #16954 from lukecwik/beam10212
48ed687 is described below
commit 48ed6878d6f886bcfba745fb5474088ef76141f3
Author: Luke Cwik
AuthorDate: Fri Feb 25 12:19:31 2022 -0800
[BEAM-10212] Clean-up comments, remove rawtypes usage.
---
.../apache/beam/fn/harness/FnApiDoFnRunner.java| 26 +++---
1 file changed, 8 insertions(+), 18 deletions(-)
diff --git
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java
index 89c05b8..cf84027 100644
---
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java
+++
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java
@@ -721,7 +721,6 @@ public class FnApiDoFnRunner primaryInFullyProcessedWindowsRoot,
+WindowedValue primarySplitRoot,
+WindowedValue residualSplitRoot,
+WindowedValue residualInUnprocessedWindowsRoot) {
return new AutoValue_FnApiDoFnRunner_WindowedSplitResult(
primaryInFullyProcessedWindowsRoot,
primarySplitRoot,
@@ -1008,18 +1000,17 @@ public class FnApiDoFnRunner
getPrimaryInFullyProcessedWindowsRoot();
-public abstract @Nullable WindowedValue getPrimarySplitRoot();
+public abstract @Nullable WindowedValue getPrimarySplitRoot();
-public abstract @Nullable WindowedValue getResidualSplitRoot();
+public abstract @Nullable WindowedValue getResidualSplitRoot();
-public abstract @Nullable WindowedValue
getResidualInUnprocessedWindowsRoot();
+public abstract @Nullable WindowedValue
getResidualInUnprocessedWindowsRoot();
}
@AutoValue
@AutoValue.CopyAnnotations
- @SuppressWarnings({"rawtypes"})
abstract static class SplitResultsWithStopIndex {
public static SplitResultsWithStopIndex of(
WindowedSplitResult windowSplit,
@@ -1750,7 +1741,6 @@ public class FnApiDoFnRunner
[beam] branch master updated (b36a557 -> 6e196b2)
This is an automated email from the ASF dual-hosted git repository. danoliveira pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b36a557 Merge pull request #17114 from ibzib/hdfs-typo add 6e196b2 [BEAM-13893] improved coverage of jobopts package (#17003) No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/options/jobopts/options.go | 4 +- sdks/go/pkg/beam/options/jobopts/options_test.go | 131 +++ 2 files changed, 134 insertions(+), 1 deletion(-) create mode 100644 sdks/go/pkg/beam/options/jobopts/options_test.go
[beam] branch master updated: [BEAM-14123] Fix typo in hdfsIntegrationTest task name.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new d699086 [BEAM-14123] Fix typo in hdfsIntegrationTest task name.
new b36a557 Merge pull request #17114 from ibzib/hdfs-typo
d699086 is described below
commit d699086b6d1a20210eae0cbb4867bddd95dd0aa1
Author: Kyle Weaver
AuthorDate: Thu Mar 17 16:09:14 2022 -0700
[BEAM-14123] Fix typo in hdfsIntegrationTest task name.
---
sdks/python/test-suites/direct/common.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sdks/python/test-suites/direct/common.gradle
b/sdks/python/test-suites/direct/common.gradle
index 4f1eac7..15df969 100644
--- a/sdks/python/test-suites/direct/common.gradle
+++ b/sdks/python/test-suites/direct/common.gradle
@@ -177,7 +177,7 @@ tasks.register("directRunnerIT") {
}
}
-tasks.register("hdfsIntegrationTest ") {
+tasks.register("hdfsIntegrationTest") {
doLast {
exec {
executable 'sh'
[beam] branch master updated (0510cec -> 3e472e2)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0510cec [BEAM-14038] Auto-startup for Python expansion service. (#17035) new b88bed5 [BEAM-12976] Test a whole pipeline using projection pushdown in BQ IO. new eb5fbd3 [BEAM-12976] Enable projection pushdown for Java pipelines on Dataflow, Flink, and Spark. new 3e472e2 Merge pull request #17107 from ibzib/enable-optimizer The 34922 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.md | 1 + .../org/apache/beam/runners/flink/FlinkRunner.java | 6 +++ .../beam/runners/dataflow/DataflowRunner.java | 8 ++- .../org/apache/beam/runners/spark/SparkRunner.java | 6 +++ .../io/gcp/bigquery/BigQueryIOStorageReadIT.java | 59 +- 5 files changed, 78 insertions(+), 2 deletions(-)
[beam] branch master updated (6da5fa6 -> 0510cec)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 6da5fa6 Populate environment capabilities in v1beta3 protos. (#17042)
add 0510cec [BEAM-14038] Auto-startup for Python expansion service.
(#17035)
No new revisions were added by this update.
Summary of changes:
.../java/extensions/{arrow => python}/build.gradle | 14 +--
.../extensions/python/ExternalPythonTransform.java | 110 +
.../beam/sdk/extensions/python/PythonService.java | 132 +
.../beam/sdk/extensions/python}/package-info.java | 4 +-
.../sdk/extensions/python/bootstrap_beam_venv.py | 115 ++
.../python/ExternalPythonTransformTest.java} | 32 ++---
settings.gradle.kts| 1 +
7 files changed, 384 insertions(+), 24 deletions(-)
copy sdks/java/extensions/{arrow => python}/build.gradle (78%)
create mode 100644
sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/extensions/python/ExternalPythonTransform.java
create mode 100644
sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/extensions/python/PythonService.java
copy
{examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/options
=>
sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/extensions/python}/package-info.java
(87%)
create mode 100644
sdks/java/extensions/python/src/main/resources/org/apache/beam/sdk/extensions/python/bootstrap_beam_venv.py
copy
sdks/java/extensions/{euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/translate/TimestampExtractTransformTest.java
=>
python/src/test/java/org/apache/beam/sdk/extensions/python/ExternalPythonTransformTest.java}
(57%)
[beam] branch master updated (62a6610 -> 6da5fa6)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 62a6610 [BEAM-14112] Avoid storing a generator in _CustomBigQuerySource (#17100) add 6da5fa6 Populate environment capabilities in v1beta3 protos. (#17042) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py | 9 - 1 file changed, 4 insertions(+), 5 deletions(-)
[beam] branch master updated (256bdd4 -> 62a6610)
This is an automated email from the ASF dual-hosted git repository. ningk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 256bdd4 [BEAM-13015] Modify metrics to begin and reset to a non-dirty state. (#16983) add 62a6610 [BEAM-14112] Avoid storing a generator in _CustomBigQuerySource (#17100) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/io/gcp/bigquery.py | 38 +- .../apache_beam/io/gcp/bigquery_read_it_test.py| 12 +++ 2 files changed, 28 insertions(+), 22 deletions(-)
[beam] branch master updated (5f7bb55 -> 256bdd4)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5f7bb55 [BEAM-14020] Adding SchemaTransform, SchemaTransformProvider, TypedSchemaTransformProvider, and PCollectionRowTuple (#16958) add 256bdd4 [BEAM-13015] Modify metrics to begin and reset to a non-dirty state. (#16983) No new revisions were added by this update. Summary of changes: .../beam/runners/core/metrics/CounterCell.java | 2 +- .../beam/runners/core/metrics/DirtyState.java | 10 +- .../runners/core/metrics/DistributionCell.java | 2 +- .../beam/runners/core/metrics/GaugeCell.java | 2 +- .../beam/runners/core/metrics/HistogramCell.java | 2 +- .../beam/runners/core/metrics/CounterCellTest.java | 3 +- .../beam/runners/core/metrics/DirtyStateTest.java | 8 +- .../runners/core/metrics/DistributionCellTest.java | 3 +- .../beam/runners/core/metrics/GaugeCellTest.java | 3 +- .../runners/core/metrics/HistogramCellTest.java| 109 + .../core/metrics/MetricsContainerImplTest.java | 38 --- .../org/apache/beam/sdk/util/HistogramData.java| 28 ++ 12 files changed, 186 insertions(+), 24 deletions(-) create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/HistogramCellTest.java
[beam] branch master updated (d08cef6 -> 9096f96)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d08cef6 Merge pull request #17055: [BEAM-14087] Mapped JOB_STATE_RESOURCE_CLEANING_UP to State.RUNNING add 9096f96 [BEAM-11934] Add enable_file_dynamic_sharding to allow DataflowRunner users to opt-in to dynamic sharding logic (#17095) No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/runners/dataflow/DataflowRunner.java | 10 ++ 1 file changed, 6 insertions(+), 4 deletions(-)
