(beam) branch asf-site updated: Publishing website 2024/02/13 05:37:32 at commit 702138b
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 7492dd5c7e2 Publishing website 2024/02/13 05:37:32 at commit 702138b 7492dd5c7e2 is described below commit 7492dd5c7e27465604de9e06eaa95de282aea1c5 Author: runner AuthorDate: Tue Feb 13 05:37:32 2024 + Publishing website 2024/02/13 05:37:32 at commit 702138b --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 90e4980c9d1..36e4d1dadbc 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/apache-beam-flink-and-kubernetes-part3/2024-02-12T17:54:24-05:00/categories/blog/2024-02-12T17:54:24-05:00/blog/2024-02-12T17:54:24-05:00/categories/2024-02-12T17:54:24-05:00http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/apache-beam-flink-and-kubernetes-part3/2024-02-12T22:55:36-05:00/categories/blog/2024-02-12T22:55:36-05:00/blog/2024-02-12T22:55:36-05:00/categories/2024-02-12T22:55:36-05:00
(beam) branch master updated: Remove WriteToFiles is experimental. (#30291)
This is an automated email from the ASF dual-hosted git repository. anandinguva pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 702138bf6b8 Remove WriteToFiles is experimental. (#30291) 702138bf6b8 is described below commit 702138bf6b8160fecd9a7893e4907aff3b1fe826 Author: liferoad AuthorDate: Mon Feb 12 22:55:36 2024 -0500 Remove WriteToFiles is experimental. (#30291) --- sdks/python/apache_beam/io/fileio.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/sdks/python/apache_beam/io/fileio.py b/sdks/python/apache_beam/io/fileio.py index c221e5bf73e..076ec9452e4 100644 --- a/sdks/python/apache_beam/io/fileio.py +++ b/sdks/python/apache_beam/io/fileio.py @@ -521,8 +521,6 @@ class WriteToFiles(beam.PTransform): **Note:** For unbounded ``PCollection``\s, this transform does not support multiple firings per Window (due to the fact that files are named only by their destination, and window, at the moment). - - WriteToFiles is experimental. No backwards-compatibility guarantees. """ # We allow up to 20 different destinations to be written in a single bundle.
(beam) branch nightly-refs/heads/master updated (6d10e0f40fd -> fa3249206d0)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly-refs/heads/master in repository https://gitbox.apache.org/repos/asf/beam.git from 6d10e0f40fd Fix two flaky tests (#30278) add 68873c81042 Update Python Dependencies (#30282) add 371576a3b17 Remove unused StreamingDataflowWorker parameter (#30256) add 47f777856bf fixed documentation in sdks/python/apache_beam/io/gcp/pubsub.py WriteToPubSub (#30285) add fa3249206d0 [JdbcIO] - Adding option for max batch buffering duration (#30259) No new revisions were added by this update. Summary of changes: .../dataflow/worker/StreamingDataflowWorker.java | 2 +- .../worker/StreamingModeExecutionContext.java | 23 +++--- .../dataflow/worker/counters/NameContext.java | 29 +++- .../dataflow/worker/streaming/StageInfo.java | 7 ++- .../worker/DataflowExecutionContextTest.java | 9 ++-- .../worker/StreamingModeExecutionContextTest.java | 14 ++ .../dataflow/worker/WorkerCustomSourcesTest.java | 4 +- .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 51 +++--- sdks/python/apache_beam/io/gcp/pubsub.py | 4 +- .../container/py310/base_image_requirements.txt| 49 +++-- .../container/py311/base_image_requirements.txt| 49 +++-- .../container/py38/base_image_requirements.txt | 47 ++-- .../container/py39/base_image_requirements.txt | 49 +++-- 13 files changed, 194 insertions(+), 143 deletions(-)
(beam) branch asf-site updated: Publishing website 2024/02/12 23:37:33 at commit fa32492
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 4441b8a893e Publishing website 2024/02/12 23:37:33 at commit fa32492 4441b8a893e is described below commit 4441b8a893e58261befd645e9b5775eb8d9acf5b Author: runner AuthorDate: Mon Feb 12 23:37:33 2024 + Publishing website 2024/02/12 23:37:33 at commit fa32492 --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 17496803e2d..90e4980c9d1 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/apache-beam-flink-and-kubernetes-part3/2024-02-12T06:33:20-08:00/categories/blog/2024-02-12T06:33:20-08:00/blog/2024-02-12T06:33:20-08:00/categories/2024-02-12T06:33:20-08:00http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/apache-beam-flink-and-kubernetes-part3/2024-02-12T17:54:24-05:00/categories/blog/2024-02-12T17:54:24-05:00/blog/2024-02-12T17:54:24-05:00/categories/2024-02-12T17:54:24-05:00
(beam) branch transform_service_test_suite updated: updates
This is an automated email from the ASF dual-hosted git repository.
chamikara pushed a commit to branch transform_service_test_suite
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/transform_service_test_suite
by this push:
new c6a74a0f706 updates
c6a74a0f706 is described below
commit c6a74a0f70659d00aa61cb26eaa9859daf4ae61c
Author: Chamikara Jayalath
AuthorDate: Mon Feb 12 15:07:19 2024 -0800
updates
---
.../workflows/beam_PostCommit_TransformService_Direct.yml | 1 +
.../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 13 ++---
2 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/beam_PostCommit_TransformService_Direct.yml
b/.github/workflows/beam_PostCommit_TransformService_Direct.yml
index ce72907c0c7..5efca5b481d 100644
--- a/.github/workflows/beam_PostCommit_TransformService_Direct.yml
+++ b/.github/workflows/beam_PostCommit_TransformService_Direct.yml
@@ -86,6 +86,7 @@ jobs:
-PuseWheelDistribution \
-PpythonVersion=${{ matrix.python_version }} \
--stacktrace \
+--info \
- name: Archive Python Test Results
uses: actions/upload-artifact@v4
if: failure()
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 8307489c604..876875d9ea7 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -2622,8 +2622,12 @@ class BeamModulePlugin implements Plugin {
def serviceArgs =
project.project(':sdks:python').mapToArgString(expansionServiceOpts)
System.err.println '** xyz123 BeamModulePlugin: starting
transform service ...'
executable 'sh'
-args '-c', ". ${project.ext.envdir}/bin/activate &&
$pythonDir/scripts/run_expansion_services.sh stop --group_id ${project.name} &&
$pythonDir/scripts/run_expansion_services.sh start $serviceArgs"
-System.err.println '** xyz123 BeamModulePlugin: DONE
starting transform service ...'
+args '-c', ". ${project.ext.envdir}/bin/activate &&
$pythonDir/scripts/run_expansion_services.sh stop --group_id ${project.name} &&
$pythonDir/scripts/run_expansion_services.sh start $serviceArgs >
./transform_service_start_log 2>&1"
+System.err.println '** xyz123 BeamModulePlugin: DONE
starting transform service LOG:\n'
+String fileContents = new
File('./transform_service_start_log').text
+System.err.println fileContents
+System.err.println "\nEND LOG\n\n"
+
}
}
}
@@ -2662,8 +2666,11 @@ class BeamModulePlugin implements Plugin {
// teardown test env
System.err.println '** xyz123 BeamModulePlugin: stopping
transform service ...'
executable 'sh'
-args '-c', ". ${project.ext.envdir}/bin/activate &&
$pythonDir/scripts/run_expansion_services.sh stop --group_id ${project.name}"
+args '-c', ". ${project.ext.envdir}/bin/activate &&
$pythonDir/scripts/run_expansion_services.sh stop --group_id ${project.name} >
./transform_service_stop_log 2>&1"
System.err.println '** xyz123 BeamModulePlugin: DONE stopping
transform service ...'
+String fileContents = new File('./transform_service_stop_log').text
+System.err.println fileContents
+System.err.println "\nEND LOG\n\n"
}
setupTask.configure {finalizedBy cleanupTask}
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 2f4a3e58968 Updating config from bot 2f4a3e58968 is described below commit 2f4a3e58968fe63fa07cbb0e45f69edafaef0d93 Author: github-actions AuthorDate: Mon Feb 12 23:05:52 2024 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-30264.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30264.json b/scripts/ci/pr-bot/state/pr-state/pr-30264.json index d0bf49e2c91..7fb614aa094 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-30264.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-30264.json @@ -7,5 +7,5 @@ "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
(beam) branch master updated: [JdbcIO] - Adding option for max batch buffering duration (#30259)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new fa3249206d0 [JdbcIO] - Adding option for max batch buffering duration
(#30259)
fa3249206d0 is described below
commit fa3249206d04295e677520b6df691e93b9b47cba
Author: Scott Strong
AuthorDate: Mon Feb 12 17:54:24 2024 -0500
[JdbcIO] - Adding option for max batch buffering duration (#30259)
* Adding option for max batch buffering duration
* adding getter for max batch duration
* ran spotless
* adding more to the javadoc for both withBatchSize and
withMaxBatchBufferingDuration
* running spotless
* using default value
-
Co-authored-by: Scott Strong
---
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 51 +++---
1 file changed, 46 insertions(+), 5 deletions(-)
diff --git
a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
index 35d94442f37..2e7334f183a 100644
--- a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
+++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
@@ -370,6 +370,7 @@ public class JdbcIO {
}
private static final long DEFAULT_BATCH_SIZE = 1000L;
+ private static final long DEFAULT_MAX_BATCH_BUFFERING_DURATION = 200L;
private static final int DEFAULT_FETCH_SIZE = 50_000;
// Default values used from fluent backoff.
private static final Duration DEFAULT_INITIAL_BACKOFF =
Duration.standardSeconds(1);
@@ -389,6 +390,7 @@ public class JdbcIO {
public static WriteVoid writeVoid() {
return new AutoValue_JdbcIO_WriteVoid.Builder()
.setBatchSize(DEFAULT_BATCH_SIZE)
+.setMaxBatchBufferingDuration(DEFAULT_MAX_BATCH_BUFFERING_DURATION)
.setRetryStrategy(new DefaultRetryStrategy())
.setRetryConfiguration(RetryConfiguration.create(5, null,
Duration.standardSeconds(5)))
.build();
@@ -1686,6 +1688,11 @@ public class JdbcIO {
return new Write<>(inner.withBatchSize(batchSize));
}
+/** See {@link WriteVoid#withMaxBatchBufferingDuration(long)}. */
+public Write withMaxBatchBufferingDuration(long
maxBatchBufferingDuration) {
+ return new
Write<>(inner.withMaxBatchBufferingDuration(maxBatchBufferingDuration));
+}
+
/** See {@link WriteVoid#withRetryStrategy(RetryStrategy)}. */
public Write withRetryStrategy(RetryStrategy retryStrategy) {
return new Write<>(inner.withRetryStrategy(retryStrategy));
@@ -1754,13 +1761,16 @@ public class JdbcIO {
/* The maximum number of elements that will be included in a batch. */
static PCollection> batchElements(
- PCollection input, @Nullable Boolean withAutoSharding, long
batchSize) {
+ PCollection input,
+ @Nullable Boolean withAutoSharding,
+ long batchSize,
+ long maxBatchBufferingDuration) {
PCollection> iterables;
if (input.isBounded() == IsBounded.UNBOUNDED) {
PCollection> keyedInput = input.apply(WithKeys.of(""));
GroupIntoBatches groupTransform =
GroupIntoBatches.ofSize(batchSize)
- .withMaxBufferingDuration(Duration.millis(200));
+
.withMaxBufferingDuration(Duration.millis(maxBatchBufferingDuration));
if (withAutoSharding != null && withAutoSharding) {
// unbounded and withAutoSharding enabled, group into batches with
shardedKey
iterables =
keyedInput.apply(groupTransform.withShardedKey()).apply(Values.create());
@@ -1958,7 +1968,8 @@ public class JdbcIO {
"Autosharding is only supported for streaming pipelines.");
PCollection> iterables =
- JdbcIO.batchElements(input, autoSharding, DEFAULT_BATCH_SIZE);
+ JdbcIO.batchElements(
+ input, autoSharding, DEFAULT_BATCH_SIZE,
DEFAULT_MAX_BATCH_BUFFERING_DURATION);
return iterables.apply(
ParDo.of(
new WriteFn(
@@ -1971,6 +1982,7 @@ public class JdbcIO {
.setRetryConfiguration(getRetryConfiguration())
.setReturnResults(true)
.setBatchSize(1L)
+
.setMaxBatchBufferingDuration(DEFAULT_MAX_BATCH_BUFFERING_DURATION)
.build(;
}
}
@@ -1990,6 +2002,8 @@ public class JdbcIO {
abstract long getBatchSize();
+abstract long getMaxBatchBufferingDuration();
+
abstract @Nullable PreparedStatementSetter getPreparedStatementSetter();
abstract @Nullable RetryStrategy getRetryStrategy();
@@ -2011,6 +2025,8 @@ public class JdbcIO {
abstract Builder setBatchSize(long batchSize);
+ abstract Builder setMaxBatchBufferingDuration(long
maxBatchBufferingDuration);
+
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 615dabb35fa Updating config from bot
615dabb35fa is described below
commit 615dabb35fa1ea132555e05151e9f6d0bf7e715d
Author: github-actions
AuthorDate: Mon Feb 12 22:40:39 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-30072.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30072.json
b/scripts/ci/pr-bot/state/pr-state/pr-30072.json
index 242a48d7d3b..37f7ab41238 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-30072.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-30072.json
@@ -2,7 +2,7 @@
"commentedAboutFailingChecks": true,
"reviewersAssignedForLabels": {},
"nextAction": "Author",
- "stopReviewerNotifications": false,
+ "stopReviewerNotifications": true,
"remindAfterTestsPass": [],
"committerAssigned": false
}
\ No newline at end of file
(beam) branch transform_service_test_suite updated: updates
This is an automated email from the ASF dual-hosted git repository.
chamikara pushed a commit to branch transform_service_test_suite
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/transform_service_test_suite
by this push:
new 6954d92dc27 updates
6954d92dc27 is described below
commit 6954d92dc27b20d1d8a22697448b1ef980bf6a12
Author: Chamikara Jayalath
AuthorDate: Mon Feb 12 14:35:41 2024 -0800
updates
---
.github/workflows/beam_PostCommit_TransformService_Direct.yml | 1 +
1 file changed, 1 insertion(+)
diff --git a/.github/workflows/beam_PostCommit_TransformService_Direct.yml
b/.github/workflows/beam_PostCommit_TransformService_Direct.yml
index fab980f63d5..ce72907c0c7 100644
--- a/.github/workflows/beam_PostCommit_TransformService_Direct.yml
+++ b/.github/workflows/beam_PostCommit_TransformService_Direct.yml
@@ -85,6 +85,7 @@ jobs:
-Pjava11Home=$JAVA_HOME_11_X64 \
-PuseWheelDistribution \
-PpythonVersion=${{ matrix.python_version }} \
+--stacktrace \
- name: Archive Python Test Results
uses: actions/upload-artifact@v4
if: failure()
(beam) branch transform_service_test_suite updated: updates
This is an automated email from the ASF dual-hosted git repository.
chamikara pushed a commit to branch transform_service_test_suite
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/transform_service_test_suite
by this push:
new d4f1b797f43 updates
d4f1b797f43 is described below
commit d4f1b797f43889056315027a331923e0b17380e9
Author: Chamikara Jayalath
AuthorDate: Mon Feb 12 14:10:03 2024 -0800
updates
---
.../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 4
1 file changed, 4 insertions(+)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 9e86fedb0e9..8307489c604 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -2620,8 +2620,10 @@ class BeamModulePlugin implements Plugin {
expansionServiceOpts.put("java_port", javaExpansionPort)
// setup test env
def serviceArgs =
project.project(':sdks:python').mapToArgString(expansionServiceOpts)
+System.err.println '** xyz123 BeamModulePlugin: starting
transform service ...'
executable 'sh'
args '-c', ". ${project.ext.envdir}/bin/activate &&
$pythonDir/scripts/run_expansion_services.sh stop --group_id ${project.name} &&
$pythonDir/scripts/run_expansion_services.sh start $serviceArgs"
+System.err.println '** xyz123 BeamModulePlugin: DONE
starting transform service ...'
}
}
}
@@ -2658,8 +2660,10 @@ class BeamModulePlugin implements Plugin {
// 3. Shuts down the expansion service
def cleanupTask = project.tasks.register(config.name+'Cleanup', Exec) {
// teardown test env
+System.err.println '** xyz123 BeamModulePlugin: stopping
transform service ...'
executable 'sh'
args '-c', ". ${project.ext.envdir}/bin/activate &&
$pythonDir/scripts/run_expansion_services.sh stop --group_id ${project.name}"
+System.err.println '** xyz123 BeamModulePlugin: DONE stopping
transform service ...'
}
setupTask.configure {finalizedBy cleanupTask}
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 04412a37685 Updating config from bot
04412a37685 is described below
commit 04412a3768580aca87322df2ce06e19ad619a486
Author: github-actions
AuthorDate: Mon Feb 12 21:32:37 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-30290.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30290.json
b/scripts/ci/pr-bot/state/pr-state/pr-30290.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-30290.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch transform_service_test_suite updated: Updates
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a commit to branch transform_service_test_suite in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/transform_service_test_suite by this push: new 234ea46c981 Updates 234ea46c981 is described below commit 234ea46c981d239a0df4fe85c966d675ed279e9b Author: Chamikara Jayalath AuthorDate: Mon Feb 12 13:24:30 2024 -0800 Updates --- .github/workflows/beam_PostCommit_TransformService_Direct.yml | 6 ++ sdks/python/scripts/run_transform_service.sh | 1 - 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/.github/workflows/beam_PostCommit_TransformService_Direct.yml b/.github/workflows/beam_PostCommit_TransformService_Direct.yml index 1497681ae99..fab980f63d5 100644 --- a/.github/workflows/beam_PostCommit_TransformService_Direct.yml +++ b/.github/workflows/beam_PostCommit_TransformService_Direct.yml @@ -62,7 +62,7 @@ jobs: matrix: job_name: ["beam_PostCommit_TransformService_Direct"] job_phrase: ["Run TransformService_Direct PostCommit"] -python_version: ['3.8','3.11'] +python_version: ['3.11'] steps: - uses: actions/checkout@v4 - name: Setup repository @@ -75,9 +75,7 @@ jobs: uses: ./.github/actions/setup-environment-action with: java-version: 11 - python-version: | -3.8 -3.11 + python-version: 3.11 - name: run TransformService Direct script uses: ./.github/actions/gradle-command-self-hosted-action with: diff --git a/sdks/python/scripts/run_transform_service.sh b/sdks/python/scripts/run_transform_service.sh index c86490f34fb..a4f63bd6288 100755 --- a/sdks/python/scripts/run_transform_service.sh +++ b/sdks/python/scripts/run_transform_service.sh @@ -87,7 +87,6 @@ case $STARTSTOP in echo "Starting the transform service for project $GROUP_ID at port $EXTERNAL_PORT for Beam version $BEAM_VERSION_DOCKER transform service startup jar is $TRANSFORM_SERVICE_LAUNCHER_JAR" java -jar $TRANSFORM_SERVICE_LAUNCHER_JAR --project_name $GROUP_ID --port $EXTERNAL_PORT --beam_version $BEAM_VERSION_DOCKER --command up >$TEMP_DIR/$FILE_BASE-java1.log 2>&1
(beam) branch transform_service_test_suite created (now d7ed73045da)
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a change to branch transform_service_test_suite in repository https://gitbox.apache.org/repos/asf/beam.git at d7ed73045da Debugging the Transform Service test suite This branch includes the following new commits: new d7ed73045da Debugging the Transform Service test suite The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Debugging the Transform Service test suite
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a commit to branch transform_service_test_suite in repository https://gitbox.apache.org/repos/asf/beam.git commit d7ed73045da1496d5023251c1c1a0214b8c9d61e Author: Chamikara Jayalath AuthorDate: Mon Feb 12 11:15:24 2024 -0800 Debugging the Transform Service test suite --- sdks/python/scripts/run_transform_service.sh | 14 ++ 1 file changed, 14 insertions(+) diff --git a/sdks/python/scripts/run_transform_service.sh b/sdks/python/scripts/run_transform_service.sh index b71d67707e0..c86490f34fb 100755 --- a/sdks/python/scripts/run_transform_service.sh +++ b/sdks/python/scripts/run_transform_service.sh @@ -16,6 +16,8 @@ #limitations under the License. # +echo " xyz123 run_transform_service.sh: 1" + read -r -d '' USAGE <$TEMP_DIR/$FILE_BASE-java1.log 2>&1
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new a017c10d13a Updating config from bot
a017c10d13a is described below
commit a017c10d13ac2268850575a4e96e53919cf58f83
Author: github-actions
AuthorDate: Mon Feb 12 18:54:47 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
index 46e2102e53f..81a56b4846c 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
@@ -3,7 +3,7 @@
"dateOfLastReviewAssignment": {
"lukecwik": 1680701852535,
"kennknowles": 170723738,
-"robertwb": 1707233990808,
+"robertwb": 1707764084416,
"kileys": 1674428678843,
"apilloud": 1678822446183,
"Abacn": 1707425392750,
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new c69a608d53a Updating config from bot c69a608d53a is described below commit c69a608d53aa06bd3404e8ad7074e27c448b6740 Author: github-actions AuthorDate: Mon Feb 12 18:54:49 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json b/scripts/ci/pr-bot/state/reviewers-for-label-io.json index 69c8a5adda5..f1c86dc5c00 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json @@ -6,7 +6,7 @@ "pabloem": 1691787951165, "Abacn": 1707503950521, "ahmedabu98": 1707343533957, -"bvolpato": 1707233990808, +"bvolpato": 1707764084416, "manavgarg": 1690826779210, "damondouglas": 170723738 }
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new e22b6699644 Updating config from bot
e22b6699644 is described below
commit e22b669964448b93fcb34595e9d1bfbfa20e91d7
Author: github-actions
AuthorDate: Mon Feb 12 18:54:46 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-30264.json | 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30264.json
b/scripts/ci/pr-bot/state/pr-state/pr-30264.json
index 242a48d7d3b..d0bf49e2c91 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-30264.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-30264.json
@@ -1,7 +1,10 @@
{
"commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "reviewersAssignedForLabels": {
+"java": "robertwb",
+"io": "bvolpato"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
(beam) branch master updated: fixed documentation in sdks/python/apache_beam/io/gcp/pubsub.py WriteToPubSub (#30285)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 47f777856bf fixed documentation in sdks/python/apache_beam/io/gcp/pubsub.py WriteToPubSub (#30285) 47f777856bf is described below commit 47f777856bff21e0c281f71402f879ef5cad6ab9 Author: Dima I <79413560+divkov...@users.noreply.github.com> AuthorDate: Mon Feb 12 13:27:01 2024 -0500 fixed documentation in sdks/python/apache_beam/io/gcp/pubsub.py WriteToPubSub (#30285) * fixed documentation in sdks/python/apache_beam/io/gcp/pubsub.py writetopubsub * Update build.gradle - I accidentally removed a character in a previous commit * Update pubsub.py * fixed linting error --- sdks/python/apache_beam/io/gcp/pubsub.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/io/gcp/pubsub.py b/sdks/python/apache_beam/io/gcp/pubsub.py index d0785fa1f21..cec65bc530f 100644 --- a/sdks/python/apache_beam/io/gcp/pubsub.py +++ b/sdks/python/apache_beam/io/gcp/pubsub.py @@ -337,7 +337,9 @@ class WriteToPubSub(PTransform): """Initializes ``WriteToPubSub``. Args: - topic: Cloud Pub/Sub topic in the form "/topics//". + topic: + Cloud Pub/Sub topic in the form + "projects//topics/". with_attributes: True - input elements will be :class:`~PubsubMessage` objects. False - input elements will be of type ``bytes`` (message
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 58e6f2f0627 Updating config from bot
58e6f2f0627 is described below
commit 58e6f2f0627e074598ad9fac7d510dc16209b183
Author: github-actions
AuthorDate: Mon Feb 12 18:06:25 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
index bd9c05f5f56..69c8a5adda5 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
@@ -2,7 +2,7 @@
"label": "io",
"dateOfLastReviewAssignment": {
"chamikaramj": 1707425392751,
-"johnjcasey": 1706997928741,
+"johnjcasey": 1707761180436,
"pabloem": 1691787951165,
"Abacn": 1707503950521,
"ahmedabu98": 1707343533957,
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new a66f652929e Updating config from bot
a66f652929e is described below
commit a66f652929e1b21d0c1e683471087281919ab266
Author: github-actions
AuthorDate: Mon Feb 12 18:06:22 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-30285.json | 9 ++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30285.json
b/scripts/ci/pr-bot/state/pr-state/pr-30285.json
index 242a48d7d3b..14513af1e4e 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-30285.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-30285.json
@@ -1,7 +1,10 @@
{
- "commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "damccorm",
+"io": "johnjcasey"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new e1cf61eab68 Updating config from bot e1cf61eab68 is described below commit e1cf61eab68ecf279b35199003b61d62af32 Author: github-actions AuthorDate: Mon Feb 12 18:06:24 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 6ac533f2bee..4aff446b813 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -8,7 +8,7 @@ "tvalentyn": 1706897164657, "pabloem": 1681281324703, "y1chi": 1667002607045, -"damccorm": 1706876018447, +"damccorm": 1707761180436, "jrmccluskey": 1706915144832, "riteshghorse": 1707135221120, "liferoad": 1707343542374
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 759a1ca9754 Updating config from bot
759a1ca9754 is described below
commit 759a1ca9754891a4f10a207871c6af9010a01d1c
Author: github-actions
AuthorDate: Mon Feb 12 15:49:57 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-30262.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-30262.json
b/scripts/ci/pr-bot/state/pr-state/pr-30262.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-30262.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch master updated: Remove unused StreamingDataflowWorker parameter (#30256)
This is an automated email from the ASF dual-hosted git repository.
scwhittle pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 371576a3b17 Remove unused StreamingDataflowWorker parameter (#30256)
371576a3b17 is described below
commit 371576a3b17b940380192378848dd00c55d0cc19
Author: martin trieu
AuthorDate: Mon Feb 12 06:33:20 2024 -0800
Remove unused StreamingDataflowWorker parameter (#30256)
---
.../dataflow/worker/StreamingDataflowWorker.java | 2 +-
.../worker/StreamingModeExecutionContext.java | 23 ++---
.../dataflow/worker/counters/NameContext.java | 29 --
.../dataflow/worker/streaming/StageInfo.java | 7 +++---
.../worker/DataflowExecutionContextTest.java | 9 +++
.../worker/StreamingModeExecutionContextTest.java | 14 +++
.../dataflow/worker/WorkerCustomSourcesTest.java | 4 +--
7 files changed, 46 insertions(+), 42 deletions(-)
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
index bca14923cfc..2f9e18cde67 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
@@ -991,7 +991,7 @@ public class StreamingDataflowWorker {
StageInfo stageInfo =
stageInfoMap.computeIfAbsent(
-mapTask.getStageName(), s -> StageInfo.create(s,
mapTask.getSystemName(), this));
+mapTask.getStageName(), s -> StageInfo.create(s,
mapTask.getSystemName()));
ExecutionState executionState = null;
String counterName = "dataflow_source_bytes_processed-" +
mapTask.getSystemName();
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
index 5b18e29293e..2e9e7e608a5 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
@@ -42,6 +42,7 @@ import org.apache.beam.runners.core.TimerInternals;
import org.apache.beam.runners.core.TimerInternals.TimerData;
import org.apache.beam.runners.core.metrics.ExecutionStateTracker;
import
org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState;
+import
org.apache.beam.runners.dataflow.worker.DataflowOperationContext.DataflowExecutionState;
import
org.apache.beam.runners.dataflow.worker.StreamingModeExecutionContext.StepContext;
import org.apache.beam.runners.dataflow.worker.counters.CounterFactory;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
@@ -440,21 +441,18 @@ public class StreamingModeExecutionContext extends
DataflowExecutionContextExamples: "MapElements/Map",
"BigShuffle.GroupByFirstNBytes/GroupByKey/Reify"
*/
public abstract @Nullable String userName();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+public abstract Builder setStageName(String value);
+
+public abstract Builder setOriginalName(@Nullable String value);
+
+public abstract Builder setSystemName(@Nullable String value);
+
+public abstract Builder setUserName(@Nullable String value);
+
+public abstract NameContext build();
+ }
}
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/StageInfo.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/StageInfo.java
index 64c97dcac51..cb6cbec7d4b 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/StageInfo.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/StageInfo.java
@@ -39,15 +39,14 @@ import
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Iterab
/** Contains a few of the stage specific fields. E.g. metrics container
registry, counters etc. */
@AutoValue
public abstract class StageInfo {
- public static StageInfo create(
- String stageName, String systemName, StreamingDataflowWorker worker) {
-NameContext nameContext = NameContext.create(stageName, null, systemName,
null);
+ public static StageInfo create(String stageName, String systemName) {
+NameContext
(beam) branch remove_cogbk updated (b8dd486cf7b -> e0fc00b8a6d)
This is an automated email from the ASF dual-hosted git repository. anandinguva pushed a change to branch remove_cogbk in repository https://gitbox.apache.org/repos/asf/beam.git from b8dd486cf7b Update sdks/python/apache_beam/ml/transforms/handlers.py add e0fc00b8a6d Make _DataCoder internal No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/ml/transforms/handlers.py | 9 + 1 file changed, 5 insertions(+), 4 deletions(-)
