[beam] branch asf-site updated: Publishing website 2019/12/28 03:14:40 at commit a861278
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 5514250 Publishing website 2019/12/28 03:14:40 at commit a861278 5514250 is described below commit 5514250432722df9b6fcf01368d519404c3adba7 Author: jenkins AuthorDate: Sat Dec 28 03:14:40 2019 + Publishing website 2019/12/28 03:14:40 at commit a861278 --- .../generated-content/documentation/runners/flink/index.html | 10 -- 1 file changed, 10 deletions(-) diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html index 029e79b..fcd6d14 100644 --- a/website/generated-content/documentation/runners/flink/index.html +++ b/website/generated-content/documentation/runners/flink/index.html @@ -626,11 +626,6 @@ which should be called before running the tests. Default: EXACTLY_ONCE - disableMetricAccumulator - By default, uses Flink accumulators to store the metrics which allows to query metrics from the PipelineResult. If set to true, metrics will still be reported but can't be queried via PipelineResult. This saves network and memory. - Default: false - - disableMetrics Disable Beam metrics in Flink Runner Default: false @@ -780,11 +775,6 @@ which should be called before running the tests. Default: EXACTLY_ONCE - disable_metric_accumulator - By default, uses Flink accumulators to store the metrics which allows to query metrics from the PipelineResult. If set to true, metrics will still be reported but can't be queried via PipelineResult. This saves network and memory. - Default: false - - disable_metrics Disable Beam metrics in Flink Runner Default: false
[beam] branch master updated (fb0353e -> a861278)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fb0353e Merge pull request #10216 from y1chi/allowed_lateness add ea2a3f6 [BEAM-8962] Report Flink metric accumulator only when pipeline ends new a861278 Merge pull request #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/flink/FlinkPipelineOptions.java | 9 -- .../flink/metrics/FlinkMetricContainer.java| 49 +- .../runners/flink/metrics/MetricsAccumulator.java | 6 +- .../translation/functions/FlinkDoFnFunction.java | 10 +- .../functions/FlinkExecutableStageFunction.java| 9 +- .../functions/FlinkStatefulDoFnFunction.java | 11 +-- .../translation/wrappers/SourceInputFormat.java| 8 +- .../wrappers/streaming/DoFnOperator.java | 4 +- .../streaming/io/UnboundedSourceWrapper.java | 9 +- .../runners/flink/FlinkPipelineOptionsTest.java| 1 - .../flink/metrics/FlinkMetricContainerTest.java| 14 +-- .../functions/FlinkDoFnFunctionTest.java | 106 + .../FlinkExecutableStageFunctionTest.java | 16 .../functions/FlinkStatefulDoFnFunctionTest.java | 106 + .../wrappers/SourceInputFormatTest.java| 64 + .../wrappers/streaming/DoFnOperatorTest.java | 77 +-- .../streaming/io/UnboundedSourceWrapperTest.java | 30 ++ .../src/_includes/flink_java_pipeline_options.html | 5 - .../_includes/flink_python_pipeline_options.html | 5 - 19 files changed, 426 insertions(+), 113 deletions(-) create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunctionTest.java create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunctionTest.java create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/SourceInputFormatTest.java
[beam] 01/01: Merge pull request #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a8612788cb77a8b9d79bfbe44321c941d38322e7 Merge: fb0353e ea2a3f6 Author: Thomas Weise AuthorDate: Fri Dec 27 19:08:19 2019 -0800 Merge pull request #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends .../beam/runners/flink/FlinkPipelineOptions.java | 9 -- .../flink/metrics/FlinkMetricContainer.java| 49 +- .../runners/flink/metrics/MetricsAccumulator.java | 6 +- .../translation/functions/FlinkDoFnFunction.java | 10 +- .../functions/FlinkExecutableStageFunction.java| 9 +- .../functions/FlinkStatefulDoFnFunction.java | 11 +-- .../translation/wrappers/SourceInputFormat.java| 8 +- .../wrappers/streaming/DoFnOperator.java | 4 +- .../streaming/io/UnboundedSourceWrapper.java | 9 +- .../runners/flink/FlinkPipelineOptionsTest.java| 1 - .../flink/metrics/FlinkMetricContainerTest.java| 14 +-- .../functions/FlinkDoFnFunctionTest.java | 106 + .../FlinkExecutableStageFunctionTest.java | 16 .../functions/FlinkStatefulDoFnFunctionTest.java | 106 + .../wrappers/SourceInputFormatTest.java| 64 + .../wrappers/streaming/DoFnOperatorTest.java | 77 +-- .../streaming/io/UnboundedSourceWrapperTest.java | 30 ++ .../src/_includes/flink_java_pipeline_options.html | 5 - .../_includes/flink_python_pipeline_options.html | 5 - 19 files changed, 426 insertions(+), 113 deletions(-)
[beam] branch asf-site updated: Publishing website 2019/12/27 22:35:29 at commit fb0353e
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new c00a9ca Publishing website 2019/12/27 22:35:29 at commit fb0353e c00a9ca is described below commit c00a9caa5c419d6b90b67624023a502567b482fe Author: jenkins AuthorDate: Fri Dec 27 22:35:29 2019 + Publishing website 2019/12/27 22:35:29 at commit fb0353e --- website/generated-content/documentation/programming-guide/index.html | 1 + 1 file changed, 1 insertion(+) diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html index 774de1d..fd56b21 100644 --- a/website/generated-content/documentation/programming-guide/index.html +++ b/website/generated-content/documentation/programming-guide/index.html @@ -3660,6 +3660,7 @@ trigger stops executing FixedWindows(1 * 60), trigger=AfterWatermark( late=AfterProcessingTime(10 * 60)), +allowed_lateness=10, accumulation_mode=AccumulationMode.DISCARDING)
[beam] branch master updated: [BEAM-8824] Add support to allow specify window allowed_lateness in python sdk
This is an automated email from the ASF dual-hosted git repository.
goenka pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 79b6e5b [BEAM-8824] Add support to allow specify window
allowed_lateness in python sdk
new fb0353e Merge pull request #10216 from y1chi/allowed_lateness
79b6e5b is described below
commit 79b6e5bb863c8cf96b690d1a96c8988d8bec72be
Author: Yichi Zhang
AuthorDate: Tue Nov 19 10:16:38 2019 -0800
[BEAM-8824] Add support to allow specify window allowed_lateness in python
sdk
---
.../apache_beam/examples/snippets/snippets_test.py | 2 ++
.../testing/data/trigger_transcripts.yaml | 22
.../python/apache_beam/testing/test_stream_test.py | 2 +-
sdks/python/apache_beam/transforms/core.py | 39 +-
sdks/python/apache_beam/transforms/trigger.py | 5 ++-
sdks/python/apache_beam/transforms/trigger_test.py | 29 ++--
6 files changed, 78 insertions(+), 21 deletions(-)
diff --git a/sdks/python/apache_beam/examples/snippets/snippets_test.py
b/sdks/python/apache_beam/examples/snippets/snippets_test.py
index f0f53e2..38bcb88 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets_test.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets_test.py
@@ -958,6 +958,7 @@ class SnippetsTest(unittest.TestCase):
| 'pair_with_one' >> beam.Map(lambda x: (x, 1))
| WindowInto(FixedWindows(15),
trigger=trigger,
+ allowed_lateness=20,
accumulation_mode=AccumulationMode.DISCARDING)
| 'group' >> beam.GroupByKey()
| 'count' >> beam.Map(
@@ -1014,6 +1015,7 @@ class SnippetsTest(unittest.TestCase):
FixedWindows(1 * 60),
trigger=AfterWatermark(
late=AfterProcessingTime(10 * 60)),
+ allowed_lateness=10,
accumulation_mode=AccumulationMode.DISCARDING)
# [END model_composite_triggers]
| 'group' >> beam.GroupByKey()
diff --git a/sdks/python/apache_beam/testing/data/trigger_transcripts.yaml
b/sdks/python/apache_beam/testing/data/trigger_transcripts.yaml
index cac0c74..b2d4e9a 100644
--- a/sdks/python/apache_beam/testing/data/trigger_transcripts.yaml
+++ b/sdks/python/apache_beam/testing/data/trigger_transcripts.yaml
@@ -30,6 +30,7 @@ name: fixed_default_late_data
window_fn: FixedWindows(10)
trigger_fn: Default
timestamp_combiner: OUTPUT_AT_EOW
+allowed_lateness: 100
transcript:
- input: [1, 2, 3, 10, 11, 25]
- watermark: 100
@@ -42,6 +43,26 @@ transcript:
- {window: [0, 9], values: [1, 2, 3, 7], timestamp: 9, late: true}
---
+name: fixed_drop_late_data_after_allowed_lateness
+window_fn: FixedWindows(10)
+trigger_fn: AfterWatermark(early=AfterCount(3), late=AfterCount(1))
+timestamp_combiner: OUTPUT_AT_EOW
+allowed_lateness: 20
+accumulation_mode: accumulating
+transcript:
+ - input: [1, 2, 10, 11, 80, 81]
+ - watermark: 100
+ - expect:
+ - {window: [0, 9], values: [1, 2], timestamp: 9, final: false}
+ - {window: [10, 19], values: [10, 11], timestamp: 19}
+ - {window: [80, 89], values: [80, 81], timestamp: 89, late: false}
+ - input: [7, 8] # no output
+ - input: [17, 18] # no output
+ - input: [82]
+ - expect:
+ - {window: [80, 89], values: [80, 81, 82], timestamp: 89, late: true}
+
+---
name: timestamp_combiner_earliest
window_fn: FixedWindows(10)
trigger_fn: Default
@@ -118,6 +139,7 @@ broken_on:
- SwitchingDirectRunner
window_fn: Sessions(10)
trigger_fn: AfterWatermark(early=AfterCount(2), late=AfterCount(3))
+allowed_lateness: 100
timestamp_combiner: OUTPUT_AT_EOW
transcript:
- input: [1, 2, 3]
diff --git a/sdks/python/apache_beam/testing/test_stream_test.py
b/sdks/python/apache_beam/testing/test_stream_test.py
index 26b54bd..bfadb5e 100644
--- a/sdks/python/apache_beam/testing/test_stream_test.py
+++ b/sdks/python/apache_beam/testing/test_stream_test.py
@@ -275,7 +275,7 @@ class TestStreamTest(unittest.TestCase):
p = TestPipeline(options=options)
records = (p
| test_stream
- | beam.WindowInto(FixedWindows(15))
+ | beam.WindowInto(FixedWindows(15), allowed_lateness=300)
| beam.Map(lambda x: ('k', x))
| beam.GroupByKey())
diff --git a/sdks/python/apache_beam/transforms/core.py
b/sdks/python/apache_beam/transforms/core.py
index 3cd5472..25cc91f 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -64,6 +64,7 @@ from apache_beam.typehints.trivial_inference import
element_type
from apache_beam.typehints.typehints import is_consistent_with
from apache_beam.utils import timestamp
from apache_beam.utils import urns
+from apache_beam.utils.timestamp
[beam] branch master updated (4c34799 -> 614ca4e)
This is an automated email from the ASF dual-hosted git repository. boyuanz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4c34799 Merge pull request #10454 Set TMPDIR for tox environments add aeae2f4 Revert "[BEAM-8932] Modify PubsubClient to use the proto message throughout." add 614ca4e Merge pull request #10470 from apache/revert-10331-master No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 6 +- .../beam/sdk/io/gcp/pubsub/PubsubClient.java | 139 - .../beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java | 29 +++-- .../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java| 9 +- .../beam/sdk/io/gcp/pubsub/PubsubJsonClient.java | 28 ++--- .../beam/sdk/io/gcp/pubsub/PubsubTestClient.java | 11 +- .../sdk/io/gcp/pubsub/PubsubUnboundedSink.java | 23 ++-- .../sdk/io/gcp/pubsub/PubsubUnboundedSource.java | 33 +++-- .../apache/beam/sdk/io/gcp/pubsub/TestPubsub.java | 20 +-- .../beam/sdk/io/gcp/pubsub/TestPubsubSignal.java | 6 +- .../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java | 1 - .../sdk/io/gcp/pubsub/PubsubGrpcClientTest.java| 19 ++- .../beam/sdk/io/gcp/pubsub/PubsubIOTest.java | 8 +- .../sdk/io/gcp/pubsub/PubsubJsonClientTest.java| 46 ++- .../sdk/io/gcp/pubsub/PubsubTestClientTest.java| 31 ++--- .../sdk/io/gcp/pubsub/PubsubUnboundedSinkTest.java | 39 ++ .../io/gcp/pubsub/PubsubUnboundedSourceTest.java | 28 ++--- 17 files changed, 226 insertions(+), 250 deletions(-)
[beam] branch master updated (90ada50 -> 4c34799)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 90ada50 Merge pull request #10468: [BEAM-9025] Update Dataflow Java container worker version add 2fece15 Set TMPDIR for tox environments add 4c34799 Merge pull request #10454 Set TMPDIR for tox environments No new revisions were added by this update. Summary of changes: sdks/python/tox.ini | 48 +--- 1 file changed, 9 insertions(+), 39 deletions(-)
[beam] 01/01: Revert "[BEAM-8932] Modify PubsubClient to use the proto message throughout."
This is an automated email from the ASF dual-hosted git repository.
boyuanz pushed a commit to branch revert-10331-master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit aeae2f417629c374cb025faa3a664a9646859a01
Author: Boyuan Zhang <36090911+boyua...@users.noreply.github.com>
AuthorDate: Fri Dec 27 10:39:26 2019 -0800
Revert "[BEAM-8932] Modify PubsubClient to use the proto message
throughout."
---
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 6 +-
.../beam/sdk/io/gcp/pubsub/PubsubClient.java | 139 -
.../beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java | 29 +++--
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java| 9 +-
.../beam/sdk/io/gcp/pubsub/PubsubJsonClient.java | 28 ++---
.../beam/sdk/io/gcp/pubsub/PubsubTestClient.java | 11 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSink.java | 23 ++--
.../sdk/io/gcp/pubsub/PubsubUnboundedSource.java | 33 +++--
.../apache/beam/sdk/io/gcp/pubsub/TestPubsub.java | 20 +--
.../beam/sdk/io/gcp/pubsub/TestPubsubSignal.java | 6 +-
.../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java | 1 -
.../sdk/io/gcp/pubsub/PubsubGrpcClientTest.java| 19 ++-
.../beam/sdk/io/gcp/pubsub/PubsubIOTest.java | 8 +-
.../sdk/io/gcp/pubsub/PubsubJsonClientTest.java| 46 ++-
.../sdk/io/gcp/pubsub/PubsubTestClientTest.java| 31 ++---
.../sdk/io/gcp/pubsub/PubsubUnboundedSinkTest.java | 39 ++
.../io/gcp/pubsub/PubsubUnboundedSourceTest.java | 28 ++---
17 files changed, 226 insertions(+), 250 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 7bdf719..f3ba610 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -365,7 +365,7 @@ class BeamModulePlugin implements Plugin {
def cassandra_driver_version = "3.6.0"
def classgraph_version = "4.8.56"
def generated_grpc_beta_version = "0.44.0"
-def generated_grpc_ga_version = "1.83.0"
+def generated_grpc_ga_version = "1.43.0"
def generated_grpc_dc_beta_version = "0.27.0-alpha"
def google_auth_version = "0.12.0"
def google_clients_version = "1.28.0"
@@ -384,7 +384,7 @@ class BeamModulePlugin implements Plugin {
def postgres_version = "42.2.2"
def powermock_version = "2.0.2"
def proto_google_common_protos_version = "1.17.0"
-def protobuf_version = "3.11.0"
+def protobuf_version = "3.6.0"
def quickcheck_version = "0.8"
def spark_version = "2.4.4"
def spark_structured_streaming_version = "2.4.0"
@@ -445,7 +445,7 @@ class BeamModulePlugin implements Plugin {
google_api_services_clouddebugger :
"com.google.apis:google-api-services-clouddebugger:v2-rev20181114-$google_clients_version",
google_api_services_cloudresourcemanager:
"com.google.apis:google-api-services-cloudresourcemanager:v1-rev20181015-$google_clients_version",
google_api_services_dataflow:
"com.google.apis:google-api-services-dataflow:v1b3-rev20190927-$google_clients_version",
-google_api_services_pubsub :
"com.google.apis:google-api-services-pubsub:v1-rev2019-$google_clients_version",
+google_api_services_pubsub :
"com.google.apis:google-api-services-pubsub:v1-rev20181213-$google_clients_version",
google_api_services_storage :
"com.google.apis:google-api-services-storage:v1-rev20181109-$google_clients_version",
google_auth_library_credentials :
"com.google.auth:google-auth-library-credentials:$google_auth_version",
google_auth_library_oauth2_http :
"com.google.auth:google-auth-library-oauth2-http:$google_auth_version",
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java
index 6f0f54d..07d6da6 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java
@@ -21,12 +21,10 @@ import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Prec
import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.client.util.DateTime;
-import com.google.auto.value.AutoValue;
-import com.google.protobuf.ByteString;
-import com.google.pubsub.v1.PubsubMessage;
import java.io.Closeable;
import java.io.IOException;
import java.io.Serializable;
+import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ThreadLocalRandom;
@@ -300,37 +298,59 @@
[beam] branch revert-10331-master created (now aeae2f4)
This is an automated email from the ASF dual-hosted git repository. boyuanz pushed a change to branch revert-10331-master in repository https://gitbox.apache.org/repos/asf/beam.git. at aeae2f4 Revert "[BEAM-8932] Modify PubsubClient to use the proto message throughout." This branch includes the following new commits: new aeae2f4 Revert "[BEAM-8932] Modify PubsubClient to use the proto message throughout." The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
