[GitHub] [beam] robertwb commented on issue #11048: [BEAM-9433] Create expansion service artifact for common Java IOs.
robertwb commented on issue #11048: [BEAM-9433] Create expansion service artifact for common Java IOs. URL: https://github.com/apache/beam/pull/11048#issuecomment-599006983 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] stale[bot] commented on issue #10165: [BEAM-7390] Add code snippet for GroupIntoBatches
stale[bot] commented on issue #10165: [BEAM-7390] Add code snippet for GroupIntoBatches URL: https://github.com/apache/beam/pull/10165#issuecomment-599001723 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the d...@beam.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] wenchenglu commented on a change in pull request #11075: [BEAM-9421] Website section that describes getting predictions using AI Platform Prediciton
wenchenglu commented on a change in pull request #11075: [BEAM-9421] Website
section that describes getting predictions using AI Platform Prediciton
URL: https://github.com/apache/beam/pull/11075#discussion_r392549961
##
File path: website/src/documentation/patterns/ai-platform.md
##
@@ -0,0 +1,87 @@
+---
+layout: section
+title: "AI Platform integration patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/ai-platform/
+---
+
+
+# AI Platform integration patterns
+
+This page describes common patterns in pipelines with Google AI Platform
transforms.
+
+
+ Adapt for:
+
+Java SDK
+Python SDK
+
+
+
+## Getting predictions
+
+This section shows how to use a cloud-hosted machine learning model to make
predictions about new data using Google Cloud AI Platform Prediction within
Beam's pipeline.
+
+[tfx_bsl](https://github.com/tensorflow/tfx-bsl) is a library that provides
`RunInference` Beam's PTransform. `RunInference` is a PTransform able to
perform two types of inference. One of them can use a service endpoint. When
using a service endpoint, the transform takes a PCollection of type
`tf.train.Example` and, for each element, sends a request to Google Cloud AI
Platform Prediction service. The transform produces a PCollection of type
`PredictLog` which contains predictions.
+
+Before getting started, deploy a machine learning model to the cloud. The
cloud service manages the infrastructure needed to handle prediction requests
in both efficient and scalable way. Only Tensorflow models are supported. For
more information, see [Exporting a SavedModel for
prediction](https://cloud.google.com/ai-platform/prediction/docs/exporting-savedmodel-for-prediction).
+
+Once a machine learning model is deployed, prepare a list of instances to get
predictions for.
+
+Here is an example of a pipeline that reads input instances from the file,
converts JSON objects to `tf.train.Example` objects and sends data to the
service. The content of a file can look like this:
+
+```
+{"input": "the quick brown"}
+{"input": "la bruja le"}
+```
+
+The example creates `tf.train.BytesList` instances, thus it expects byte-like
strings as input, but other data types, like `tf.train.FloatList` and
`tf.train.Int64List`, are also supported by the transform. To send binary data,
make sure that the name of an input ends in `_bytes`.
+
+Here is the code:
+
+{:.language-java}
+```java
+// Getting predictions is not yet available for Java. [BEAM-9501]
+```
+
+{:.language-py}
+```py
+import json
+
+import apache_beam as beam
+
+import tensorflow as tf
+from tfx_bsl.beam.run_inference import RunInference
+from tfx_bsl.proto import model_spec_pb2
+
+def convert_json_to_tf_example(json_obj):
+ dict_ = json.loads(json_obj)
+ for name, text in dict_.items():
+ value = tf.train.Feature(bytes_list=tf.train.BytesList(
+value=[text.encode('utf-8')]))
+ feature = {name: value}
+ return tf.train.Example(features=tf.train.Features(feature=feature))
+
+with beam.Pipeline() as p:
+ _ = (p
+ | beam.io.ReadFromText('gs://my-bucket/samples.json')
+ | beam.Map(convert_json_to_tf_example)
+ | RunInference(
Review comment:
Is RunInference a HTTP call? is there a plan to support gRPC in the future?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] wenchenglu commented on a change in pull request #11075: [BEAM-9421] Website section that describes getting predictions using AI Platform Prediciton
wenchenglu commented on a change in pull request #11075: [BEAM-9421] Website section that describes getting predictions using AI Platform Prediciton URL: https://github.com/apache/beam/pull/11075#discussion_r392548549 ## File path: website/src/documentation/patterns/ai-platform.md ## @@ -0,0 +1,87 @@ +--- +layout: section +title: "AI Platform integration patterns" +section_menu: section-menu/documentation.html +permalink: /documentation/patterns/ai-platform/ +--- + + +# AI Platform integration patterns + +This page describes common patterns in pipelines with Google AI Platform transforms. + + + Adapt for: + +Java SDK +Python SDK + + + +## Getting predictions + +This section shows how to use a cloud-hosted machine learning model to make predictions about new data using Google Cloud AI Platform Prediction within Beam's pipeline. + +[tfx_bsl](https://github.com/tensorflow/tfx-bsl) is a library that provides `RunInference` Beam's PTransform. `RunInference` is a PTransform able to perform two types of inference. One of them can use a service endpoint. When using a service endpoint, the transform takes a PCollection of type `tf.train.Example` and, for each element, sends a request to Google Cloud AI Platform Prediction service. The transform produces a PCollection of type `PredictLog` which contains predictions. Review comment: "one of them can use a service endpoint", just out of curiosity, what is the other type of inference This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] wenchenglu commented on a change in pull request #11075: [BEAM-9421] Website section that describes getting predictions using AI Platform Prediciton
wenchenglu commented on a change in pull request #11075: [BEAM-9421] Website section that describes getting predictions using AI Platform Prediciton URL: https://github.com/apache/beam/pull/11075#discussion_r392549097 ## File path: website/src/documentation/patterns/ai-platform.md ## @@ -0,0 +1,87 @@ +--- +layout: section +title: "AI Platform integration patterns" +section_menu: section-menu/documentation.html +permalink: /documentation/patterns/ai-platform/ +--- + + +# AI Platform integration patterns + +This page describes common patterns in pipelines with Google AI Platform transforms. + + + Adapt for: + +Java SDK +Python SDK + + + +## Getting predictions + +This section shows how to use a cloud-hosted machine learning model to make predictions about new data using Google Cloud AI Platform Prediction within Beam's pipeline. + +[tfx_bsl](https://github.com/tensorflow/tfx-bsl) is a library that provides `RunInference` Beam's PTransform. `RunInference` is a PTransform able to perform two types of inference. One of them can use a service endpoint. When using a service endpoint, the transform takes a PCollection of type `tf.train.Example` and, for each element, sends a request to Google Cloud AI Platform Prediction service. The transform produces a PCollection of type `PredictLog` which contains predictions. + +Before getting started, deploy a machine learning model to the cloud. The cloud service manages the infrastructure needed to handle prediction requests in both efficient and scalable way. Only Tensorflow models are supported. For more information, see [Exporting a SavedModel for prediction](https://cloud.google.com/ai-platform/prediction/docs/exporting-savedmodel-for-prediction). Review comment: Does that mean users will need to separately deploy a model first? will it be a better user experience if some initial setup stage for Beam can call AI Platform prediction public API to deploy model and get the service endpoint? Also, for batch inference scenario, model deployment is a one-off job, users then need to un-deploy models to avoid unnecessary charges. Should they do that separately, or is there a BEAM final stage we could plug in a API call to delete that model deployment? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] wenchenglu commented on a change in pull request #11075: [BEAM-9421] Website section that describes getting predictions using AI Platform Prediciton
wenchenglu commented on a change in pull request #11075: [BEAM-9421] Website
section that describes getting predictions using AI Platform Prediciton
URL: https://github.com/apache/beam/pull/11075#discussion_r392549383
##
File path: website/src/documentation/patterns/ai-platform.md
##
@@ -0,0 +1,87 @@
+---
+layout: section
+title: "AI Platform integration patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/ai-platform/
+---
+
+
+# AI Platform integration patterns
+
+This page describes common patterns in pipelines with Google AI Platform
transforms.
+
+
+ Adapt for:
+
+Java SDK
+Python SDK
+
+
+
+## Getting predictions
+
+This section shows how to use a cloud-hosted machine learning model to make
predictions about new data using Google Cloud AI Platform Prediction within
Beam's pipeline.
+
+[tfx_bsl](https://github.com/tensorflow/tfx-bsl) is a library that provides
`RunInference` Beam's PTransform. `RunInference` is a PTransform able to
perform two types of inference. One of them can use a service endpoint. When
using a service endpoint, the transform takes a PCollection of type
`tf.train.Example` and, for each element, sends a request to Google Cloud AI
Platform Prediction service. The transform produces a PCollection of type
`PredictLog` which contains predictions.
+
+Before getting started, deploy a machine learning model to the cloud. The
cloud service manages the infrastructure needed to handle prediction requests
in both efficient and scalable way. Only Tensorflow models are supported. For
more information, see [Exporting a SavedModel for
prediction](https://cloud.google.com/ai-platform/prediction/docs/exporting-savedmodel-for-prediction).
+
+Once a machine learning model is deployed, prepare a list of instances to get
predictions for.
+
+Here is an example of a pipeline that reads input instances from the file,
converts JSON objects to `tf.train.Example` objects and sends data to the
service. The content of a file can look like this:
+
+```
+{"input": "the quick brown"}
+{"input": "la bruja le"}
+```
+
+The example creates `tf.train.BytesList` instances, thus it expects byte-like
strings as input, but other data types, like `tf.train.FloatList` and
`tf.train.Int64List`, are also supported by the transform. To send binary data,
make sure that the name of an input ends in `_bytes`.
+
+Here is the code:
+
+{:.language-java}
+```java
+// Getting predictions is not yet available for Java. [BEAM-9501]
+```
+
+{:.language-py}
+```py
+import json
+
+import apache_beam as beam
+
+import tensorflow as tf
+from tfx_bsl.beam.run_inference import RunInference
+from tfx_bsl.proto import model_spec_pb2
+
+def convert_json_to_tf_example(json_obj):
+ dict_ = json.loads(json_obj)
+ for name, text in dict_.items():
+ value = tf.train.Feature(bytes_list=tf.train.BytesList(
+value=[text.encode('utf-8')]))
+ feature = {name: value}
+ return tf.train.Example(features=tf.train.Features(feature=feature))
+
+with beam.Pipeline() as p:
+ _ = (p
+ | beam.io.ReadFromText('gs://my-bucket/samples.json')
Review comment:
is this a single inference input item? or a set of items? FYI, I think AI
Platform Prediction supports both. For the latter one, a single HTTP request
will embed multiple input items, which might provide better throughput once AI
Platform Prediction enables batching mode in their model server.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] acrites commented on issue #10988: [BEAM-9382] Clean up of TestStreamTranscriptTests.
acrites commented on issue #10988: [BEAM-9382] Clean up of TestStreamTranscriptTests. URL: https://github.com/apache/beam/pull/10988#issuecomment-598997790 Apparently the failing test in the PreCommit has been "Failing for the past 11,721 builds (Since #0 )" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] suztomo edited a comment on issue #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky
suztomo edited a comment on issue #11090: [BEAM-9470]
:sdks:java:io:kinesis:test is flaky
URL: https://github.com/apache/beam/pull/11090#issuecomment-598926854
In the test case, the two iterators return the following data:
- firstIterator: `throw(e)`; `[a, b]`; `[]`; `[]`; `[]`; `[]` ...
- secondIterator: `[c]`; `[d]`; `[]`; `[]`; `[]` ...
The test case waits for 5 elements from shardReadersPool:
```
List fetchedRecords = new ArrayList<>();
while (fetchedRecords.size() < 4) {
CustomOptional nextRecord =
shardReadersPool.nextRecord();
if (nextRecord.isPresent()) {
fetchedRecords.add(nextRecord.get());
}
}
```
It does not care what are the content of 5 elements.
The test's expectation is to capture the following 4 elements:
```
verify(customRateLimitPolicy).onThrottle(same(e));
verify(customRateLimitPolicy).onSuccess(eq(ImmutableList.of(a, b)));
verify(customRateLimitPolicy).onSuccess(eq(singletonList(c)));
verify(customRateLimitPolicy).onSuccess(eq(singletonList(d)));
```
My assumption: there's no guarantee that 2 elements from firstIterator and
secondIterator are served evenly when 5 elements are consumed from them.
My suggestion: make the while loop condition to confirm the expected 5
elements (at least [a,b] and [d]) , not just the counting of the number of
items, are in `fetchedRecords` variable.
(Sorry if this is missing the point, I wrote this without checking @jfarr
's updated solution)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] suztomo edited a comment on issue #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky
suztomo edited a comment on issue #11090: [BEAM-9470]
:sdks:java:io:kinesis:test is flaky
URL: https://github.com/apache/beam/pull/11090#issuecomment-598926854
In the test case, the two iterators return the following data:
- firstIterator: `throw(e)`; `[a, b]`; `[]`; `[]`; `[]`; `[]` ...
- secondIterator: `[c]`; `[d]`; `[]`; `[]`; `[]` ...
The test case waits for 5 elements from shardReadersPool:
```
List fetchedRecords = new ArrayList<>();
while (fetchedRecords.size() < 4) {
CustomOptional nextRecord =
shardReadersPool.nextRecord();
if (nextRecord.isPresent()) {
fetchedRecords.add(nextRecord.get());
}
}
```
It does not care what are the content of 5 elements.
The test's expectation is to capture the following 4 elements:
```
verify(customRateLimitPolicy).onThrottle(same(e));
verify(customRateLimitPolicy).onSuccess(eq(ImmutableList.of(a, b)));
verify(customRateLimitPolicy).onSuccess(eq(singletonList(c)));
verify(customRateLimitPolicy).onSuccess(eq(singletonList(d)));
```
My assumption: there's no guarantee that 2 elements from firstIterator and
secondIterator are served evenly when 5 elements are consumed from them.
My suggestion: make the while loop until it confirms the expected 5 elements
(at least [a,b] and [d]) , not just the counting of them, are in
`fetchedRecords` variable.
(Sorry if this is missing the point, I wrote this without checking @jfarr
's updated solution)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[beam] branch master updated: [BEAM-8335] Final PR to merge the InteractiveBeam feature branch
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c14491d [BEAM-8335] Final PR to merge the InteractiveBeam feature
branch
new e95d066 Merge pull request #11109 from [BEAM-8335] Final PR to merge
the InteractiveBeam feature branch
c14491d is described below
commit c14491dd66cbb64a8f670e58434e0d916ab26a81
Author: Sam Rohde
AuthorDate: Thu Mar 12 10:23:59 2020 -0700
[BEAM-8335] Final PR to merge the InteractiveBeam feature branch
Change-Id: Icc5564bb5bc1d771394f3e9272984366f8242dfa
---
.../direct/consumer_tracking_pipeline_visitor.py | 7 +-
.../consumer_tracking_pipeline_visitor_test.py | 37 ++
.../apache_beam/runners/direct/test_stream_impl.py | 3 +-
.../runners/interactive/background_caching_job.py | 2 +-
.../interactive/background_caching_job_test.py | 34 ++
.../runners/interactive/caching/streaming_cache.py | 10 +-
.../runners/interactive/display/pipeline_graph.py | 1 +
.../runners/interactive/interactive_beam.py| 127 +
.../runners/interactive/interactive_runner.py | 34 +-
.../runners/interactive/interactive_runner_test.py | 61 +++---
.../runners/interactive/options/capture_control.py | 2 +-
.../runners/interactive/pipeline_fragment.py | 3 +
.../runners/interactive/pipeline_instrument.py | 3 +-
.../interactive/pipeline_instrument_test.py| 22 ++--
.../interactive/testing/pipeline_assertion.py | 4 +-
.../apache_beam/runners/interactive/utils.py | 8 +-
16 files changed, 315 insertions(+), 43 deletions(-)
diff --git
a/sdks/python/apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py
b/sdks/python/apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py
index acf48d8..4290b55 100644
---
a/sdks/python/apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py
+++
b/sdks/python/apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py
@@ -23,7 +23,6 @@ from __future__ import absolute_import
from typing import TYPE_CHECKING
from typing import Dict
-from typing import List
from typing import Set
from apache_beam import pvalue
@@ -44,7 +43,7 @@ class ConsumerTrackingPipelineVisitor(PipelineVisitor):
"""
def __init__(self):
self.value_to_consumers = {
-} # type: Dict[pvalue.PValue, List[AppliedPTransform]]
+} # type: Dict[pvalue.PValue, Set[AppliedPTransform]]
self.root_transforms = set() # type: Set[AppliedPTransform]
self.step_names = {} # type: Dict[AppliedPTransform, str]
@@ -68,8 +67,8 @@ class ConsumerTrackingPipelineVisitor(PipelineVisitor):
if isinstance(input_value, pvalue.PBegin):
self.root_transforms.add(applied_ptransform)
if input_value not in self.value_to_consumers:
- self.value_to_consumers[input_value] = []
-self.value_to_consumers[input_value].append(applied_ptransform)
+ self.value_to_consumers[input_value] = set()
+self.value_to_consumers[input_value].add(applied_ptransform)
else:
self.root_transforms.add(applied_ptransform)
self.step_names[applied_ptransform] = 's%d' % (self._num_transforms)
diff --git
a/sdks/python/apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py
b/sdks/python/apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py
index aebf452..ec9dd81 100644
---
a/sdks/python/apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py
+++
b/sdks/python/apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py
@@ -126,6 +126,43 @@ class
ConsumerTrackingPipelineVisitorTest(unittest.TestCase):
len(self.visitor.step_names), 3) # 2 creates + expanded CoGBK
self.assertEqual(len(self.visitor.views), 0)
+ def test_visitor_not_sorted(self):
+p = Pipeline()
+# pylint: disable=expression-not-assigned
+from apache_beam.testing.test_stream import TestStream
+p | TestStream().add_elements(['']) | beam.Map(lambda _: _)
+
+original_graph = p.to_runner_api(return_context=False)
+out_of_order_graph = p.to_runner_api(return_context=False)
+
+root_id = out_of_order_graph.root_transform_ids[0]
+root = out_of_order_graph.components.transforms[root_id]
+tmp = root.subtransforms[0]
+root.subtransforms[0] = root.subtransforms[1]
+root.subtransforms[1] = tmp
+
+p = beam.Pipeline().from_runner_api(
+out_of_order_graph, runner='BundleBasedDirectRunner', options=None)
+v_out_of_order = ConsumerTrackingPipelineVisitor()
+p.visit(v_out_of_order)
+
+p = beam.Pipeline().from_runner_api(
+original_graph, runner='BundleBasedDirectRunner', options=None)
+v_original = ConsumerTrackingPipelineVisitor()
+p.visit(v_original)
+
+# Convert to string to assert
[GitHub] [beam] pabloem merged pull request #11109: [BEAM-8335] Final PR to merge the InteractiveBeam feature branch
pabloem merged pull request #11109: [BEAM-8335] Final PR to merge the InteractiveBeam feature branch URL: https://github.com/apache/beam/pull/11109 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
chamikaramj commented on a change in pull request #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r392530455
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -752,6 +759,27 @@ private Debuggee registerDebuggee(CloudDebugger
debuggerClient, String uniquifie
}
}
+ private List stageArtifacts(RunnerApi.Pipeline pipeline) {
+ImmutableList.Builder filesToStageBuilder =
ImmutableList.builder();
+for (Map.Entry entry :
+pipeline.getComponents().getEnvironmentsMap().entrySet()) {
+ for (RunnerApi.ArtifactInformation info :
entry.getValue().getDependenciesList()) {
+if
(!BeamUrns.getUrn(RunnerApi.StandardArtifacts.Types.FILE).equals(info.getTypeUrn()))
{
Review comment:
Check the ROLE as well ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
chamikaramj commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#discussion_r392530581 ## File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py ## @@ -563,19 +562,31 @@ def _gcs_file_copy(self, from_path, to_path): with open(from_path, 'rb') as f: self.stage_file(to_folder, to_name, f, total_size=total_size) - def _stage_resources(self, options): + def _stage_resources(self, pipeline, options): Review comment: Please add a unit test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
chamikaramj commented on a change in pull request #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r392530472
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -752,6 +759,27 @@ private Debuggee registerDebuggee(CloudDebugger
debuggerClient, String uniquifie
}
}
+ private List stageArtifacts(RunnerApi.Pipeline pipeline) {
Review comment:
Please add a unit test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
chamikaramj commented on a change in pull request #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r392530711
##
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##
@@ -563,19 +562,31 @@ def _gcs_file_copy(self, from_path, to_path):
with open(from_path, 'rb') as f:
self.stage_file(to_folder, to_name, f, total_size=total_size)
- def _stage_resources(self, options):
+ def _stage_resources(self, pipeline, options):
google_cloud_options = options.view_as(GoogleCloudOptions)
if google_cloud_options.staging_location is None:
raise RuntimeError('The --staging_location option must be specified.')
if google_cloud_options.temp_location is None:
raise RuntimeError('The --temp_location option must be specified.')
+resources = []
+for _, env in pipeline.components.environments.items():
+ for dep in env.dependencies:
+if dep.type_urn != common_urns.artifact_types.FILE.urn:
+ raise RuntimeError('unsupported artifact type %s' % dep.type_urn)
+if dep.role_urn != common_urns.artifact_roles.STAGING_TO.urn:
+ raise RuntimeError('unsupported role type %s' % dep.role_urn)
+type_payload = beam_runner_api_pb2.ArtifactFilePayload.FromString(
+dep.type_payload)
+role_payload = \
Review comment:
Nit: Pls use ( instead of \ for formatting.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] tweise opened a new pull request #11134: Skip removal of manifest when no artifacts were staged.
tweise opened a new pull request #11134: Skip removal of manifest when no
artifacts were staged.
URL: https://github.com/apache/beam/pull/11134
Attempt to trim logging noise:
```
[flink-runner-job-invoker] WARN
org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService - Failed
to remove job staging directory for token
{"sessionId":"job_0a368558-2554-4476-8aa5-9fe42374c541","basePath":"/tmp/flinktestLfDytb"}:
{}
15:57:13 java.io.FileNotFoundException:
/tmp/flinktestLfDytb/job_0a368558-2554-4476-8aa5-9fe42374c541/MANIFEST (No such
file or directory)
```
https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Phrase/106/console
Post-Commit Tests Status (on master branch)
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)
| --- | [![Build
[GitHub] [beam] apilloud commented on issue #11131: [BEAM-8057] Reject Infinite or NaN literals at parse time
apilloud commented on issue #11131: [BEAM-8057] Reject Infinite or NaN literals at parse time URL: https://github.com/apache/beam/pull/11131#issuecomment-598972391 R: @robinyqiu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] apilloud commented on issue #11130: [BEAM-8070] Preserve type for empty array
apilloud commented on issue #11130: [BEAM-8070] Preserve type for empty array URL: https://github.com/apache/beam/pull/11130#issuecomment-598972372 R: @robinyqiu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] apilloud commented on issue #11133: [BEAM-7832] Uncollect takes arbitrary expressions
apilloud commented on issue #11133: [BEAM-7832] Uncollect takes arbitrary expressions URL: https://github.com/apache/beam/pull/11133#issuecomment-598972419 R: @robinyqiu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] apilloud commented on issue #11132: [BEAM-7832] Translate ZetaSQL joins without condition
apilloud commented on issue #11132: [BEAM-7832] Translate ZetaSQL joins without condition URL: https://github.com/apache/beam/pull/11132#issuecomment-598972397 R: @robinyqiu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] apilloud opened a new pull request #11133: [BEAM-7832] Uncollect takes arbitrary expressions
apilloud opened a new pull request #11133: [BEAM-7832] Uncollect takes arbitrary expressions URL: https://github.com/apache/beam/pull/11133 Uncollect can take arbitrary expressions, not just literals. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] apilloud opened a new pull request #11132: [BEAM-7832] Translate ZetaSQL joins without condition
apilloud opened a new pull request #11132: [BEAM-7832] Translate ZetaSQL joins without condition URL: https://github.com/apache/beam/pull/11132 This adds support for translating ZetaSQL joins without conditions instead of crashing. The pipeline still fails to run as we don't support unconditional joins. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[GitHub] [beam] apilloud opened a new pull request #11131: [BEAM-8057] Reject Infinite or NaN literals at parse time
apilloud opened a new pull request #11131: [BEAM-8057] Reject Infinite or NaN literals at parse time URL: https://github.com/apache/beam/pull/11131 We want to support this eventually, but provide a more useful error message for now. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[GitHub] [beam] apilloud opened a new pull request #11130: [BEAM-8070] Preserve type for empty array
apilloud opened a new pull request #11130: [BEAM-8070] Preserve type for empty array URL: https://github.com/apache/beam/pull/11130 Pass the type in instead of trying to infer. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] jfarr commented on a change in pull request #9758: [BEAM-8374] Enable returning missing PublishResult fields in SnsIO.Write
jfarr commented on a change in pull request #9758: [BEAM-8374] Enable returning
missing PublishResult fields in SnsIO.Write
URL: https://github.com/apache/beam/pull/9758#discussion_r392522322
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/coders/AwsCoders.java
##
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.coders;
+
+import com.amazonaws.ResponseMetadata;
+import com.amazonaws.http.HttpResponse;
+import com.amazonaws.http.SdkHttpMetadata;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.CustomCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/** {@link Coder}s for common AWS SDK objects. */
+public final class AwsCoders {
+
+ private AwsCoders() {}
+
+ /**
+ * Returns a new coder for ResponseMetadata.
+ *
+ * @return the ResponseMetadata coder
+ */
+ public static Coder responseMetadata() {
+return ResponseMetadataCoder.of();
+ }
+
+ /**
+ * Returns a new coder for SdkHttpMetadata.
+ *
+ * @return the SdkHttpMetadata coder
+ */
+ public static Coder sdkHttpMetadata() {
+return new SdkHttpMetadataCoder(true);
+ }
+
+ /**
+ * Returns a new coder for SdkHttpMetadata that does not serialize the
response headers.
+ *
+ * @return the SdkHttpMetadata coder
+ */
+ public static Coder sdkHttpMetadataWithoutHeaders() {
+return new SdkHttpMetadataCoder(false);
+ }
+
+ private static class ResponseMetadataCoder extends
AtomicCoder {
+
+private static final Coder> METADATA_ENCODER =
+NullableCoder.of(MapCoder.of(StringUtf8Coder.of(),
StringUtf8Coder.of()));
+private static final ResponseMetadataCoder INSTANCE = new
ResponseMetadataCoder();
+
+private ResponseMetadataCoder() {}
+
+public static ResponseMetadataCoder of() {
+ return INSTANCE;
+}
+
+@Override
+public void encode(ResponseMetadata value, OutputStream outStream)
+throws CoderException, IOException {
+ METADATA_ENCODER.encode(
+ ImmutableMap.of(ResponseMetadata.AWS_REQUEST_ID,
value.getRequestId()), outStream);
+}
+
+@Override
+public ResponseMetadata decode(InputStream inStream) throws
CoderException, IOException {
+ return new ResponseMetadata(METADATA_ENCODER.decode(inStream));
+}
+ }
+
+ private static class SdkHttpMetadataCoder extends
CustomCoder {
+
+private static final Coder STATUS_CODE_CODER = VarIntCoder.of();
+private static final Coder> HEADERS_ENCODER =
+NullableCoder.of(MapCoder.of(StringUtf8Coder.of(),
StringUtf8Coder.of()));
+
+private final boolean includeHeaders;
+
+protected SdkHttpMetadataCoder(boolean includeHeaders) {
+ this.includeHeaders = includeHeaders;
+}
+
+@Override
+public void encode(SdkHttpMetadata value, OutputStream outStream)
+throws CoderException, IOException {
+ STATUS_CODE_CODER.encode(value.getHttpStatusCode(), outStream);
+ if (includeHeaders) {
+HEADERS_ENCODER.encode(value.getHttpHeaders(), outStream);
+ }
+}
+
+@Override
+public SdkHttpMetadata decode(InputStream inStream) throws CoderException,
IOException {
+ final int httpStatusCode = STATUS_CODE_CODER.decode(inStream);
+ HttpResponse httpResponse = new HttpResponse(null, null);
+ httpResponse.setStatusCode(httpStatusCode);
+ if (includeHeaders) {
+Optional.ofNullable(HEADERS_ENCODER.decode(inStream))
+.ifPresent(
+headers ->
+headers.keySet().forEach(k -> httpResponse.addHeader(k,
headers.get(k;
+ }
+ return SdkHttpMetadata.from(httpResponse);
+}
+
+@Override
[GitHub] [beam] lukecwik commented on issue #9758: [BEAM-8374] Enable returning missing PublishResult fields in SnsIO.Write
lukecwik commented on issue #9758: [BEAM-8374] Enable returning missing PublishResult fields in SnsIO.Write URL: https://github.com/apache/beam/pull/9758#issuecomment-598964904 @iemejia Any final comments? Otherwise I intend to merge when the tests are green. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] lukecwik edited a comment on issue #9758: [BEAM-8374] Enable returning missing PublishResult fields in SnsIO.Write
lukecwik edited a comment on issue #9758: [BEAM-8374] Enable returning missing PublishResult fields in SnsIO.Write URL: https://github.com/apache/beam/pull/9758#issuecomment-598964756 > One last question @lukecwik, would you prefer to apply these changes in amazon-web-services2 in this PR? Or should we take that to a separate PR? That might be a good opportunity to discuss making some improvements to how errors are handled in this IO. In my opinion it's somewhat suboptimal that `IOException`s and the like are handled in the IO but HTTP error responses have to be handled in the caller, for example. I would prefer a separate PR so that people can start using this sooner then later. Also smaller PRs are easier to review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] lukecwik commented on issue #9758: [BEAM-8374] Enable returning missing PublishResult fields in SnsIO.Write
lukecwik commented on issue #9758: [BEAM-8374] Enable returning missing PublishResult fields in SnsIO.Write URL: https://github.com/apache/beam/pull/9758#issuecomment-598964756 > One last question @lukecwik, would you prefer to apply these changes in amazon-web-services2 in this PR? Or should we take that to a separate PR? That might be a good opportunity to discuss making some improvements to how errors are handled in this IO. In my opinion it's somewhat suboptimal that `IOException`s and the like are handled in the IO but HTTP error responses have to be handled in the caller, for example. I would prefer a separate PR so that people can start using this sooner then later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #9758: [BEAM-8374] Enable returning missing PublishResult fields in SnsIO.Write
lukecwik commented on a change in pull request #9758: [BEAM-8374] Enable
returning missing PublishResult fields in SnsIO.Write
URL: https://github.com/apache/beam/pull/9758#discussion_r392520094
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/coders/AwsCoders.java
##
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.coders;
+
+import com.amazonaws.ResponseMetadata;
+import com.amazonaws.http.HttpResponse;
+import com.amazonaws.http.SdkHttpMetadata;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.CustomCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/** {@link Coder}s for common AWS SDK objects. */
+public final class AwsCoders {
+
+ private AwsCoders() {}
+
+ /**
+ * Returns a new coder for ResponseMetadata.
+ *
+ * @return the ResponseMetadata coder
+ */
+ public static Coder responseMetadata() {
+return ResponseMetadataCoder.of();
+ }
+
+ /**
+ * Returns a new coder for SdkHttpMetadata.
+ *
+ * @return the SdkHttpMetadata coder
+ */
+ public static Coder sdkHttpMetadata() {
+return new SdkHttpMetadataCoder(true);
+ }
+
+ /**
+ * Returns a new coder for SdkHttpMetadata that does not serialize the
response headers.
+ *
+ * @return the SdkHttpMetadata coder
+ */
+ public static Coder sdkHttpMetadataWithoutHeaders() {
+return new SdkHttpMetadataCoder(false);
+ }
+
+ private static class ResponseMetadataCoder extends
AtomicCoder {
+
+private static final Coder> METADATA_ENCODER =
+NullableCoder.of(MapCoder.of(StringUtf8Coder.of(),
StringUtf8Coder.of()));
+private static final ResponseMetadataCoder INSTANCE = new
ResponseMetadataCoder();
+
+private ResponseMetadataCoder() {}
+
+public static ResponseMetadataCoder of() {
+ return INSTANCE;
+}
+
+@Override
+public void encode(ResponseMetadata value, OutputStream outStream)
+throws CoderException, IOException {
+ METADATA_ENCODER.encode(
+ ImmutableMap.of(ResponseMetadata.AWS_REQUEST_ID,
value.getRequestId()), outStream);
+}
+
+@Override
+public ResponseMetadata decode(InputStream inStream) throws
CoderException, IOException {
+ return new ResponseMetadata(METADATA_ENCODER.decode(inStream));
+}
+ }
+
+ private static class SdkHttpMetadataCoder extends
CustomCoder {
+
+private static final Coder STATUS_CODE_CODER = VarIntCoder.of();
+private static final Coder> HEADERS_ENCODER =
+NullableCoder.of(MapCoder.of(StringUtf8Coder.of(),
StringUtf8Coder.of()));
+
+private final boolean includeHeaders;
+
+protected SdkHttpMetadataCoder(boolean includeHeaders) {
+ this.includeHeaders = includeHeaders;
+}
+
+@Override
+public void encode(SdkHttpMetadata value, OutputStream outStream)
+throws CoderException, IOException {
+ STATUS_CODE_CODER.encode(value.getHttpStatusCode(), outStream);
+ if (includeHeaders) {
+HEADERS_ENCODER.encode(value.getHttpHeaders(), outStream);
+ }
+}
+
+@Override
+public SdkHttpMetadata decode(InputStream inStream) throws CoderException,
IOException {
+ final int httpStatusCode = STATUS_CODE_CODER.decode(inStream);
+ HttpResponse httpResponse = new HttpResponse(null, null);
+ httpResponse.setStatusCode(httpStatusCode);
+ if (includeHeaders) {
+Optional.ofNullable(HEADERS_ENCODER.decode(inStream))
+.ifPresent(
+headers ->
+headers.keySet().forEach(k -> httpResponse.addHeader(k,
headers.get(k;
+ }
+ return SdkHttpMetadata.from(httpResponse);
+}
+
+
[GitHub] [beam] jfarr commented on a change in pull request #9758: [BEAM-8374] Enable returning missing PublishResult fields in SnsIO.Write
jfarr commented on a change in pull request #9758: [BEAM-8374] Enable returning
missing PublishResult fields in SnsIO.Write
URL: https://github.com/apache/beam/pull/9758#discussion_r392519864
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsCoderProviderRegistrar.java
##
@@ -32,6 +32,7 @@
@Override
public List getCoderProviders() {
return ImmutableList.of(
-CoderProviders.forCoder(TypeDescriptor.of(PublishResult.class),
PublishResultCoder.of()));
+CoderProviders.forCoder(
+TypeDescriptor.of(PublishResult.class),
PublishResultCoders.defaultPublishResult()));
Review comment:
It's sort of the same reason as above, PublishResultCoders is a factory for
creating several different coder implementations so the factory methods have
different names to indicate which one. I could rename this method to `of()` but
since we are introducing new behavior I wanted to make it very clear that this
is the default, or existing, behavior. Also I'm not sure if it makes sense to
have a method called `of()` on the factory not on the coder itself.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #9758: [BEAM-8374] Enable returning missing PublishResult fields in SnsIO.Write
lukecwik commented on a change in pull request #9758: [BEAM-8374] Enable
returning missing PublishResult fields in SnsIO.Write
URL: https://github.com/apache/beam/pull/9758#discussion_r392519456
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/coders/AwsCoders.java
##
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.coders;
+
+import com.amazonaws.ResponseMetadata;
+import com.amazonaws.http.HttpResponse;
+import com.amazonaws.http.SdkHttpMetadata;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.CustomCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/** {@link Coder}s for common AWS SDK objects. */
+public final class AwsCoders {
+
+ private AwsCoders() {}
+
+ /**
+ * Returns a new coder for ResponseMetadata.
+ *
+ * @return the ResponseMetadata coder
+ */
+ public static Coder responseMetadata() {
+return ResponseMetadataCoder.of();
+ }
+
+ /**
+ * Returns a new coder for SdkHttpMetadata.
+ *
+ * @return the SdkHttpMetadata coder
+ */
+ public static Coder sdkHttpMetadata() {
+return new SdkHttpMetadataCoder(true);
Review comment:
of() prevents moving the class while a top level static method allows for
stuff to still be moved around.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] jfarr commented on a change in pull request #9758: [BEAM-8374] Enable returning missing PublishResult fields in SnsIO.Write
jfarr commented on a change in pull request #9758: [BEAM-8374] Enable returning
missing PublishResult fields in SnsIO.Write
URL: https://github.com/apache/beam/pull/9758#discussion_r392518426
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/coders/AwsCoders.java
##
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.coders;
+
+import com.amazonaws.ResponseMetadata;
+import com.amazonaws.http.HttpResponse;
+import com.amazonaws.http.SdkHttpMetadata;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.CustomCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/** {@link Coder}s for common AWS SDK objects. */
+public final class AwsCoders {
+
+ private AwsCoders() {}
+
+ /**
+ * Returns a new coder for ResponseMetadata.
+ *
+ * @return the ResponseMetadata coder
+ */
+ public static Coder responseMetadata() {
+return ResponseMetadataCoder.of();
+ }
+
+ /**
+ * Returns a new coder for SdkHttpMetadata.
+ *
+ * @return the SdkHttpMetadata coder
+ */
+ public static Coder sdkHttpMetadata() {
+return new SdkHttpMetadataCoder(true);
Review comment:
@cmachgodaddy That's because this class is designed as a factory for coders,
with the coder implementation being an internal detail, as @lukecwik suggested
in an earlier comment thread.
> Style note, you can create one class called `AwsCoders` which has three
static methods `minimalSdkHttpMetadata()` and `fullSdkHttpMetadata()` and
`responseMetadata()` that return the coders. This way you can make all the
coder implementations private inner static classes and they all become
implementation details without needing to expose them to users. Users can still
get an instance if they need one but it gives us flexibility from a maintenance
point of view how the coders are organized.
I could create a static factory method `of()` on SdkHttpMetadata but it
wouldn't serve any purpose, it would only be called by sdkHttpMetadata() which
is already a static factory method. SdkHttpMetadata is not meant to be
instantiated directly by the caller.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11109: [BEAM-8335] Final PR to merge the InteractiveBeam feature branch
pabloem commented on issue #11109: [BEAM-8335] Final PR to merge the InteractiveBeam feature branch URL: https://github.com/apache/beam/pull/11109#issuecomment-598958588 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] udim commented on a change in pull request #11060: [BEAM-9454] Create Deduplication transform based on user timer/state
udim commented on a change in pull request #11060: [BEAM-9454] Create Deduplication transform based on user timer/state URL: https://github.com/apache/beam/pull/11060#discussion_r392514885 ## File path: sdks/python/apache_beam/runners/sdf_utils.py ## @@ -244,3 +251,63 @@ def get_estimator_state(self): return None return _NoOpWatermarkEstimator() + + +class DeduplictaionWithinDuration(ptransform.PTransform): Review comment: No, no reason. Go ahead with using a separate file This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated (eb59dde -> 33ec0bb)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from eb59dde Merge pull request #11127 from BioGeek/patch-1 add 33ec0bb [BEAM-9477] RowCoder should be hashable and picklable (#11088) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/coders/row_coder.py | 18 +++--- sdks/python/apache_beam/coders/row_coder_test.py | 23 ++- 2 files changed, 37 insertions(+), 4 deletions(-)
[GitHub] [beam] TheNeuralBit merged pull request #11088: [BEAM-9477] RowCoder should be hashable and picklable
TheNeuralBit merged pull request #11088: [BEAM-9477] RowCoder should be hashable and picklable URL: https://github.com/apache/beam/pull/11088 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] udim commented on issue #9907: [BEAM-4091] Pass type hints in ptransform_fn
udim commented on issue #9907: [BEAM-4091] Pass type hints in ptransform_fn URL: https://github.com/apache/beam/pull/9907#issuecomment-598956817 still not ready This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11109: [BEAM-8335] Final PR to merge the InteractiveBeam feature branch
pabloem commented on issue #11109: [BEAM-8335] Final PR to merge the InteractiveBeam feature branch URL: https://github.com/apache/beam/pull/11109#issuecomment-598955515 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] udim closed pull request #11112: Color Jenkins logs support
udim closed pull request #2: Color Jenkins logs support URL: https://github.com/apache/beam/pull/2 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] udim commented on issue #11112: Color Jenkins logs support
udim commented on issue #2: Color Jenkins logs support URL: https://github.com/apache/beam/pull/2#issuecomment-598954651 Best result so far: https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1596/consoleFull but still not usable (even after removing timestamps). Aborting This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ananvay commented on issue #11129: [BEAM-9504] Sickbay streaming test for batch VR
ananvay commented on issue #11129: [BEAM-9504] Sickbay streaming test for batch VR URL: https://github.com/apache/beam/pull/11129#issuecomment-598937665 lgtm. Thanks Ankur! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka commented on issue #11129: [BEAM-9504] Sickbay streaming test for batch VR
angoenka commented on issue #11129: [BEAM-9504] Sickbay streaming test for batch VR URL: https://github.com/apache/beam/pull/11129#issuecomment-598936252 Run Python Dataflow V2 ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka commented on issue #11129: [BEAM-9504] Sickbay streaming test for batch VR
angoenka commented on issue #11129: [BEAM-9504] Sickbay streaming test for batch VR URL: https://github.com/apache/beam/pull/11129#issuecomment-598936044 Run Python Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka commented on issue #11129: [BEAM-9504] Sickbay streaming test for batch VR
angoenka commented on issue #11129: [BEAM-9504] Sickbay streaming test for batch VR URL: https://github.com/apache/beam/pull/11129#issuecomment-598930868 R: @Ardagan @ananvay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka opened a new pull request #11129: [BEAM-9504] Sickbay streaming test for batch VR
angoenka opened a new pull request #11129: [BEAM-9504] Sickbay streaming test for batch VR URL: https://github.com/apache/beam/pull/11129 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] suztomo edited a comment on issue #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky
suztomo edited a comment on issue #11090: [BEAM-9470]
:sdks:java:io:kinesis:test is flaky
URL: https://github.com/apache/beam/pull/11090#issuecomment-598926854
In the test case, the two iterators return the following data:
- firstIterator: `throw(e)`; `[a, b]`; `[]`; `[]`; `[]`; `[]` ...
- secondIterator: `[c]`; `[d]`; `[]`; `[]`; `[]` ...
The test case waits for 5 elements from shardReadersPool:
```
List fetchedRecords = new ArrayList<>();
while (fetchedRecords.size() < 4) {
CustomOptional nextRecord =
shardReadersPool.nextRecord();
if (nextRecord.isPresent()) {
fetchedRecords.add(nextRecord.get());
}
}
```
The test's expectation was:
```
verify(customRateLimitPolicy).onThrottle(same(e));
verify(customRateLimitPolicy).onSuccess(eq(ImmutableList.of(a, b)));
verify(customRateLimitPolicy).onSuccess(eq(singletonList(c)));
verify(customRateLimitPolicy).onSuccess(eq(singletonList(d)));
```
My assumption: there's no guarantee that 2 elements from firstIterator and
secondIterator are served evenly when 5 elements are consumed.
My suggestion: make the while loop until it confirms the expected 5 elements
are in `fetchedRecords` variable.
(Sorry if this is missing the point, I wrote this without checking @jfarr
's updated solution)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] suztomo commented on issue #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky
suztomo commented on issue #11090: [BEAM-9470] :sdks:java:io:kinesis:test is
flaky
URL: https://github.com/apache/beam/pull/11090#issuecomment-598926854
In the test case, the two iterators return the following data:
- firstIterator: `throw(e)`; `[a, b]`; `[]`; `[]`; `[]`; `[]` ...
- secondIterator: `[c]`; `[d]`; `[]`; `[]`; `[]` ...
The test case waits for 5 elements from shardReadersPool:
```
List fetchedRecords = new ArrayList<>();
while (fetchedRecords.size() < 4) {
CustomOptional nextRecord =
shardReadersPool.nextRecord();
if (nextRecord.isPresent()) {
fetchedRecords.add(nextRecord.get());
}
}
```
The test's expectation was:
```
verify(customRateLimitPolicy).onThrottle(same(e));
verify(customRateLimitPolicy).onSuccess(eq(ImmutableList.of(a, b)));
verify(customRateLimitPolicy).onSuccess(eq(singletonList(c)));
verify(customRateLimitPolicy).onSuccess(eq(singletonList(d)));
```
My assumption: there's no guarantee that 2 elements from firstIterator and
secondIterator are served evenly when 5 elements are consumed.
My suggestion: make the while loop until it confirms the expected 5 elements
are in `fetchedRecords` variable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rohdesamuel opened a new pull request #11128: Streamingcache refactor
rohdesamuel opened a new pull request #11128: Streamingcache refactor URL: https://github.com/apache/beam/pull/11128 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[beam] branch asf-site updated: Publishing website 2020/03/13 21:34:54 at commit eb59dde
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6d40afe Publishing website 2020/03/13 21:34:54 at commit eb59dde
6d40afe is described below
commit 6d40afe0881b30b858f9338cf469b6e7eb78aed7
Author: jenkins
AuthorDate: Fri Mar 13 21:34:55 2020 +
Publishing website 2020/03/13 21:34:54 at commit eb59dde
---
website/generated-content/documentation/programming-guide/index.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/website/generated-content/documentation/programming-guide/index.html
b/website/generated-content/documentation/programming-guide/index.html
index 48b04c6..74b9065 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -785,7 +785,7 @@ a command-line argument, and a default value.
void setInput(String input);
@Description("Output for the pipeline")
-@Default.String("gs://my-bucket/input")
+@Default.String("gs://my-bucket/output")
String getOutput();
void setOutput(String output);
}
[GitHub] [beam] tweise commented on issue #11108: [BEAM-9490] Guard referencing for environment expiration via a lock
tweise commented on issue #11108: [BEAM-9490] Guard referencing for environment expiration via a lock URL: https://github.com/apache/beam/pull/11108#issuecomment-598921032 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] lukecwik merged pull request #11127: Update default value in Java snippet
lukecwik merged pull request #11127: Update default value in Java snippet URL: https://github.com/apache/beam/pull/11127 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated: Update default value in Java snippet
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new eda1e24 Update default value in Java snippet
new eb59dde Merge pull request #11127 from BioGeek/patch-1
eda1e24 is described below
commit eda1e248dac8a75b3fb824fe52d5206626a39f6e
Author: Jeroen Van Goey
AuthorDate: Fri Mar 13 22:30:45 2020 +0100
Update default value in Java snippet
`@Default.String("gs://my-bucket/input")` was used as the default value for
both the input and the output custom option.
---
website/src/documentation/programming-guide.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/src/documentation/programming-guide.md
b/website/src/documentation/programming-guide.md
index 7db7171..1000865 100644
--- a/website/src/documentation/programming-guide.md
+++ b/website/src/documentation/programming-guide.md
@@ -227,7 +227,7 @@ public interface MyOptions extends PipelineOptions {
void setInput(String input);
@Description("Output for the pipeline")
-@Default.String("gs://my-bucket/input")
+@Default.String("gs://my-bucket/output")
String getOutput();
void setOutput(String output);
}
[GitHub] [beam] BioGeek opened a new pull request #11127: Update default value in Java snippet
BioGeek opened a new pull request #11127: Update default value in Java snippet
URL: https://github.com/apache/beam/pull/11127
`@Default.String("gs://my-bucket/input")` was used as the default value for
both the input and the output custom option.
**Please** add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] [**Choose
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[GitHub] [beam] lukecwik opened a new pull request #11126: [WIP][BEAM-9430] Migrate from ProcessContext#updateWatermark to WatermarkEstimators
lukecwik opened a new pull request #11126: [WIP][BEAM-9430] Migrate from ProcessContext#updateWatermark to WatermarkEstimators URL: https://github.com/apache/beam/pull/11126 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] amaliujia commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script.
amaliujia commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11124#issuecomment-598915263 LGTM after tests pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] suztomo commented on a change in pull request #11125: [BEAM-9444] Use Google Cloud Libraries BOM to set GCP library dependencies
suztomo commented on a change in pull request #11125: [BEAM-9444] Use Google
Cloud Libraries BOM to set GCP library dependencies
URL: https://github.com/apache/beam/pull/11125#discussion_r392459657
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -444,33 +443,36 @@ class BeamModulePlugin implements Plugin {
commons_lang3 :
"org.apache.commons:commons-lang3:3.9",
commons_math3 :
"org.apache.commons:commons-math3:3.6.1",
error_prone_annotations :
"com.google.errorprone:error_prone_annotations:2.0.15",
-gax :
"com.google.api:gax:$gax_version",
-gax_grpc:
"com.google.api:gax-grpc:$gax_version",
+gax : "com.google.api:gax",
+gax_grpc:
"com.google.api:gax-grpc",
google_api_client :
"com.google.api-client:google-api-client:$google_clients_version",
google_api_client_jackson2 :
"com.google.api-client:google-api-client-jackson2:$google_clients_version",
google_api_client_java6 :
"com.google.api-client:google-api-client-java6:$google_clients_version",
-google_api_common :
"com.google.api:api-common:1.8.1",
+google_api_common :
"com.google.api:api-common",
google_api_services_bigquery:
"com.google.apis:google-api-services-bigquery:v2-rev20191211-$google_clients_version",
google_api_services_clouddebugger :
"com.google.apis:google-api-services-clouddebugger:v2-rev20191003-$google_clients_version",
google_api_services_cloudresourcemanager:
"com.google.apis:google-api-services-cloudresourcemanager:v1-rev20191206-$google_clients_version",
google_api_services_dataflow:
"com.google.apis:google-api-services-dataflow:v1b3-rev20190927-$google_clients_version",
google_api_services_pubsub :
"com.google.apis:google-api-services-pubsub:v1-rev2019-$google_clients_version",
google_api_services_storage :
"com.google.apis:google-api-services-storage:v1-rev20191011-$google_clients_version",
-google_auth_library_credentials :
"com.google.auth:google-auth-library-credentials:$google_auth_version",
-google_auth_library_oauth2_http :
"com.google.auth:google-auth-library-oauth2-http:$google_auth_version",
-google_cloud_bigquery :
"com.google.cloud:google-cloud-bigquery:1.108.0",
-google_cloud_bigquery_storage :
"com.google.cloud:google-cloud-bigquerystorage:0.125.0-beta",
+google_auth_library_credentials :
"com.google.auth:google-auth-library-credentials",
+google_auth_library_oauth2_http :
"com.google.auth:google-auth-library-oauth2-http",
+google_cloud_bigquery :
"com.google.cloud:google-cloud-bigquery",
+google_cloud_bigquery_storage :
"com.google.cloud:google-cloud-bigquerystorage",
google_cloud_bigtable_client_core :
"com.google.cloud.bigtable:bigtable-client-core:1.13.0",
Review comment:
bigtable-client-core is not part of GCP Libraries BOM.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] boyuanzz commented on a change in pull request #11060: [BEAM-9454] Create Deduplication transform based on user timer/state
boyuanzz commented on a change in pull request #11060: [BEAM-9454] Create Deduplication transform based on user timer/state URL: https://github.com/apache/beam/pull/11060#discussion_r392461155 ## File path: sdks/python/apache_beam/runners/sdf_utils.py ## @@ -244,3 +251,63 @@ def get_estimator_state(self): return None return _NoOpWatermarkEstimator() + + +class DeduplictaionWithinDuration(ptransform.PTransform): Review comment: Thanks Udi! Similar to `watermark_estimators.py`, I'm thinking about providing a collection of transforms for deduplicating purpose. Any reason that we don't put a group of similar transforms into one module? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka commented on issue #11071: [cherry-pick][BEAM-9465] Fire repeatedly in reshuffle
angoenka commented on issue #11071: [cherry-pick][BEAM-9465] Fire repeatedly in reshuffle URL: https://github.com/apache/beam/pull/11071#issuecomment-598901989 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] suztomo commented on a change in pull request #11125: [BEAM-9444] Use Google Cloud Libraries BOM to set GCP library dependencies
suztomo commented on a change in pull request #11125: [BEAM-9444] Use Google
Cloud Libraries BOM to set GCP library dependencies
URL: https://github.com/apache/beam/pull/11125#discussion_r392459312
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -444,33 +443,36 @@ class BeamModulePlugin implements Plugin {
commons_lang3 :
"org.apache.commons:commons-lang3:3.9",
commons_math3 :
"org.apache.commons:commons-math3:3.6.1",
error_prone_annotations :
"com.google.errorprone:error_prone_annotations:2.0.15",
-gax :
"com.google.api:gax:$gax_version",
-gax_grpc:
"com.google.api:gax-grpc:$gax_version",
+gax : "com.google.api:gax",
+gax_grpc:
"com.google.api:gax-grpc",
google_api_client :
"com.google.api-client:google-api-client:$google_clients_version",
google_api_client_jackson2 :
"com.google.api-client:google-api-client-jackson2:$google_clients_version",
google_api_client_java6 :
"com.google.api-client:google-api-client-java6:$google_clients_version",
-google_api_common :
"com.google.api:api-common:1.8.1",
+google_api_common :
"com.google.api:api-common",
google_api_services_bigquery:
"com.google.apis:google-api-services-bigquery:v2-rev20191211-$google_clients_version",
google_api_services_clouddebugger :
"com.google.apis:google-api-services-clouddebugger:v2-rev20191003-$google_clients_version",
google_api_services_cloudresourcemanager:
"com.google.apis:google-api-services-cloudresourcemanager:v1-rev20191206-$google_clients_version",
google_api_services_dataflow:
"com.google.apis:google-api-services-dataflow:v1b3-rev20190927-$google_clients_version",
google_api_services_pubsub :
"com.google.apis:google-api-services-pubsub:v1-rev2019-$google_clients_version",
google_api_services_storage :
"com.google.apis:google-api-services-storage:v1-rev20191011-$google_clients_version",
-google_auth_library_credentials :
"com.google.auth:google-auth-library-credentials:$google_auth_version",
-google_auth_library_oauth2_http :
"com.google.auth:google-auth-library-oauth2-http:$google_auth_version",
-google_cloud_bigquery :
"com.google.cloud:google-cloud-bigquery:1.108.0",
-google_cloud_bigquery_storage :
"com.google.cloud:google-cloud-bigquerystorage:0.125.0-beta",
+google_auth_library_credentials :
"com.google.auth:google-auth-library-credentials",
+google_auth_library_oauth2_http :
"com.google.auth:google-auth-library-oauth2-http",
+google_cloud_bigquery :
"com.google.cloud:google-cloud-bigquery",
+google_cloud_bigquery_storage :
"com.google.cloud:google-cloud-bigquerystorage",
google_cloud_bigtable_client_core :
"com.google.cloud.bigtable:bigtable-client-core:1.13.0",
-google_cloud_core :
"com.google.cloud:google-cloud-core:$google_cloud_core_version",
-google_cloud_core_grpc :
"com.google.cloud:google-cloud-core-grpc:$google_cloud_core_version",
-google_cloud_datacatalog_v1beta1:
"com.google.cloud:google-cloud-datacatalog:$google_cloud_datacatalog_version",
+google_cloud_core :
"com.google.cloud:google-cloud-core",
+google_cloud_core_grpc :
"com.google.cloud:google-cloud-core-grpc",
+google_cloud_datacatalog_v1beta1:
"com.google.cloud:google-cloud-datacatalog",
google_cloud_dataflow_java_proto_library_all:
"com.google.cloud.dataflow:google-cloud-dataflow-java-proto-library-all:0.5.160304",
google_cloud_datastore_v1_proto_client :
"com.google.cloud.datastore:datastore-v1-proto-client:1.6.3",
-google_cloud_spanner:
"com.google.cloud:google-cloud-spanner:$google_cloud_spanner_version",
-google_http_client :
"com.google.http-client:google-http-client:$google_http_clients_version",
+google_cloud_platform_libraries_bom :
"com.google.cloud:libraries-bom:4.2.0",
+google_cloud_spanner:
"com.google.cloud:google-cloud-spanner",
+// google-http-client's version is explicitly declared for
sdks/java/maven-archetypes/examples
+// This version should
[GitHub] [beam] suztomo commented on a change in pull request #11125: [BEAM-9444] Use Google Cloud Libraries BOM to set GCP library dependencies
suztomo commented on a change in pull request #11125: [BEAM-9444] Use Google
Cloud Libraries BOM to set GCP library dependencies
URL: https://github.com/apache/beam/pull/11125#discussion_r392458321
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -444,33 +443,36 @@ class BeamModulePlugin implements Plugin {
commons_lang3 :
"org.apache.commons:commons-lang3:3.9",
commons_math3 :
"org.apache.commons:commons-math3:3.6.1",
error_prone_annotations :
"com.google.errorprone:error_prone_annotations:2.0.15",
-gax :
"com.google.api:gax:$gax_version",
-gax_grpc:
"com.google.api:gax-grpc:$gax_version",
+gax : "com.google.api:gax",
+gax_grpc:
"com.google.api:gax-grpc",
google_api_client :
"com.google.api-client:google-api-client:$google_clients_version",
google_api_client_jackson2 :
"com.google.api-client:google-api-client-jackson2:$google_clients_version",
google_api_client_java6 :
"com.google.api-client:google-api-client-java6:$google_clients_version",
-google_api_common :
"com.google.api:api-common:1.8.1",
+google_api_common :
"com.google.api:api-common",
google_api_services_bigquery:
"com.google.apis:google-api-services-bigquery:v2-rev20191211-$google_clients_version",
google_api_services_clouddebugger :
"com.google.apis:google-api-services-clouddebugger:v2-rev20191003-$google_clients_version",
google_api_services_cloudresourcemanager:
"com.google.apis:google-api-services-cloudresourcemanager:v1-rev20191206-$google_clients_version",
google_api_services_dataflow:
"com.google.apis:google-api-services-dataflow:v1b3-rev20190927-$google_clients_version",
google_api_services_pubsub :
"com.google.apis:google-api-services-pubsub:v1-rev2019-$google_clients_version",
google_api_services_storage :
"com.google.apis:google-api-services-storage:v1-rev20191011-$google_clients_version",
Review comment:
These "google-api-services-" artifacts are not part of GCP Libraries BOM.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] angoenka commented on issue #11111: [cherry-pick][BEAM-9485] Raise error when transform urn is not implemented
angoenka commented on issue #1: [cherry-pick][BEAM-9485] Raise error when transform urn is not implemented URL: https://github.com/apache/beam/pull/1#issuecomment-598902203 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated (64c2b41 -> a338431)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 64c2b41 [BEAM-7923] Streaming support and pipeline pruning when
instrumenting a pipeline with interactivity (#11100)
new c2945b6 Supporting infrastructure for dataframes on beam.
new 6de32d7 Basic deferred data frame implementation.
new 94ed5f1 yapf, py2
new a0b9c69 typings and docs for expressions.py
new 2dd615a Minor cleanup, lint.
new a338431 Merge pull request #10757 from robertwb/dataframe
The 26015 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../python/apache_beam/dataframe}/__init__.py | 1 -
sdks/python/apache_beam/dataframe/expressions.py | 213 +
.../apache_beam/dataframe/expressions_test.py | 58 ++
sdks/python/apache_beam/dataframe/frame_base.py| 132 +
.../apache_beam/dataframe/frame_base_test.py | 46 +
sdks/python/apache_beam/dataframe/frames.py| 210
sdks/python/apache_beam/dataframe/frames_test.py | 84
7 files changed, 743 insertions(+), 1 deletion(-)
copy {.test-infra/jenkins/dependency_check =>
sdks/python/apache_beam/dataframe}/__init__.py (99%)
create mode 100644 sdks/python/apache_beam/dataframe/expressions.py
create mode 100644 sdks/python/apache_beam/dataframe/expressions_test.py
create mode 100644 sdks/python/apache_beam/dataframe/frame_base.py
create mode 100644 sdks/python/apache_beam/dataframe/frame_base_test.py
create mode 100644 sdks/python/apache_beam/dataframe/frames.py
create mode 100644 sdks/python/apache_beam/dataframe/frames_test.py
[beam] branch master updated (08ac97f -> 64c2b41)
This is an automated email from the ASF dual-hosted git repository. altay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 08ac97f Update Dataflow py container version (#11120) add 64c2b41 [BEAM-7923] Streaming support and pipeline pruning when instrumenting a pipeline with interactivity (#11100) No new revisions were added by this update. Summary of changes: .../apache_beam/runners/interactive/README.md | 104 ++-- .../runners/interactive/background_caching_job.py | 6 + .../interactive/display/pcoll_visualization.py | 56 +-- .../display/pcoll_visualization_test.py| 15 +- .../runners/interactive/interactive_runner.py | 22 +- .../runners/interactive/pipeline_fragment.py | 3 + .../runners/interactive/pipeline_fragment_test.py | 20 + .../runners/interactive/pipeline_instrument.py | 400 +-- .../interactive/pipeline_instrument_test.py| 550 - .../interactive/testing/pipeline_assertion.py | 38 +- .../apache_beam/runners/interactive/utils.py | 28 ++ sdks/python/setup.py | 4 +- 12 files changed, 1095 insertions(+), 151 deletions(-)
[GitHub] [beam] suztomo commented on a change in pull request #11125: [BEAM-9444] Use Google Cloud Libraries BOM to set GCP library dependencies
suztomo commented on a change in pull request #11125: [BEAM-9444] Use Google
Cloud Libraries BOM to set GCP library dependencies
URL: https://github.com/apache/beam/pull/11125#discussion_r392459657
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -444,33 +443,36 @@ class BeamModulePlugin implements Plugin {
commons_lang3 :
"org.apache.commons:commons-lang3:3.9",
commons_math3 :
"org.apache.commons:commons-math3:3.6.1",
error_prone_annotations :
"com.google.errorprone:error_prone_annotations:2.0.15",
-gax :
"com.google.api:gax:$gax_version",
-gax_grpc:
"com.google.api:gax-grpc:$gax_version",
+gax : "com.google.api:gax",
+gax_grpc:
"com.google.api:gax-grpc",
google_api_client :
"com.google.api-client:google-api-client:$google_clients_version",
google_api_client_jackson2 :
"com.google.api-client:google-api-client-jackson2:$google_clients_version",
google_api_client_java6 :
"com.google.api-client:google-api-client-java6:$google_clients_version",
-google_api_common :
"com.google.api:api-common:1.8.1",
+google_api_common :
"com.google.api:api-common",
google_api_services_bigquery:
"com.google.apis:google-api-services-bigquery:v2-rev20191211-$google_clients_version",
google_api_services_clouddebugger :
"com.google.apis:google-api-services-clouddebugger:v2-rev20191003-$google_clients_version",
google_api_services_cloudresourcemanager:
"com.google.apis:google-api-services-cloudresourcemanager:v1-rev20191206-$google_clients_version",
google_api_services_dataflow:
"com.google.apis:google-api-services-dataflow:v1b3-rev20190927-$google_clients_version",
google_api_services_pubsub :
"com.google.apis:google-api-services-pubsub:v1-rev2019-$google_clients_version",
google_api_services_storage :
"com.google.apis:google-api-services-storage:v1-rev20191011-$google_clients_version",
-google_auth_library_credentials :
"com.google.auth:google-auth-library-credentials:$google_auth_version",
-google_auth_library_oauth2_http :
"com.google.auth:google-auth-library-oauth2-http:$google_auth_version",
-google_cloud_bigquery :
"com.google.cloud:google-cloud-bigquery:1.108.0",
-google_cloud_bigquery_storage :
"com.google.cloud:google-cloud-bigquerystorage:0.125.0-beta",
+google_auth_library_credentials :
"com.google.auth:google-auth-library-credentials",
+google_auth_library_oauth2_http :
"com.google.auth:google-auth-library-oauth2-http",
+google_cloud_bigquery :
"com.google.cloud:google-cloud-bigquery",
+google_cloud_bigquery_storage :
"com.google.cloud:google-cloud-bigquerystorage",
google_cloud_bigtable_client_core :
"com.google.cloud.bigtable:bigtable-client-core:1.13.0",
Review comment:
This is not part of GCP Libraries BOM.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] suztomo commented on a change in pull request #11125: [BEAM-9444] Use Google Cloud Libraries BOM to set GCP library dependencies
suztomo commented on a change in pull request #11125: [BEAM-9444] Use Google
Cloud Libraries BOM to set GCP library dependencies
URL: https://github.com/apache/beam/pull/11125#discussion_r392458718
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -444,33 +443,36 @@ class BeamModulePlugin implements Plugin {
commons_lang3 :
"org.apache.commons:commons-lang3:3.9",
commons_math3 :
"org.apache.commons:commons-math3:3.6.1",
error_prone_annotations :
"com.google.errorprone:error_prone_annotations:2.0.15",
-gax :
"com.google.api:gax:$gax_version",
-gax_grpc:
"com.google.api:gax-grpc:$gax_version",
+gax : "com.google.api:gax",
+gax_grpc:
"com.google.api:gax-grpc",
google_api_client :
"com.google.api-client:google-api-client:$google_clients_version",
google_api_client_jackson2 :
"com.google.api-client:google-api-client-jackson2:$google_clients_version",
google_api_client_java6 :
"com.google.api-client:google-api-client-java6:$google_clients_version",
-google_api_common :
"com.google.api:api-common:1.8.1",
+google_api_common :
"com.google.api:api-common",
google_api_services_bigquery:
"com.google.apis:google-api-services-bigquery:v2-rev20191211-$google_clients_version",
google_api_services_clouddebugger :
"com.google.apis:google-api-services-clouddebugger:v2-rev20191003-$google_clients_version",
google_api_services_cloudresourcemanager:
"com.google.apis:google-api-services-cloudresourcemanager:v1-rev20191206-$google_clients_version",
google_api_services_dataflow:
"com.google.apis:google-api-services-dataflow:v1b3-rev20190927-$google_clients_version",
google_api_services_pubsub :
"com.google.apis:google-api-services-pubsub:v1-rev2019-$google_clients_version",
google_api_services_storage :
"com.google.apis:google-api-services-storage:v1-rev20191011-$google_clients_version",
-google_auth_library_credentials :
"com.google.auth:google-auth-library-credentials:$google_auth_version",
-google_auth_library_oauth2_http :
"com.google.auth:google-auth-library-oauth2-http:$google_auth_version",
-google_cloud_bigquery :
"com.google.cloud:google-cloud-bigquery:1.108.0",
-google_cloud_bigquery_storage :
"com.google.cloud:google-cloud-bigquerystorage:0.125.0-beta",
+google_auth_library_credentials :
"com.google.auth:google-auth-library-credentials",
+google_auth_library_oauth2_http :
"com.google.auth:google-auth-library-oauth2-http",
+google_cloud_bigquery :
"com.google.cloud:google-cloud-bigquery",
+google_cloud_bigquery_storage :
"com.google.cloud:google-cloud-bigquerystorage",
google_cloud_bigtable_client_core :
"com.google.cloud.bigtable:bigtable-client-core:1.13.0",
-google_cloud_core :
"com.google.cloud:google-cloud-core:$google_cloud_core_version",
-google_cloud_core_grpc :
"com.google.cloud:google-cloud-core-grpc:$google_cloud_core_version",
-google_cloud_datacatalog_v1beta1:
"com.google.cloud:google-cloud-datacatalog:$google_cloud_datacatalog_version",
+google_cloud_core :
"com.google.cloud:google-cloud-core",
+google_cloud_core_grpc :
"com.google.cloud:google-cloud-core-grpc",
+google_cloud_datacatalog_v1beta1:
"com.google.cloud:google-cloud-datacatalog",
google_cloud_dataflow_java_proto_library_all:
"com.google.cloud.dataflow:google-cloud-dataflow-java-proto-library-all:0.5.160304",
google_cloud_datastore_v1_proto_client :
"com.google.cloud.datastore:datastore-v1-proto-client:1.6.3",
Review comment:
These are not part of GCP Libraries BOM.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb merged pull request #10757: [BEAM-9496] Starting implementation of dataframes for Beam
robertwb merged pull request #10757: [BEAM-9496] Starting implementation of dataframes for Beam URL: https://github.com/apache/beam/pull/10757 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] aaltay merged pull request #11100: [BEAM-7923] Streaming support and pipeline pruning when instrumenting a pipeline with interactivity
aaltay merged pull request #11100: [BEAM-7923] Streaming support and pipeline pruning when instrumenting a pipeline with interactivity URL: https://github.com/apache/beam/pull/11100 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] suztomo commented on a change in pull request #11125: [BEAM-9444] Use Google Cloud Libraries BOM to set GCP library dependencies
suztomo commented on a change in pull request #11125: [BEAM-9444] Use Google
Cloud Libraries BOM to set GCP library dependencies
URL: https://github.com/apache/beam/pull/11125#discussion_r392456523
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1274,7 +1276,7 @@ class BeamModulePlugin implements Plugin {
// has different dependencies than our project.
if (config.getName() != "errorprone" && !inDependencyUpdates) {
config.resolutionStrategy {
-force project.library.java.values()
+force project.library.java.values().findAll { it.split(':').size()
> 2 }
Review comment:
`force project.library.java.values()` does not take GCP libraries BOM into
account. So excluding the versionless coordinates.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] suztomo opened a new pull request #11125: [BEAM-9444] Use Google Cloud Libraries BOM to set GCP library dependencies
suztomo opened a new pull request #11125: [BEAM-9444] Use Google Cloud Libraries BOM to set GCP library dependencies URL: https://github.com/apache/beam/pull/11125 As per [Subject: Proposal: Beam to use GCP Libraries BOM](https://lists.apache.org/thread.html/r7c84383ee3e3ae8e0819b52b188a0676602e3915ce80acef4f1a7bdd%40%3Cdev.beam.apache.org%3E), specifying library versions through GCP Libraries BOM. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[GitHub] [beam] robertwb commented on issue #11048: [BEAM-9433] Create expansion service artifact for common Java IOs.
robertwb commented on issue #11048: [BEAM-9433] Create expansion service artifact for common Java IOs. URL: https://github.com/apache/beam/pull/11048#issuecomment-598891273 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script.
ibzib commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11124#issuecomment-598887440 Run Python 3.6 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script.
ibzib commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11124#issuecomment-598887460 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script.
ibzib commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11124#issuecomment-598887381 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] stale[bot] commented on issue #8824: [BEAM-6766] Implement SMB file operations
stale[bot] commented on issue #8824: [BEAM-6766] Implement SMB file operations URL: https://github.com/apache/beam/pull/8824#issuecomment-598887535 This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] stale[bot] closed pull request #9251: [BEAM-6766] Implement SMB source
stale[bot] closed pull request #9251: [BEAM-6766] Implement SMB source URL: https://github.com/apache/beam/pull/9251 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script.
ibzib commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11124#issuecomment-598887413 Run Python 3.5 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] stale[bot] commented on issue #9251: [BEAM-6766] Implement SMB source
stale[bot] commented on issue #9251: [BEAM-6766] Implement SMB source URL: https://github.com/apache/beam/pull/9251#issuecomment-598887538 This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] stale[bot] closed pull request #8824: [BEAM-6766] Implement SMB file operations
stale[bot] closed pull request #8824: [BEAM-6766] Implement SMB file operations URL: https://github.com/apache/beam/pull/8824 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib commented on issue #11123: [BEAM-9503] Insert missing comma in process worker script.
ibzib commented on issue #11123: [BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11123#issuecomment-598887272 Run Python 3.6 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib removed a comment on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script.
ibzib removed a comment on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11124#issuecomment-598886460 Run Python2 Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib removed a comment on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script.
ibzib removed a comment on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11124#issuecomment-598886496 Run Python3 Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib commented on issue #11123: [BEAM-9503] Insert missing comma in process worker script.
ibzib commented on issue #11123: [BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11123#issuecomment-598887299 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib commented on issue #11123: [BEAM-9503] Insert missing comma in process worker script.
ibzib commented on issue #11123: [BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11123#issuecomment-598887235 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib commented on issue #11123: [BEAM-9503] Insert missing comma in process worker script.
ibzib commented on issue #11123: [BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11123#issuecomment-598887261 Run Python 3.5 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script.
ibzib commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11124#issuecomment-598886460 Run Python2 Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script.
ibzib commented on issue #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11124#issuecomment-598886496 Run Python3 Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] ibzib opened a new pull request #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script.
ibzib opened a new pull request #11124: [cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11124 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[GitHub] [beam] robertwb commented on issue #11067: [BEAM-9136]Add licenses for dependencies
robertwb commented on issue #11067: [BEAM-9136]Add licenses for dependencies URL: https://github.com/apache/beam/pull/11067#issuecomment-598882830 In general, it's preferable to avoid checking in script-created files, preferring instead to create them on demand at build time. (There are exceptions, e.g. bootstrapping and/or difficult to obtain build tools, but I don't think that applies here.) As another principle, it's preferable, where possible, to enforce correctness of the repository state before things get committed/merged (e.g. at pre-commit time), rather than have a separate, asynchronous, external process to try to fix things up after the fact. Perhaps it's worth expanding the design doc on how we could do this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] tvalentyn commented on issue #11120: Update Dataflow py container version
tvalentyn commented on issue #11120: Update Dataflow py container version URL: https://github.com/apache/beam/pull/11120#issuecomment-598882401 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] tvalentyn merged pull request #11120: Update Dataflow py container version
tvalentyn merged pull request #11120: Update Dataflow py container version URL: https://github.com/apache/beam/pull/11120 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated (aa35a30 -> 08ac97f)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from aa35a30 Merge pull request #4 from lukecwik/beam9481 add 08ac97f Update Dataflow py container version (#11120) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/dataflow/internal/names.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[GitHub] [beam] pabloem commented on issue #11104: Update BigQuery source in bigquery_tornadoes example
pabloem commented on issue #11104: Update BigQuery source in bigquery_tornadoes example URL: https://github.com/apache/beam/pull/11104#issuecomment-598880690 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] kennknowles commented on issue #10988: [BEAM-9382] Clean up of TestStreamTranscriptTests.
kennknowles commented on issue #10988: [BEAM-9382] Clean up of TestStreamTranscriptTests. URL: https://github.com/apache/beam/pull/10988#issuecomment-598878883 Yea I don't think anyone is happy about it. But it has always been that way and also is the only way we meet the spec of panes being (EARLY* ON_TIME? LATE*) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] y1chi commented on issue #11104: Update BigQuery source in bigquery_tornadoes example
y1chi commented on issue #11104: Update BigQuery source in bigquery_tornadoes example URL: https://github.com/apache/beam/pull/11104#issuecomment-598877763 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] jaketf commented on a change in pull request #11107: [BEAM-9468] [WIP] add HL7v2IO and FhirIO
jaketf commented on a change in pull request #11107: [BEAM-9468] [WIP] add
HL7v2IO and FhirIO
URL: https://github.com/apache/beam/pull/11107#discussion_r392424941
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FetchHL7v2Message.java
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.healthcare;
+
+import com.google.api.services.healthcare.v1alpha2.model.Message;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * DoFn to fetch a message from an Google Cloud Healthcare HL7v2 store based
on msgID
+ *
+ * This DoFn consumes a {@link PCollection} of notifications from
the HL7v2 store, and
+ * fetches the actual {@link Message} object based on the id in the
notification and will output a
+ * {@link PCollectionTuple} which contains the output and dead-letter {@link
PCollection}.
+ *
+ * The {@link PCollectionTuple} output will contain the following {@link
PCollection}:
+ *
+ *
+ * {@link FetchHL7v2Message#OUT} - Contains all {@link FailsafeElement}
records successfully
+ * read from the HL7v2 store.
+ * {@link FetchHL7v2Message#DEAD_LETTER} - Contains all {@link
FailsafeElement} records which
+ * failed to be fetched from the HL7v2 store, with error message and
stacktrace.
+ *
+ *
+ * Example:
+ *
+ * {@code
+ * PipelineOptions options = ...;
+ * Pipeline pipeline = Pipeline.create(options)
+ *
+ * PCollection msgIDs = pipeline.apply(
+ *"ReadHL7v2Notifications",
+ *PubsubIO.readStrings().fromSubscription(options.getInputSubscription()));
+ *
+ * PCollectionTuple fetchResults = msgIDs.apply(
+ *"FetchHL7v2Messages",
+ *new FetchHL7v2Message;
+ *
+ * // Write errors to your favorite dead letter queue (e.g. Pub/Sub, GCS,
BigQuery)
+ * fetchResults.get(PubsubNotificationToHL7v2Message.DEAD_LETTER)
+ *.apply("WriteToDeadLetterQueue", ...);
+ *
+ * PCollection fetchedMessages =
fetchResults.get(PubsubNotificationToHL7v2Message.OUT)
+ *.apply("ExtractFetchedMessage",
+ *MapElements
+ *.into(TypeDescriptor.of(Message.class))
+ *.via(FailsafeElement::getPayload));
+ *
+ * // Go about your happy path transformations.
+ * fetchedMessages.apply("ProcessFetchedMessages", ...)
+ *
+ * }***
+ *
+ */
+public class FetchHL7v2Message extends PTransform,
PCollectionTuple> {
Review comment:
Totally Agreed. This is why I elected to use adaptive throttling in the
HL7MessageGetFn.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11081: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch
amaliujia commented on issue #11081: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch URL: https://github.com/apache/beam/pull/11081#issuecomment-598876085 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] lastomato commented on a change in pull request #11107: [BEAM-9468] [WIP] add HL7v2IO and FhirIO
lastomato commented on a change in pull request #11107: [BEAM-9468] [WIP] add
HL7v2IO and FhirIO
URL: https://github.com/apache/beam/pull/11107#discussion_r392414499
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FetchHL7v2Message.java
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.healthcare;
+
+import com.google.api.services.healthcare.v1alpha2.model.Message;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * DoFn to fetch a message from an Google Cloud Healthcare HL7v2 store based
on msgID
+ *
+ * This DoFn consumes a {@link PCollection} of notifications from
the HL7v2 store, and
+ * fetches the actual {@link Message} object based on the id in the
notification and will output a
+ * {@link PCollectionTuple} which contains the output and dead-letter {@link
PCollection}.
+ *
+ * The {@link PCollectionTuple} output will contain the following {@link
PCollection}:
+ *
+ *
+ * {@link FetchHL7v2Message#OUT} - Contains all {@link FailsafeElement}
records successfully
+ * read from the HL7v2 store.
+ * {@link FetchHL7v2Message#DEAD_LETTER} - Contains all {@link
FailsafeElement} records which
+ * failed to be fetched from the HL7v2 store, with error message and
stacktrace.
+ *
+ *
+ * Example:
+ *
+ * {@code
+ * PipelineOptions options = ...;
+ * Pipeline pipeline = Pipeline.create(options)
+ *
+ * PCollection msgIDs = pipeline.apply(
+ *"ReadHL7v2Notifications",
+ *PubsubIO.readStrings().fromSubscription(options.getInputSubscription()));
+ *
+ * PCollectionTuple fetchResults = msgIDs.apply(
+ *"FetchHL7v2Messages",
+ *new FetchHL7v2Message;
+ *
+ * // Write errors to your favorite dead letter queue (e.g. Pub/Sub, GCS,
BigQuery)
+ * fetchResults.get(PubsubNotificationToHL7v2Message.DEAD_LETTER)
+ *.apply("WriteToDeadLetterQueue", ...);
+ *
+ * PCollection fetchedMessages =
fetchResults.get(PubsubNotificationToHL7v2Message.OUT)
+ *.apply("ExtractFetchedMessage",
+ *MapElements
+ *.into(TypeDescriptor.of(Message.class))
+ *.via(FailsafeElement::getPayload));
+ *
+ * // Go about your happy path transformations.
+ * fetchedMessages.apply("ProcessFetchedMessages", ...)
+ *
+ * }***
+ *
+ */
+public class FetchHL7v2Message extends PTransform,
PCollectionTuple> {
Review comment:
This is probably the best we can do right now, but at some point we should
use a batch approach to avoid DDoS the service with large datasets. I would
check with the service owner to see if this is OK.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lastomato commented on a change in pull request #11107: [BEAM-9468] [WIP] add HL7v2IO and FhirIO
lastomato commented on a change in pull request #11107: [BEAM-9468] [WIP] add
HL7v2IO and FhirIO
URL: https://github.com/apache/beam/pull/11107#discussion_r392423142
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java
##
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.healthcare;
+
+import com.google.api.services.healthcare.v1alpha2.model.HttpBody;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+
+/**
+ * {@link FhirIO} provides an API for writing resources to https://cloud.google.com/healthcare/docs/concepts/fhir;>Google Cloud
Healthcare Fhir API.
+ *
+ */
+public class FhirIO {
+
+ /** The type Write. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+
+/** The enum Write method. */
+public enum WriteMethod {
+ /**
+ * Execute Bundle Method executes a batch of requests as a single
transaction @see https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/executeBundle>.
+ */
+ EXECUTE_BUNDLE,
+ /**
+ * Create Method creates a single FHIR resource @see https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/create>.
+ */
+ CREATE
+}
+
+/**
+ * Gets Fhir store.
+ *
+ * @return the Fhir store
+ */
+abstract String getFhirStore();
+
+/**
+ * Gets write method.
+ *
+ * @return the write method
+ */
+abstract WriteMethod getWriteMethod();
+
+/** The type Builder. */
+@AutoValue.Builder
+abstract static class Builder {
+
+ /**
+ * Sets Fhir store.
+ *
+ * @param fhirStore the Fhir store
+ * @return the Fhir store
+ */
+ abstract Builder setFhirStore(String fhirStore);
+
+ /**
+ * Sets write method.
+ *
+ * @param writeMethod the write method
+ * @return the write method
+ */
+ abstract Builder setWriteMethod(WriteMethod writeMethod);
+
+ /**
+ * Build write.
+ *
+ * @return the write
+ */
+ abstract Write build();
+}
+
+private static Write.Builder write(String fhirStore) {
+ return new AutoValue_FhirIO_Write.Builder().setFhirStore(fhirStore);
+}
+
+/**
+ * Create Method creates a single FHIR resource. @see https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/create>
+ *
+ * @param fhirStore the hl 7 v 2 store
+ * @return the write
+ */
+public static Write create(String fhirStore) {
+ return new AutoValue_FhirIO_Write.Builder()
+ .setFhirStore(fhirStore)
+ .setWriteMethod(Write.WriteMethod.CREATE)
+ .build();
+}
+
+/**
+ * Execute Bundle Method executes a batch of requests as a single
transaction @see https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/executeBundle>.
+ *
+ * @param fhirStore the hl 7 v 2 store
+ * @return the write
+ */
+public static Write executeBundles(String fhirStore) {
+ return new AutoValue_FhirIO_Write.Builder()
+ .setFhirStore(fhirStore)
+ .setWriteMethod(WriteMethod.EXECUTE_BUNDLE)
+ .build();
+}
+
+@Override
+public PDone expand(PCollection messages) {
+ messages.apply(ParDo.of(new WriteFhirFn(this.getFhirStore(),
this.getWriteMethod(;
+ return PDone.in(messages.getPipeline());
+}
+ }
+
+ /** The type Write Fhir fn. */
+ static class WriteFhirFn extends DoFn {
+// TODO when the healthcare API releases a bulk import method this should
use that to improve
+// throughput.
+
+private transient HealthcareApiClient client;
Review comment:
You might want to reconstruct the
[GitHub] [beam] lastomato commented on a change in pull request #11107: [BEAM-9468] [WIP] add HL7v2IO and FhirIO
lastomato commented on a change in pull request #11107: [BEAM-9468] [WIP] add
HL7v2IO and FhirIO
URL: https://github.com/apache/beam/pull/11107#discussion_r392422555
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java
##
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.healthcare;
+
+import com.google.api.services.healthcare.v1alpha2.model.HttpBody;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+
+/**
+ * {@link FhirIO} provides an API for writing resources to https://cloud.google.com/healthcare/docs/concepts/fhir;>Google Cloud
Healthcare Fhir API.
+ *
+ */
+public class FhirIO {
Review comment:
We can read from FHIR stores with the export API:
https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores/export.
Similarly, we should also support writing to FHIR store with the import API.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11123: [BEAM-9503] Insert missing comma in process worker script.
amaliujia commented on issue #11123: [BEAM-9503] Insert missing comma in process worker script. URL: https://github.com/apache/beam/pull/11123#issuecomment-598871213 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
