[GitHub] [beam] pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-598029240 Run Python 3.5 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-598029216 Run Python 3.6 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] Hannah-Jiang commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
Hannah-Jiang commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-598029301 > fyi @Hannah-Jiang I'm trying to fix python postcommits in this PR : ) Thanks for letting me know. Which PR introduced the failure? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-598029192 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
chamikaramj commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-598026769 LGTM. Thanks. Should we document differences in semantics in BQ sink when using Avro and JSON formats somewhere ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11103: Reifying outputs from BQ file writing
pabloem commented on issue #11103: Reifying outputs from BQ file writing URL: https://github.com/apache/beam/pull/11103#issuecomment-598026472 yes. I'm working on https://github.com/apache/beam/pull/11106 to hopefully fix Python postcommits. Thanks @amaliujia This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-598026604 fyi @Hannah-Jiang I'm trying to fix python postcommits in this PR : ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new c69d409 [BEAM-8335] Modify the StreamingCache to subclass the CacheManager new bb9826c Merge pull request #11005 from [BEAM-8335] Modify the StreamingCache to subclass the CacheManager c69d409 is described below commit c69d409429cf4dd3234667150458fb5b7f4b Author: Sam Rohde AuthorDate: Fri Feb 28 13:32:30 2020 -0800 [BEAM-8335] Modify the StreamingCache to subclass the CacheManager Change-Id: Ib61aa3fac53d9109178744e11eeebe5c5da0929c --- .../runners/interactive/cache_manager.py | 71 - .../runners/interactive/cache_manager_test.py | 30 +- .../runners/interactive/caching/streaming_cache.py | 303 +- .../interactive/caching/streaming_cache_test.py| 338 +++-- .../runners/interactive/display/display_manager.py | 2 +- .../runners/interactive/interactive_runner_test.py | 6 +- .../runners/interactive/pipeline_fragment_test.py | 2 +- .../interactive/pipeline_instrument_test.py| 28 +- .../interactive/testing/test_cache_manager.py | 119 9 files changed, 733 insertions(+), 166 deletions(-) diff --git a/sdks/python/apache_beam/runners/interactive/cache_manager.py b/sdks/python/apache_beam/runners/interactive/cache_manager.py index 7274015..09c44d6 100644 --- a/sdks/python/apache_beam/runners/interactive/cache_manager.py +++ b/sdks/python/apache_beam/runners/interactive/cache_manager.py @@ -51,26 +51,34 @@ class CacheManager(object): derivation. """ def exists(self, *labels): +# type (*str) -> bool + """Returns if the PCollection cache exists.""" raise NotImplementedError def is_latest_version(self, version, *labels): +# type (str, *str) -> bool + """Returns if the given version number is the latest.""" return version == self._latest_version(*labels) def _latest_version(self, *labels): +# type (*str) -> str + """Returns the latest version number of the PCollection cache.""" raise NotImplementedError def read(self, *labels): +# type (*str) -> Tuple[str, Generator[Any]] + """Return the PCollection as a list as well as the version number. Args: *labels: List of labels for PCollection instance. Returns: - Tuple[List[Any], int]: A tuple containing a list of items in the -PCollection and the version number. + A tuple containing an iterator for the items in the PCollection and the +version number. It is possible that the version numbers from read() and_latest_version() are different. This usually means that the cache's been evicted (thus @@ -79,15 +87,32 @@ class CacheManager(object): """ raise NotImplementedError + def write(self, value, *labels): +# type (Any, *str) -> None + +"""Writes the value to the given cache. + +Args: + value: An encodable (with corresponding PCoder) value + *labels: List of labels for PCollection instance +""" +raise NotImplementedError + def source(self, *labels): -"""Returns a beam.io.Source that reads the PCollection cache.""" +# type (*str) -> ptransform.PTransform + +"""Returns a PTransform that reads the PCollection cache.""" raise NotImplementedError - def sink(self, *labels): -"""Returns a beam.io.Sink that writes the PCollection cache.""" + def sink(self, labels): +# type (*str) -> ptransform.PTransform + +"""Returns a PTransform that writes the PCollection cache.""" raise NotImplementedError def save_pcoder(self, pcoder, *labels): +# type (coders.Coder, *str) -> None + """Saves pcoder for given PCollection. Correct reading of PCollection from Cache requires PCoder to be known. @@ -103,10 +128,14 @@ class CacheManager(object): raise NotImplementedError def load_pcoder(self, *labels): +# type (*str) -> coders.Coder + """Returns previously saved PCoder for reading and writing PCollection.""" raise NotImplementedError def cleanup(self): +# type () -> None + """Cleans up all the PCollection caches.""" raise NotImplementedError @@ -167,22 +196,34 @@ class FileBasedCacheManager(CacheManager): self._saved_pcoders[self._path(*labels)]) def read(self, *labels): +# Return an iterator to an empty list if it doesn't exist. if not self.exists(*labels): - return [], -1 + return iter([]), -1 -source = self.source(*labels) +# Otherwise, return a generator to the cached PCollection. +source = self.source(*labels)._source range_tracker = source.get_range_tracker(None, None) -result = list(source.read(range_tracker)) +reader = source.read(range_tracker) version = self._latest_version(*labels) -r
[GitHub] [beam] pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005#issuecomment-598025782 Merging as Robert's last comment was addressed by re-reading from the previous postion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem merged pull request #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem merged pull request #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] sunjincheng121 commented on issue #10884: [BEAM-9298] Drop support for Flink 1.7
sunjincheng121 commented on issue #10884: [BEAM-9298] Drop support for Flink 1.7 URL: https://github.com/apache/beam/pull/10884#issuecomment-598024234 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] sunjincheng121 commented on issue #10884: [BEAM-9298] Drop support for Flink 1.7
sunjincheng121 commented on issue #10884: [BEAM-9298] Drop support for Flink 1.7 URL: https://github.com/apache/beam/pull/10884#issuecomment-598024199 Run CommunityMetrics PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-598022746 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-598017978 sample py37 postcom: https://builds.apache.org/job/beam_PostCommit_Python37_PR/104/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-598017085 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] chadrik commented on issue #11038: [BEAM-7746] More typing fixes
chadrik commented on issue #11038: [BEAM-7746] More typing fixes URL: https://github.com/apache/beam/pull/11038#issuecomment-598014481 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] amaliujia commented on issue #11103: Reifying outputs from BQ file writing
amaliujia commented on issue #11103: Reifying outputs from BQ file writing URL: https://github.com/apache/beam/pull/11103#issuecomment-598000970 My current concern is `python2 postcommit` is failing. Could we hold this PR a bit until we figure out the resolution/answer to that? Just don't want to make the debugging worse. Cherry-picking this PR would be easy if 1) a Jira to describe what it is doing 2) this PR does not affect postcommit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] jfarr commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky
jfarr commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky URL: https://github.com/apache/beam/pull/11090#discussion_r391390299 ## File path: sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java ## @@ -324,10 +325,15 @@ public void shouldCallRateLimitPolicy() } } +ArgumentCaptor> recordsCaptor = ArgumentCaptor.forClass(List.class); +verify(customRateLimitPolicy, atLeastOnce()).onSuccess(recordsCaptor.capture()); +List> capturedRecords = recordsCaptor.getAllValues(); +assertThat(capturedRecords).contains( +ImmutableList.of(a, b), +singletonList(c), +singletonList(d), +Collections.emptyList() +); verify(customRateLimitPolicy).onThrottle(same(e)); -verify(customRateLimitPolicy).onSuccess(eq(ImmutableList.of(a, b))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(c))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(d))); Review comment: @aromanenko-dev If you could share with us how you got @RepeatedTest working I'd be happy to retest. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] jfarr commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky
jfarr commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky URL: https://github.com/apache/beam/pull/11090#discussion_r391389538 ## File path: sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java ## @@ -324,10 +325,15 @@ public void shouldCallRateLimitPolicy() } } +ArgumentCaptor> recordsCaptor = ArgumentCaptor.forClass(List.class); +verify(customRateLimitPolicy, atLeastOnce()).onSuccess(recordsCaptor.capture()); +List> capturedRecords = recordsCaptor.getAllValues(); +assertThat(capturedRecords).contains( +ImmutableList.of(a, b), +singletonList(c), +singletonList(d), +Collections.emptyList() +); verify(customRateLimitPolicy).onThrottle(same(e)); -verify(customRateLimitPolicy).onSuccess(eq(ImmutableList.of(a, b))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(c))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(d))); Review comment: @suztomo You have a good point. The intention of this test is to validate that every time readNextBatch() returns a list of records onSuccess() gets called and every time it throws `KinesisClientThrottledException` onThrottle() gets called. I can see a couple of code paths where readNextBatch() could return records without onSuccess() getting called, namely if any exceptions are thrown along in here: https://github.com/apache/beam/blob/d62521f69ead4b58924043f041978d49e9beeb62/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java#L146-L149 I haven't seen evidence that this is what's happening, but let's try my latest changes in ShardReadersPool and see if that fixes it. With this change it should not be possible for readNextBatch() to return without onSuccess() being called. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-597984563 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-597984578 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-597980134 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] sunjincheng121 commented on issue #10884: [BEAM-9298] Drop support for Flink 1.7
sunjincheng121 commented on issue #10884: [BEAM-9298] Drop support for Flink 1.7 URL: https://github.com/apache/beam/pull/10884#issuecomment-597974581 > @sunjincheng121 Could you update the PR so we can get this in? Sure, Thanks for the reminder @mxm , I would like to update this PR soon. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] suztomo commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky
suztomo commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky URL: https://github.com/apache/beam/pull/11090#discussion_r391365716 ## File path: sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java ## @@ -324,10 +325,15 @@ public void shouldCallRateLimitPolicy() } } +ArgumentCaptor> recordsCaptor = ArgumentCaptor.forClass(List.class); +verify(customRateLimitPolicy, atLeastOnce()).onSuccess(recordsCaptor.capture()); +List> capturedRecords = recordsCaptor.getAllValues(); +assertThat(capturedRecords).contains( +ImmutableList.of(a, b), +singletonList(c), +singletonList(d), +Collections.emptyList() +); verify(customRateLimitPolicy).onThrottle(same(e)); -verify(customRateLimitPolicy).onSuccess(eq(ImmutableList.of(a, b))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(c))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(d))); Review comment: bug: sometimes customRateLimitPolicy does not receive items, "[d]" in the test case. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] Akshay-Iyangar commented on issue #10078: [BEAM-8542] Change write to async in AWS SNS IO & remove retry logic
Akshay-Iyangar commented on issue #10078: [BEAM-8542] Change write to async in AWS SNS IO & remove retry logic URL: https://github.com/apache/beam/pull/10078#issuecomment-597966021 @aromanenko-dev - done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] chadrik commented on issue #11038: [BEAM-7746] More typing fixes
chadrik commented on issue #11038: [BEAM-7746] More typing fixes URL: https://github.com/apache/beam/pull/11038#issuecomment-597961202 Arg. Github showed this as failed even after I refreshed, and then as soon as I did `Run Python PreCommit`, it updated to show that it had succeeded briefly and then started the job again, which now failed. Every time I run it, it seems to fail with a different error. I don't think they are related to the changes here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] chadrik commented on issue #11038: [BEAM-7746] More typing fixes
chadrik commented on issue #11038: [BEAM-7746] More typing fixes URL: https://github.com/apache/beam/pull/11038#issuecomment-597961227 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] jaketf commented on issue #11107: [BEAM-9468] [WIP] add HL7v2IO and FhirIO
jaketf commented on issue #11107: [BEAM-9468] [WIP] add HL7v2IO and FhirIO URL: https://github.com/apache/beam/pull/11107#issuecomment-597960845 @lastomato @yeweidaniel @toby-hu Would really appreciate your input on this initial sketch of what IO connectors for healthcare API could look like. I took inspiration from how some things were done in the CDF plugins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] jaketf opened a new pull request #11107: [BEAM-9468] [WIP] add HL7v2IO and FhirIO
jaketf opened a new pull request #11107: [BEAM-9468] [WIP] add HL7v2IO and FhirIO URL: https://github.com/apache/beam/pull/11107 Add IO connectors for the Google Cloud Healthcare API. Work for these connectors is being prioritized based to support various customer use cases for. HL7v2 -> FHIR ETL pipelines. - [HL7v2](https://cloud.google.com/healthcare/docs/how-tos/hl7v2-messages): - Unbounded Read: Uses PubsubIO to read [notification subscription](https://cloud.google.com/healthcare/docs/how-tos/pubsub#hl7v2_messages) - Bounded Read: DoFn to get message IDs using the [Messages.List](https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.hl7V2Stores.messages/list) REST API method - [FhirIO](https://cloud.google.com/healthcare/docs/how-tos/fhir-bundles) - Write use the [executeBundle](https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/executeBundle) REST API method to execute transactions on the FHIR Store Future Work will include: - [ ] Adding Read methods to FhirIO for: - [ ] read(String name) Read all of a given resource type by name using [read](https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/read) REST API method - [ ] search(String query): use the [search](https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/search) REST API method to read all results from a search - [ ] Adding an additional Write method using the [create](https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/create) REST API mehod Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_P
[GitHub] [beam] tvalentyn commented on issue #11092: [BEAM-9085] Fix performance regression in SyntheticSource
tvalentyn commented on issue #11092: [BEAM-9085] Fix performance regression in SyntheticSource URL: https://github.com/apache/beam/pull/11092#issuecomment-597953191 It is testing a pipeline with a side input: '--sideinput_num_records=1', '--sideinput_key_size=10', '--maininput_key_size=10', '--maininput_num_records=10', I reran the test, and the regression was on the order of 3x; running the tests again 5 times to see the difference. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] jfarr commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky
jfarr commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky URL: https://github.com/apache/beam/pull/11090#discussion_r391341830 ## File path: sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java ## @@ -324,10 +325,15 @@ public void shouldCallRateLimitPolicy() } } +ArgumentCaptor> recordsCaptor = ArgumentCaptor.forClass(List.class); +verify(customRateLimitPolicy, atLeastOnce()).onSuccess(recordsCaptor.capture()); +List> capturedRecords = recordsCaptor.getAllValues(); +assertThat(capturedRecords).contains( +ImmutableList.of(a, b), +singletonList(c), +singletonList(d), +Collections.emptyList() +); verify(customRateLimitPolicy).onThrottle(same(e)); -verify(customRateLimitPolicy).onSuccess(eq(ImmutableList.of(a, b))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(c))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(d))); Review comment: > Would you share this? +1, please @suztomo What bug are you suggesting this exposes? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] jfarr commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky
jfarr commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky URL: https://github.com/apache/beam/pull/11090#discussion_r391341830 ## File path: sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java ## @@ -324,10 +325,15 @@ public void shouldCallRateLimitPolicy() } } +ArgumentCaptor> recordsCaptor = ArgumentCaptor.forClass(List.class); +verify(customRateLimitPolicy, atLeastOnce()).onSuccess(recordsCaptor.capture()); +List> capturedRecords = recordsCaptor.getAllValues(); +assertThat(capturedRecords).contains( +ImmutableList.of(a, b), +singletonList(c), +singletonList(d), +Collections.emptyList() +); verify(customRateLimitPolicy).onThrottle(same(e)); -verify(customRateLimitPolicy).onSuccess(eq(ImmutableList.of(a, b))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(c))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(d))); Review comment: > Would you share this? +1, please @suztomo What bug are you suggesting this exposes? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11103: Reifying outputs from BQ file writing
pabloem commented on issue #11103: Reifying outputs from BQ file writing URL: https://github.com/apache/beam/pull/11103#issuecomment-597944726 for my info: may also need to use beam bq sink here https://github.com/apache/beam/blob/d62521f69ead4b58924043f041978d49e9beeb62/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L1009 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11103: Reifying outputs from BQ file writing
pabloem commented on issue #11103: Reifying outputs from BQ file writing URL: https://github.com/apache/beam/pull/11103#issuecomment-597944352 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11103: Reifying outputs from BQ file writing
pabloem commented on issue #11103: Reifying outputs from BQ file writing URL: https://github.com/apache/beam/pull/11103#issuecomment-597944307 I'll make a case to include this tonight. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-597943970 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem commented on issue #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head URL: https://github.com/apache/beam/pull/11106#issuecomment-597944001 Run Python 2 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem opened a new pull request #11106: Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
pabloem opened a new pull request #11106: Fixing
apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT. at head
URL: https://github.com/apache/beam/pull/11106
Fixing apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT.test_file_loads
which are causing trouble at head.
The problem here is that AVRO does not encode b'1' into int(1) thus failing
the pipeline. I would say this test relies on the BQ implementation detail that
is able to import {'number': '1'} json as numeric. This fixes the test. Another
possible fix of the test is to parse the bytes into actual numbers.
r: @chamikaramj
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] [**Choose
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
| --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/
[GitHub] [beam] TheNeuralBit opened a new pull request #11105: [BEAM-9297] Verify schema early in ToJson and JsonToRow
TheNeuralBit opened a new pull request #11105: [BEAM-9297] Verify schema early in ToJson and JsonToRow URL: https://github.com/apache/beam/pull/11105 Move RowJsonValidation functions into RowJson. Do schema validation early in ToJsoin and JsonToRow, so we can fail early for schemas that won't work. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)[
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0e88219 [BEAM-7815] update MemoryReporter comments about using guppy3 (#11073) add cf219ba [BEAM-9474] Improve robustness of BundleFactory and ProcessEnvironment add d62521f Merge pull request #11084: [BEAM-9474] Improve robustness of BundleFactory and ProcessEnvironment No new revisions were added by this update. Summary of changes: .../control/DefaultJobBundleFactory.java | 101 + .../environment/ProcessEnvironment.java| 28 -- .../fnexecution/environment/ProcessManager.java| 50 +- 3 files changed, 128 insertions(+), 51 deletions(-)
[GitHub] [beam] pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005#issuecomment-597934081 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005#issuecomment-597934026 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] tweise commented on a change in pull request #11084: [BEAM-9474] Improve robustness of BundleFactory and ProcessEnvironment
tweise commented on a change in pull request #11084: [BEAM-9474] Improve
robustness of BundleFactory and ProcessEnvironment
URL: https://github.com/apache/beam/pull/11084#discussion_r391330585
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessManager.java
##
@@ -186,31 +186,28 @@ private void stopProcess(String id, Process process) {
LOG.debug("Attempting to stop process with id {}", id);
// first try to kill gracefully
process.destroy();
- long maxTimeToWait = 2000;
- if (waitForProcessToDie(process, maxTimeToWait)) {
-LOG.debug("Process for worker {} shut down gracefully.", id);
- } else {
-LOG.info("Process for worker {} still running. Killing.", id);
-process.destroyForcibly();
+ long maxTimeToWait = 500;
Review comment:
Won't delay the PR for it!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chadrik commented on issue #11038: [BEAM-7746] More typing fixes
chadrik commented on issue #11038: [BEAM-7746] More typing fixes URL: https://github.com/apache/beam/pull/11038#issuecomment-597933410 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka commented on issue #11103: Reifying outputs from BQ file writing
angoenka commented on issue #11103: Reifying outputs from BQ file writing URL: https://github.com/apache/beam/pull/11103#issuecomment-597933267 Let's get this cherry picked in 2.20 if possible This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11104: Update BigQuery source in bigquery_tornadoes example
pabloem commented on issue #11104: Update BigQuery source in bigquery_tornadoes example URL: https://github.com/apache/beam/pull/11104#issuecomment-597931204 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-597930882 ah good catch. Yes, that would make it a bug.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] y1chi commented on issue #11104: Update BigQuery source in bigquery_tornadoes example
y1chi commented on issue #11104: Update BigQuery source in bigquery_tornadoes example URL: https://github.com/apache/beam/pull/11104#issuecomment-597930317 R: @pabloem This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] y1chi opened a new pull request #11104: Update BigQuery source in bigquery_tornadoes example
y1chi opened a new pull request #11104: Update BigQuery source in bigquery_tornadoes example URL: https://github.com/apache/beam/pull/11104 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[GitHub] [beam] TheNeuralBit commented on issue #11102: [BEAM-9294] Move RowJsonException out of RowJsonSerializer
TheNeuralBit commented on issue #11102: [BEAM-9294] Move RowJsonException out of RowJsonSerializer URL: https://github.com/apache/beam/pull/11102#issuecomment-597928454 Yeah, sorry looks like IntelliJ refactoring somehow missed some references. Working on fixes now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] apilloud commented on issue #11102: [BEAM-9294] Move RowJsonException out of RowJsonSerializer
apilloud commented on issue #11102: [BEAM-9294] Move RowJsonException out of RowJsonSerializer URL: https://github.com/apache/beam/pull/11102#issuecomment-597927414 Doesn't look unreasonable, but as is this breaks the build: ``` 15:29:44 > Task :sdks:java:core:compileJava 15:29:44 sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValueExtractors.java:25: error: cannot find symbol 15:29:44 import org.apache.beam.sdk.util.RowJson.RowJsonDeserializer.UnsupportedRowJsonException; 15:29:44^ 15:29:44 symbol: class UnsupportedRowJsonException 15:29:44 location: class RowJsonDeserializer 15:29:44 sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonUtils.java:27: error: cannot find symbol 15:29:44 import org.apache.beam.sdk.util.RowJson.RowJsonDeserializer.UnsupportedRowJsonException; 15:29:44^ 15:29:44 symbol: class UnsupportedRowJsonException 15:29:44 location: class RowJsonDeserializer 15:29:44 Note: Some input files use or override a deprecated API. 15:29:44 Note: Recompile with -Xlint:deprecation for details. 15:29:44 2 errors 15:29:44 15:29:44 > Task :sdks:java:core:compileJava FAILED ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11103: Reifying outputs from BQ file writing
pabloem commented on issue #11103: Reifying outputs from BQ file writing URL: https://github.com/apache/beam/pull/11103#issuecomment-597926517 r: @lukecwik This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005#issuecomment-597925638 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005#issuecomment-597925574 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005#issuecomment-597925663 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] aaltay commented on issue #11075: [BEAM-9421] Website section that describes getting predictions using AI Platform Prediciton
aaltay commented on issue #11075: [BEAM-9421] Website section that describes getting predictions using AI Platform Prediciton URL: https://github.com/apache/beam/pull/11075#issuecomment-597922627 According to https://github.com/tensorflow/tfx-bsl > This package is not intended for direct use by TFX users, and its APIs should be considered internal to TFX (therefore there is no backward or forward compatibility guarantee) unless otherwise remarked. Maybe we need to wait until tfx_bsl is available as a public solution. @yifanmai @katsiapis -- please advise on when/how we should move forward. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated (facb4ef -> 0e88219)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from facb4ef Merge pull request #10997 from angoenka/fix_test_multi_triggered_gbk_side_input add 0e88219 [BEAM-7815] update MemoryReporter comments about using guppy3 (#11073) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/utils/profiler.py | 4 +++- sdks/python/container/base_image_requirements.txt | 3 ++- 2 files changed, 5 insertions(+), 2 deletions(-)
[GitHub] [beam] tvalentyn commented on issue #11073: [BEAM-7815] update MemoryReporter comments about using guppy3
tvalentyn commented on issue #11073: [BEAM-7815] update MemoryReporter comments about using guppy3 URL: https://github.com/apache/beam/pull/11073#issuecomment-597922353 Thanks, @lazylynx ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] tvalentyn edited a comment on issue #11073: [BEAM-7815] update MemoryReporter comments about using guppy3
tvalentyn edited a comment on issue #11073: [BEAM-7815] update MemoryReporter comments about using guppy3 URL: https://github.com/apache/beam/pull/11073#issuecomment-597921931 Test passed: https://builds.apache.org/job/beam_PostCommit_Py_ValCont_PR/120/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] tvalentyn merged pull request #11073: [BEAM-7815] update MemoryReporter comments about using guppy3
tvalentyn merged pull request #11073: [BEAM-7815] update MemoryReporter comments about using guppy3 URL: https://github.com/apache/beam/pull/11073 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11103: Reifying outputs from BQ file writing
pabloem commented on issue #11103: Reifying outputs from BQ file writing URL: https://github.com/apache/beam/pull/11103#issuecomment-597922046 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem opened a new pull request #11103: Reifying outputs from BQ file writing
pabloem opened a new pull request #11103: Reifying outputs from BQ file writing URL: https://github.com/apache/beam/pull/11103 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] tvalentyn commented on issue #11086: [BEAM-8910] Make custom BQ source read from Avro
tvalentyn commented on issue #11086: [BEAM-8910] Make custom BQ source read from Avro URL: https://github.com/apache/beam/pull/11086#issuecomment-597911424 > * For bytes, AvroSource is currently returning base64 encoded bytes. I am trying to figure out how to handle this. For py2, strings and bytes may not be differentiable, so we may not be able to handle this properly... Do you have a summary of Py2 vs Py3 x Current Behavior vs New behavior? What is returned on Py3 - is it also base64 encoded bytes? This is a prior discussion on Py3 migration issues in Bigquery: https://lists.apache.org/thread.html/f35c836887014e059527ed1a806e730321e2f9726164a3030575f455%40%3Cdev.beam.apache.org%3E. It took us some time to understand what was going on (https://lists.apache.org/thread.html/0c2178cf8e5d9e77c4f233f05a0b87b6011a1daa1a5ae47b41463af5%40%3Cdev.beam.apache.org%3E) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] suztomo commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky
suztomo commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky URL: https://github.com/apache/beam/pull/11090#discussion_r391300387 ## File path: sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java ## @@ -324,10 +325,15 @@ public void shouldCallRateLimitPolicy() } } +ArgumentCaptor> recordsCaptor = ArgumentCaptor.forClass(List.class); +verify(customRateLimitPolicy, atLeastOnce()).onSuccess(recordsCaptor.capture()); +List> capturedRecords = recordsCaptor.getAllValues(); +assertThat(capturedRecords).contains( +ImmutableList.of(a, b), +singletonList(c), +singletonList(d), +Collections.emptyList() +); verify(customRateLimitPolicy).onThrottle(same(e)); -verify(customRateLimitPolicy).onSuccess(eq(ImmutableList.of(a, b))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(c))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(d))); Review comment: I disagree (I’m not a committer). We just revealed a bug, which is very hard to reproduce. @aromanenko-dev > @RepeatedTest(1) and some hacks to run it on Junit5 Would you share this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] angoenka commented on issue #10997: [BEAM-9402] Remove options overwrite
angoenka commented on issue #10997: [BEAM-9402] Remove options overwrite URL: https://github.com/apache/beam/pull/10997#issuecomment-597903850 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] TheNeuralBit opened a new pull request #11102: [BEAM-9294] Move RowJsonException out of RowJsonSerializer
TheNeuralBit opened a new pull request #11102: [BEAM-9294] Move RowJsonException out of RowJsonSerializer URL: https://github.com/apache/beam/pull/11102 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[ were added to refs/heads/master by this push: new e6d1420 [BEAM-9402] Remove options overwrite new facb4ef Merge pull request #10997 from angoenka/fix_test_multi_triggered_gbk_side_input e6d1420 is described below commit e6d142087bd15a7b12661371fcb3572b218f7e42 Author: Ankur Goenka AuthorDate: Thu Feb 27 18:16:43 2020 -0800 [BEAM-9402] Remove options overwrite --- sdks/python/apache_beam/transforms/sideinputs_test.py | 11 --- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/sdks/python/apache_beam/transforms/sideinputs_test.py b/sdks/python/apache_beam/transforms/sideinputs_test.py index 20d2f6e..ddec122 100644 --- a/sdks/python/apache_beam/transforms/sideinputs_test.py +++ b/sdks/python/apache_beam/transforms/sideinputs_test.py @@ -27,8 +27,6 @@ import unittest from nose.plugins.attrib import attr import apache_beam as beam -from apache_beam.options.pipeline_options import DebugOptions -from apache_beam.options.pipeline_options import StandardOptions from apache_beam.testing.test_pipeline import TestPipeline from apache_beam.testing.test_stream import TestStream from apache_beam.testing.util import assert_that @@ -349,11 +347,10 @@ class SideInputsTest(unittest.TestCase): # This flag is only necessary when using the multi-output TestStream b/c # it relies on using the PCollection output tags as the PCollection output # ids. -options = StandardOptions(streaming=True) -options.view_as(DebugOptions).add_experiment( -'passthrough_pcollection_output_ids') - -p = TestPipeline(options=options) +p = TestPipeline( +additional_pipeline_args=[ +'--experiments=' + 'passthrough_pcollection_output_ids' +]) test_stream = ( p
[GitHub] [beam] angoenka merged pull request #10997: [BEAM-9402] Remove options overwrite
angoenka merged pull request #10997: [BEAM-9402] Remove options overwrite URL: https://github.com/apache/beam/pull/10997 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11086: [BEAM-8910] Make custom BQ source read from Avro
pabloem commented on issue #11086: [BEAM-8910] Make custom BQ source read from Avro URL: https://github.com/apache/beam/pull/11086#issuecomment-597900858 @chamikaramj @tvalentyn Hey there, I'm working on this improvement. There's some interesting outcomes here: - For datetime types (date,time,datetime,timestamp), AvroSource returns python types (datetime.date, datetime.time,datetime.datetime, datetime.datetime) - along with timezone info. I feel that in this case, Python types are the most desirable. Thoughts? _CustomBQSource is experimental, so I wouldn't think it's subject to backwards compatibility concerns. - For bytes, AvroSource is currently returning base64 encoded bytes. I am trying to figure out how to handle this. For py2, strings and bytes may not be differentiable, so we may not be able to handle this properly... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] jfarr commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky
jfarr commented on a change in pull request #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky URL: https://github.com/apache/beam/pull/11090#discussion_r391291506 ## File path: sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java ## @@ -324,10 +325,15 @@ public void shouldCallRateLimitPolicy() } } +ArgumentCaptor> recordsCaptor = ArgumentCaptor.forClass(List.class); +verify(customRateLimitPolicy, atLeastOnce()).onSuccess(recordsCaptor.capture()); +List> capturedRecords = recordsCaptor.getAllValues(); +assertThat(capturedRecords).contains( +ImmutableList.of(a, b), +singletonList(c), +singletonList(d), +Collections.emptyList() +); verify(customRateLimitPolicy).onThrottle(same(e)); -verify(customRateLimitPolicy).onSuccess(eq(ImmutableList.of(a, b))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(c))); -verify(customRateLimitPolicy).onSuccess(eq(singletonList(d))); Review comment: Yes, thanks Alexey. I was just working on something similar. Maybe the test is just overspecified then. I propose we change the validations to just: ``` verify(customRateLimitPolicy).onThrottle(any(KinesisClientThrottledException.class)); verify(customRateLimitPolicy, atLeastOnce()).onSuccess(any(List.class)); ``` If you both agree I'll make that change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] KevinGG commented on issue #11100: [BEAM-7923] Streaming support and pipeline pruning when instrumenting a pipeline with interactivity
KevinGG commented on issue #11100: [BEAM-7923] Streaming support and pipeline pruning when instrumenting a pipeline with interactivity URL: https://github.com/apache/beam/pull/11100#issuecomment-597895621 > I think this change is fine. But it was really difficult to review. > For this PR - Please add a more descriptive PR title. > For future PRs - Please separate unrelated changes from each other have logical commit sets. > > LGTM once the PR title and description is updated to be more specific. Thanks, Ahmet! Updated the title to `Streaming support and pipeline pruning when instrumenting a pipeline with interactivity`. I'll do several rebases when dependency PRs are merged and IM you for tests. For future PRs, we'll separate changes into more fine grained commit sets, thanks for the review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] HuangLED commented on issue #10997: [BEAM-9402] Remove options overwrite
HuangLED commented on issue #10997: [BEAM-9402] Remove options overwrite URL: https://github.com/apache/beam/pull/10997#issuecomment-597892970 Thanks for the fix. LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] youngoli commented on issue #10991: [BEAM-3301] Refactor DoFn validation & allow specifying main inputs.
youngoli commented on issue #10991: [BEAM-3301] Refactor DoFn validation & allow specifying main inputs. URL: https://github.com/apache/beam/pull/10991#issuecomment-597891830 R: @lukecwik Adding Luke to finish up this review since Robert (lostluck@) is on vacation for a bit. Since this was already mostly reviewed, the main thing I'm looking for is someone to confirm that I addressed Robert's previous review comments with the latest commit. You don't need to review the full change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch asf-site updated: Publishing website 2020/03/11 21:34:34 at commit e0a0041
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 2ae6b53 Publishing website 2020/03/11 21:34:34 at commit e0a0041 2ae6b53 is described below commit 2ae6b53eb14bb1099cb660f77209ea258fd2ca6e Author: jenkins AuthorDate: Wed Mar 11 21:34:34 2020 + Publishing website 2020/03/11 21:34:34 at commit e0a0041 --- website/generated-content/roadmap/python-sdk/index.html | 13 ++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/website/generated-content/roadmap/python-sdk/index.html b/website/generated-content/roadmap/python-sdk/index.html index 3b65828..0901cc7 100644 --- a/website/generated-content/roadmap/python-sdk/index.html +++ b/website/generated-content/roadmap/python-sdk/index.html @@ -227,6 +227,7 @@ + Sunsetting Python 2 Support Python 3 Support @@ -250,9 +251,13 @@ limitations under the License. Python SDK Roadmap +Sunsetting Python 2 Support + +The Apache Beam community voted to join https://python3statement.org/";>the pledge to sunset Python 2 support in 2020. The Beam community will discontinue Python 2 support in 2020 and cannot guarantee long-term functional support or maintenance of Beam on Python 2. To ensure minimal disruption to your service, we strongly recommend that you upgrade to Python 3 as soon as possible. + Python 3 Support -Apache Beam 2.14.0 and higher support Python 3.5, 3.6, and 3.7. We continue to https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050";>improve the experience for Python 3 users and phase out Python 2 support (https://issues.apache.org/jira/browse/BEAM-8371";>BEAM-8371): +Apache Beam 2.14.0 and higher support Python 3.5, 3.6, and 3.7. We’re continuing to https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050";>improve the experience for Python 3 users and add support for Python 3.x minor versions (https://issues.apache.org/jira/browse/BEAM-8494";>BEAM-8494): https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail";>Kanban Board @@ -263,9 +268,11 @@ limitations under the License. Contributions and feedback are welcome! -If you are interested to help, you can select an unassigned issue in the Kanban board and assign it to yourself. Comment on the issue if you cannot assign it yourself. When submitting a new PR, please tag https://github.com/aaltay";>@aaltay, and https://github.com/tvalentyn";>@tvalentyn. +If you are interested in helping, you can select an unassigned issue on the Kanban board and assign it to yourself. If you cannot assign the issue to yourself, comment on the issue. When submitting a new PR, please tag https://github.com/aaltay";>@aaltay, and https://github.com/tvalentyn";>@tvalentyn. + +To report a Python 3 related issue, create a subtask in https://issues.apache.org/jira/browse/BEAM-1251";>BEAM-1251 and cc: [~altay] and [~tvalentyn] in a JIRA comment. The best way to help us identify and investigate the issue is with a minimal pipeline that reproduces the issue. -To report a Python3-related issue, it is best to create a subtask to https://issues.apache.org/jira/browse/BEAM-1251";>BEAM-1251 , and cc: [~altay] and [~tvalentyn] in JIRA comment. You can also discuss encountered issues on user@ or dev@ mailing lists as appropriate. +You can also discuss encountered issues on user@ or dev@ mailing lists as appropriate.
[GitHub] [beam] tvalentyn commented on issue #11101: [BEAM-9074] Update Python roadmap for 2.7 eol
tvalentyn commented on issue #11101: [BEAM-9074] Update Python roadmap for 2.7 eol URL: https://github.com/apache/beam/pull/11101#issuecomment-597888718 Thank you, @rosetn ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005#issuecomment-597886665 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] aaltay merged pull request #11101: [BEAM-9074] Update Python roadmap for 2.7 eol
aaltay merged pull request #11101: [BEAM-9074] Update Python roadmap for 2.7 eol URL: https://github.com/apache/beam/pull/11101 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[beam] branch master updated: Update Python roadmap for 2.7 eol
This is an automated email from the ASF dual-hosted git repository. altay pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new a257613 Update Python roadmap for 2.7 eol new e0a0041 Merge pull request #11101 from rosetn/patch-1 a257613 is described below commit a25761342e5cb283e681c23661164c26af682390 Author: rosetn <40248483+ros...@users.noreply.github.com> AuthorDate: Wed Mar 11 12:52:12 2020 -0700 Update Python roadmap for 2.7 eol --- website/src/roadmap/python-sdk.md | 12 +--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/website/src/roadmap/python-sdk.md b/website/src/roadmap/python-sdk.md index f9e6b24..85e5a45 100644 --- a/website/src/roadmap/python-sdk.md +++ b/website/src/roadmap/python-sdk.md @@ -20,9 +20,13 @@ limitations under the License. # Python SDK Roadmap +## Sunsetting Python 2 Support + +The Apache Beam community voted to join [the pledge](https://python3statement.org/) to sunset Python 2 support in 2020. The Beam community will discontinue Python 2 support in 2020 and cannot guarantee long-term functional support or maintenance of Beam on Python 2. To ensure minimal disruption to your service, we strongly recommend that you upgrade to Python 3 as soon as possible. + ## Python 3 Support -Apache Beam 2.14.0 and higher support Python 3.5, 3.6, and 3.7. We continue to [improve](https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050) the experience for Python 3 users and phase out Python 2 support ([BEAM-8371](https://issues.apache.org/jira/browse/BEAM-8371)): +Apache Beam 2.14.0 and higher support Python 3.5, 3.6, and 3.7. We're continuing to [improve](https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050) the experience for Python 3 users and add support for Python 3.x minor versions ([BEAM-8494](https://issues.apache.org/jira/browse/BEAM-8494)): - [Kanban Board](https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail) @@ -32,6 +36,8 @@ Apache Beam 2.14.0 and higher support Python 3.5, 3.6, and 3.7. We continue to [ Contributions and feedback are welcome! -If you are interested to help, you can select an unassigned issue in the Kanban board and assign it to yourself. Comment on the issue if you cannot assign it yourself. When submitting a new PR, please tag [@aaltay](https://github.com/aaltay), and [@tvalentyn](https://github.com/tvalentyn). +If you are interested in helping, you can select an unassigned issue on the Kanban board and assign it to yourself. If you cannot assign the issue to yourself, comment on the issue. When submitting a new PR, please tag [@aaltay](https://github.com/aaltay), and [@tvalentyn](https://github.com/tvalentyn). + +To report a Python 3 related issue, create a subtask in [BEAM-1251](https://issues.apache.org/jira/browse/BEAM-1251) and cc: [~altay] and [~tvalentyn] in a JIRA comment. The best way to help us identify and investigate the issue is with a minimal pipeline that reproduces the issue. -To report a Python3-related issue, it is best to create a subtask to [BEAM-1251](https://issues.apache.org/jira/browse/BEAM-1251) , and cc: [~altay] and [~tvalentyn] in JIRA comment. You can also discuss encountered issues on user@ or dev@ mailing lists as appropriate. +You can also discuss encountered issues on user@ or dev@ mailing lists as appropriate.
[GitHub] [beam] pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005#issuecomment-597884434 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005#issuecomment-597884370 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] EDjur edited a comment on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
EDjur edited a comment on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-597872563 The `gcs_location` value provider is evaluated in the `expand` method: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py#L1575. Is that a bug then? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
EDjur commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-597872563 The `gcs_location` value provider is evaluated in the `expand` method: https://github.com/apache/beam/pull/11040/files#diff-683ba04a0e750cf0456b9b119393ad64R1587. Is that a bug then? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005#issuecomment-597871799 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11084: [BEAM-9474] Improve robustness of BundleFactory and ProcessEnvironment
mxm commented on a change in pull request #11084: [BEAM-9474] Improve
robustness of BundleFactory and ProcessEnvironment
URL: https://github.com/apache/beam/pull/11084#discussion_r391257741
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java
##
@@ -487,13 +553,14 @@ private int ref() {
}
private int unref() {
- int count = bundleRefCount.decrementAndGet();
- if (count == 0) {
+ int refCount = bundleRefCount.decrementAndGet();
+ Preconditions.checkState(refCount >= 0, "Reference count must not be
negative.");
Review comment:
FYI, I've added this check instead to check for correct bounds.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005#issuecomment-597869429 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11084: [BEAM-9474] Improve robustness of BundleFactory and ProcessEnvironment
mxm commented on a change in pull request #11084: [BEAM-9474] Improve robustness of BundleFactory and ProcessEnvironment URL: https://github.com/apache/beam/pull/11084#discussion_r391256990 ## File path: runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java ## @@ -352,20 +407,18 @@ public RemoteBundle getBundle( // The blocking queue of caches for serving multiple bundles concurrently. currentCache = availableCaches.take(); client = currentCache.getUnchecked(executableStage.getEnvironment()); -client.ref(); Review comment: Yes, that makes sense. I've already reverted the change. I suppose there is a race condition where we retrieve an environment X and before we can call `ref()` on it, we evict the environment X, close all its references, and shut it down. This will result in a job restart. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource
pabloem commented on issue #11040: [BEAM-9305] Allow value provider query strings in _CustomBigQuerySource URL: https://github.com/apache/beam/pull/11040#issuecomment-597869174 I'm srry @EDjur - the `expand` method runs at template construction, so the change currently will not work with parameters provided at execution time. You'd need to call `get` on the valueprovider in estimate_size/split/etc like we had discussed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager
pabloem commented on issue #11005: [BEAM-8335] Modify the StreamingCache to subclass the CacheManager URL: https://github.com/apache/beam/pull/11005#issuecomment-597866914 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11074: Store logical type values in Row instead of base values
reuvenlax commented on a change in pull request #11074: Store logical type
values in Row instead of base values
URL: https://github.com/apache/beam/pull/11074#discussion_r391253578
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoderHelpers.java
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.BigDecimalCoder;
+import org.apache.beam.sdk.coders.BigEndianShortCoder;
+import org.apache.beam.sdk.coders.BooleanCoder;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.ByteCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.DoubleCoder;
+import org.apache.beam.sdk.coders.FloatCoder;
+import org.apache.beam.sdk.coders.InstantCoder;
+import org.apache.beam.sdk.coders.IterableCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.LogicalType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+public class SchemaCoderHelpers {
+ // This contains a map of primitive types to their coders.
+ public static final Map CODER_MAP =
+ ImmutableMap.builder()
+ .put(TypeName.BYTE, ByteCoder.of())
+ .put(TypeName.BYTES, ByteArrayCoder.of())
+ .put(TypeName.INT16, BigEndianShortCoder.of())
+ .put(TypeName.INT32, VarIntCoder.of())
+ .put(TypeName.INT64, VarLongCoder.of())
+ .put(TypeName.DECIMAL, BigDecimalCoder.of())
+ .put(TypeName.FLOAT, FloatCoder.of())
+ .put(TypeName.DOUBLE, DoubleCoder.of())
+ .put(TypeName.STRING, StringUtf8Coder.of())
+ .put(TypeName.DATETIME, InstantCoder.of())
+ .put(TypeName.BOOLEAN, BooleanCoder.of())
+ .build();
+
+ private static class LogicalTypeCoder extends Coder {
Review comment:
Made these changes.
We should probably update SchemaCoder as well to do this properly, but that
can be in a different PR IMO.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #10834: Logs BQ insert failures
chamikaramj commented on issue #10834: Logs BQ insert failures URL: https://github.com/apache/beam/pull/10834#issuecomment-597863944 Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #10834: Logs BQ insert failures
chamikaramj commented on issue #10834: Logs BQ insert failures URL: https://github.com/apache/beam/pull/10834#issuecomment-597864107 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #10834: Logs BQ insert failures
chamikaramj commented on issue #10834: Logs BQ insert failures URL: https://github.com/apache/beam/pull/10834#issuecomment-597864589 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
