[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=419073&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-419073 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 09/Apr/20 02:10 Start Date: 09/Apr/20 02:10 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11356: [BEAM-9618] Mark push registration as deprecated. URL: https://github.com/apache/beam/pull/11356 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 419073) Time Spent: 5h 20m (was: 5h 10m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Blocker > Fix For: 2.21.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=418976&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418976 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 08/Apr/20 23:27 Start Date: 08/Apr/20 23:27 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11356: [BEAM-9618] Mark push registration as deprecated. URL: https://github.com/apache/beam/pull/11356 R: @lukecwik Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=418951&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418951 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 08/Apr/20 22:38 Start Date: 08/Apr/20 22:38 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11328: [BEAM-9618] Java SDK worker support for pulling bundle descriptors. URL: https://github.com/apache/beam/pull/11328 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 418951) Time Spent: 5h (was: 4h 50m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Blocker > Fix For: 2.21.0 > > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=418929&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418929 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 08/Apr/20 21:54 Start Date: 08/Apr/20 21:54 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11328: [BEAM-9618] Java SDK worker support for pulling bundle descriptors. URL: https://github.com/apache/beam/pull/11328#issuecomment-611214335 sgtm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 418929) Time Spent: 4h 50m (was: 4h 40m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Blocker > Fix For: 2.21.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=418921&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418921 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 08/Apr/20 21:41 Start Date: 08/Apr/20 21:41 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11328: [BEAM-9618] Java SDK worker support for pulling bundle descriptors. URL: https://github.com/apache/beam/pull/11328#issuecomment-611209675 Thanks, I'll try to R: @lukecwik you next time. I'll clean up RegisterHandler in a follow-up, but wanted to defer this 'till we're sure the service is rolled out and we have good testing there. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 418921) Time Spent: 4h 40m (was: 4.5h) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Blocker > Fix For: 2.21.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=418919&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418919 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 08/Apr/20 21:40 Start Date: 08/Apr/20 21:40 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11328: [BEAM-9618] Java SDK worker support for pulling bundle descriptors. URL: https://github.com/apache/beam/pull/11328#discussion_r405831616 ## File path: sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java ## @@ -194,10 +204,27 @@ public static void main( FinalizeBundleHandler finalizeBundleHandler = new FinalizeBundleHandler(options.as(GcsOptions.class).getExecutorService()); + LoadingCache bundleProcessors = Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 418919) Time Spent: 4.5h (was: 4h 20m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Blocker > Fix For: 2.21.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=418857&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418857 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 08/Apr/20 20:17 Start Date: 08/Apr/20 20:17 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11328: [BEAM-9618] Java SDK worker support for pulling bundle descriptors. URL: https://github.com/apache/beam/pull/11328#discussion_r405787478 ## File path: sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java ## @@ -194,10 +204,27 @@ public static void main( FinalizeBundleHandler finalizeBundleHandler = new FinalizeBundleHandler(options.as(GcsOptions.class).getExecutorService()); + LoadingCache bundleProcessors = Review comment: ```suggestion LoadingCache processBundleDescriptors = ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 418857) Time Spent: 4h 20m (was: 4h 10m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Blocker > Fix For: 2.21.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=418848&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418848 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 08/Apr/20 20:02 Start Date: 08/Apr/20 20:02 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11328: [BEAM-9618] Java SDK worker support for pulling bundle descriptors. URL: https://github.com/apache/beam/pull/11328#issuecomment-611164680 I typically miss reviews that aren't tagged with @lukecwik as I rarely filter by the review requested box since non committers can't use it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 418848) Time Spent: 4h 10m (was: 4h) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Blocker > Fix For: 2.21.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=418244&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418244 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 08/Apr/20 05:33 Start Date: 08/Apr/20 05:33 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11328: [BEAM-9618] Java SDK worker support for pulling bundle descriptors. URL: https://github.com/apache/beam/pull/11328#issuecomment-610760498 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 418244) Time Spent: 4h (was: 3h 50m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Blocker > Fix For: 2.21.0 > > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=418074&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418074 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 07/Apr/20 23:36 Start Date: 07/Apr/20 23:36 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11324: [BEAM-9618] Pull bundle descriptors for Go. URL: https://github.com/apache/beam/pull/11324 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 418074) Time Spent: 3h 50m (was: 3h 40m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Blocker > Fix For: 2.21.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=418070&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418070 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 07/Apr/20 23:18 Start Date: 07/Apr/20 23:18 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11328: [BEAM-9618] Java SDK worker support for pulling bundle descriptors. URL: https://github.com/apache/beam/pull/11328#issuecomment-610666263 The ParameterizedUnboundedSourceWrapperTest.testWatermarkEmission failures seem unrelated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 418070) Time Spent: 3h 40m (was: 3.5h) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=418068&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418068 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 07/Apr/20 23:17 Start Date: 07/Apr/20 23:17 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11324: [BEAM-9618] Pull bundle descriptors for Go. URL: https://github.com/apache/beam/pull/11324#issuecomment-610665988 Thanks. This also uses the new pull API for getting the bundle descriptors. I'll update the commit description and wait for all internal tests to pass before merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 418068) Time Spent: 3.5h (was: 3h 20m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=418054&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418054 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 07/Apr/20 22:47 Start Date: 07/Apr/20 22:47 Worklog Time Spent: 10m Work Description: lostluck commented on issue #11324: [BEAM-9618] Pull bundle descriptors for Go. URL: https://github.com/apache/beam/pull/11324#issuecomment-610657233 The only improvement I'd consider making in the future is using a [sync.Pool](https://golang.org/pkg/sync/#Pool) but that would require messing with finalizers to trigger the down runs. For now, lets not worry about garbage collecting the plans. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 418054) Time Spent: 3h 20m (was: 3h 10m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=417302&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-417302 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 06/Apr/20 23:28 Start Date: 06/Apr/20 23:28 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11328: [BEAM-9618] Java SDK worker support for pulling bundle descriptors. URL: https://github.com/apache/beam/pull/11328 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=417154&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-417154 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 06/Apr/20 21:53 Start Date: 06/Apr/20 21:53 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11235: [BEAM-9618] Pull bundle descriptors. URL: https://github.com/apache/beam/pull/11235 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 417154) Time Spent: 3h (was: 2h 50m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=417091&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-417091 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 06/Apr/20 20:24 Start Date: 06/Apr/20 20:24 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11235: [BEAM-9618] Pull bundle descriptors. URL: https://github.com/apache/beam/pull/11235#issuecomment-610017389 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 417091) Time Spent: 2h 50m (was: 2h 40m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=417057&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-417057 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 06/Apr/20 19:34 Start Date: 06/Apr/20 19:34 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11324: [BEAM-9618] Pull bundle descriptors for Go. URL: https://github.com/apache/beam/pull/11324 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build St
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=416795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-416795 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 06/Apr/20 17:52 Start Date: 06/Apr/20 17:52 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11235: [BEAM-9618] Pull bundle descriptors. URL: https://github.com/apache/beam/pull/11235#issuecomment-609943402 Fixed typo and strengthened test. I actually did think about GCing based on worker ID, but don't think this'll be a problem in practice as the lifetimes tend to be the same (and it is still possible to do if the server wants to keep track of what bundle processor descriptors are used by which clients, even if they don't send worker IDs, which is also being fixed). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 416795) Time Spent: 2.5h (was: 2h 20m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[
https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=416763&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-416763
]
ASF GitHub Bot logged work on BEAM-9618:
Author: ASF GitHub Bot
Created on: 06/Apr/20 16:55
Start Date: 06/Apr/20 16:55
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11235: [BEAM-9618]
Pull bundle descriptors.
URL: https://github.com/apache/beam/pull/11235#discussion_r404241747
##
File path:
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolServiceTest.java
##
@@ -120,4 +121,16 @@ public void onCompleted() {
latch.await();
assertThat(sawComplete.get(), is(true));
}
+
+ @Test
+ public void testUnknownBundle() throws Exception {
+BeamFnApi.GetProcessBundleDescriptorRequest request =
+BeamFnApi.GetProcessBundleDescriptorRequest.newBuilder()
+.setProcessBundleDescriptorId("missing")
+.build();
+StreamObserver responseObserver =
mock(StreamObserver.class);
+controlService.getProcessBundleDescriptor(request, responseObserver);
+
+verify(responseObserver).onError(any(StatusException.class));
Review comment:
Validate the "not found" shows up in the status message and/or status code
to ensure that we got the right StatusException
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 416763)
Time Spent: 2h 10m (was: 2h)
> Allow SDKs to pull process bundle descriptors.
> --
>
> Key: BEAM-9618
> URL: https://issues.apache.org/jira/browse/BEAM-9618
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[
https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=416764&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-416764
]
ASF GitHub Bot logged work on BEAM-9618:
Author: ASF GitHub Bot
Created on: 06/Apr/20 16:55
Start Date: 06/Apr/20 16:55
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11235: [BEAM-9618]
Pull bundle descriptors.
URL: https://github.com/apache/beam/pull/11235#discussion_r404242047
##
File path:
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientTest.java
##
@@ -175,4 +179,23 @@ public void testOnCloseMultipleListener() throws
Exception {
verify(mockConsumer1).accept(client);
verify(mockConsumer2).accept(client);
}
+
+ @Test
+ public void testRegisterNundle() {
Review comment:
```suggestion
public void testRegisterBundle() {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 416764)
Time Spent: 2h 20m (was: 2h 10m)
> Allow SDKs to pull process bundle descriptors.
> --
>
> Key: BEAM-9618
> URL: https://issues.apache.org/jira/browse/BEAM-9618
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=416684&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-416684 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 06/Apr/20 15:30 Start Date: 06/Apr/20 15:30 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11235: [BEAM-9618] Pull bundle descriptors. URL: https://github.com/apache/beam/pull/11235#discussion_r404183803 ## File path: sdks/python/apache_beam/runners/worker/worker_id_interceptor.py ## @@ -48,17 +49,23 @@ def __init__(self, worker_id=None): if worker_id: self._worker_id = worker_id + def intercept_unary_unary(self, continuation, client_call_details, request): +return self._intercept(continuation, client_call_details, request) Review comment: Removing the worker id means that process bundle descriptors need to be unique per control server instance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 416684) Time Spent: 1h 50m (was: 1h 40m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=416686&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-416686 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 06/Apr/20 15:30 Start Date: 06/Apr/20 15:30 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11235: [BEAM-9618] Pull bundle descriptors. URL: https://github.com/apache/beam/pull/11235#discussion_r404183803 ## File path: sdks/python/apache_beam/runners/worker/worker_id_interceptor.py ## @@ -48,17 +49,23 @@ def __init__(self, worker_id=None): if worker_id: self._worker_id = worker_id + def intercept_unary_unary(self, continuation, client_call_details, request): +return self._intercept(continuation, client_call_details, request) Review comment: Removing the worker id means that process bundle descriptors need to be unique per control server instance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 416686) Time Spent: 2h (was: 1h 50m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=415743&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-415743 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 03/Apr/20 21:23 Start Date: 03/Apr/20 21:23 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11235: [BEAM-9618] Pull bundle descriptors. URL: https://github.com/apache/beam/pull/11235#issuecomment-608674803 @lukecwik Was there anything more you wanted me to do on this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 415743) Time Spent: 1h 40m (was: 1.5h) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[
https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=414499&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414499
]
ASF GitHub Bot logged work on BEAM-9618:
Author: ASF GitHub Bot
Created on: 02/Apr/20 06:48
Start Date: 02/Apr/20 06:48
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11235: [BEAM-9618]
Pull bundle descriptors.
URL: https://github.com/apache/beam/pull/11235#discussion_r402086684
##
File path: sdks/python/apache_beam/runners/worker/sdk_worker.py
##
@@ -112,7 +113,11 @@ def __init__(self,
self._state_handler_factory = GrpcStateHandlerFactory(
self._state_cache, credentials)
self._profiler_factory = profiler_factory
-self._fns = {} # type: Dict[str, beam_fn_api_pb2.ProcessBundleDescriptor]
+self._fns = KeyedDefaultDict(
+lambda id: self._control_stub.GetProcessBundleDescriptor(
+beam_fn_api_pb2.GetProcessBundleDescriptorRequest(
Review comment:
The error gets raised and nothing gets into the map.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 414499)
Time Spent: 1.5h (was: 1h 20m)
> Allow SDKs to pull process bundle descriptors.
> --
>
> Key: BEAM-9618
> URL: https://issues.apache.org/jira/browse/BEAM-9618
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[
https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=414354&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414354
]
ASF GitHub Bot logged work on BEAM-9618:
Author: ASF GitHub Bot
Created on: 01/Apr/20 22:52
Start Date: 01/Apr/20 22:52
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11235: [BEAM-9618]
Pull bundle descriptors.
URL: https://github.com/apache/beam/pull/11235#discussion_r401943249
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java
##
@@ -105,12 +105,38 @@ public static FnApiControlClientPoolService
offeringClientsToPool(
return newClient.asResponseObserver();
}
+ @Override
+ public void getProcessBundleDescriptor(
+ BeamFnApi.GetProcessBundleDescriptorRequest request,
+ StreamObserver responseObserver) {
+final String workerId = headerAccessor.getSdkWorkerId();
+if (Strings.isNullOrEmpty(workerId)) {
+ // TODO(BEAM-4149): Enforce proper worker id.
+ LOG.warn("No worker_id header provided in getProcessBundleDescriptor
request");
Review comment:
Since this is a new method, we should force all users to supply it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 414354)
Time Spent: 20m (was: 10m)
> Allow SDKs to pull process bundle descriptors.
> --
>
> Key: BEAM-9618
> URL: https://issues.apache.org/jira/browse/BEAM-9618
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=414357&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414357 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 01/Apr/20 22:52 Start Date: 01/Apr/20 22:52 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11235: [BEAM-9618] Pull bundle descriptors. URL: https://github.com/apache/beam/pull/11235#discussion_r401948665 ## File path: sdks/python/apache_beam/runners/worker/worker_id_interceptor.py ## @@ -48,17 +49,23 @@ def __init__(self, worker_id=None): if worker_id: self._worker_id = worker_id + def intercept_unary_unary(self, continuation, client_call_details, request): Review comment: You might as well and implement the UnaryStreamClientInterceptor and StreamUnaryClientInterceptor now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 414357) Time Spent: 40m (was: 0.5h) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=414363&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414363 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 01/Apr/20 22:52 Start Date: 01/Apr/20 22:52 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11235: [BEAM-9618] Pull bundle descriptors. URL: https://github.com/apache/beam/pull/11235#discussion_r401955503 ## File path: sdks/python/apache_beam/runners/worker/worker_id_interceptor.py ## @@ -35,7 +35,8 @@ class _ClientCallDetails(collections.namedtuple( pass -class WorkerIdInterceptor(grpc.StreamStreamClientInterceptor): +class WorkerIdInterceptor(grpc.UnaryUnaryClientInterceptor, Review comment: test? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 414363) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[
https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=414358&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414358
]
ASF GitHub Bot logged work on BEAM-9618:
Author: ASF GitHub Bot
Created on: 01/Apr/20 22:52
Start Date: 01/Apr/20 22:52
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11235: [BEAM-9618]
Pull bundle descriptors.
URL: https://github.com/apache/beam/pull/11235#discussion_r401948041
##
File path: sdks/python/apache_beam/runners/worker/sdk_worker_test.py
##
@@ -115,8 +115,9 @@ def _check_fn_registration_multi_request(self, *args):
{item.id: item
for item in process_bundle_descriptors})
- def test_fn_registration(self):
-self._check_fn_registration_multi_request((1, 4), (4, 4))
+
+# def test_fn_registration(self):
+# self._check_fn_registration_multi_request((1, 4), (4, 4))
Review comment:
if we can, we should test the registration lookup failure case.
The success case will be easily be covered by other tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 414358)
Time Spent: 50m (was: 40m)
> Allow SDKs to pull process bundle descriptors.
> --
>
> Key: BEAM-9618
> URL: https://issues.apache.org/jira/browse/BEAM-9618
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[
https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=414356&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414356
]
ASF GitHub Bot logged work on BEAM-9618:
Author: ASF GitHub Bot
Created on: 01/Apr/20 22:52
Start Date: 01/Apr/20 22:52
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11235: [BEAM-9618]
Pull bundle descriptors.
URL: https://github.com/apache/beam/pull/11235#discussion_r401945553
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java
##
@@ -105,12 +105,38 @@ public static FnApiControlClientPoolService
offeringClientsToPool(
return newClient.asResponseObserver();
}
+ @Override
+ public void getProcessBundleDescriptor(
+ BeamFnApi.GetProcessBundleDescriptorRequest request,
+ StreamObserver responseObserver) {
+final String workerId = headerAccessor.getSdkWorkerId();
+if (Strings.isNullOrEmpty(workerId)) {
+ // TODO(BEAM-4149): Enforce proper worker id.
+ LOG.warn("No worker_id header provided in getProcessBundleDescriptor
request");
+}
+
+LOG.info("getProcessBundleDescriptor request with id {}", workerId);
+
+FnApiControlClient client;
+synchronized (lock) {
+ client = vendedClients.get(workerId);
+}
+LOG.warn("getProcessBundleDescriptor client {}", client);
+LOG.warn("getProcessBundleDescriptor id {}",
request.getProcessBundleDescriptorId());
+LOG.warn(
+"getProcessBundleDescriptor result {}",
+
client.getProcessBundleDescriptor(request.getProcessBundleDescriptorId()));
+responseObserver.onNext(
+
client.getProcessBundleDescriptor(request.getProcessBundleDescriptorId()));
Review comment:
Please return the NOT_FOUND status error if the process bundle descriptor is
null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 414356)
Time Spent: 0.5h (was: 20m)
> Allow SDKs to pull process bundle descriptors.
> --
>
> Key: BEAM-9618
> URL: https://issues.apache.org/jira/browse/BEAM-9618
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=414362&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414362 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 01/Apr/20 22:52 Start Date: 01/Apr/20 22:52 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11235: [BEAM-9618] Pull bundle descriptors. URL: https://github.com/apache/beam/pull/11235#discussion_r401952609 ## File path: runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java ## @@ -105,12 +105,38 @@ public static FnApiControlClientPoolService offeringClientsToPool( return newClient.asResponseObserver(); } + @Override + public void getProcessBundleDescriptor( Review comment: Please add failure tests to FnApiControlClientPoolService for NOT_FOUND and missing worker id. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 414362) Time Spent: 1h 10m (was: 1h) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[
https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=414364&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414364
]
ASF GitHub Bot logged work on BEAM-9618:
Author: ASF GitHub Bot
Created on: 01/Apr/20 22:52
Start Date: 01/Apr/20 22:52
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11235: [BEAM-9618]
Pull bundle descriptors.
URL: https://github.com/apache/beam/pull/11235#discussion_r401946646
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java
##
@@ -105,12 +105,38 @@ public static FnApiControlClientPoolService
offeringClientsToPool(
return newClient.asResponseObserver();
}
+ @Override
+ public void getProcessBundleDescriptor(
+ BeamFnApi.GetProcessBundleDescriptorRequest request,
+ StreamObserver responseObserver) {
+final String workerId = headerAccessor.getSdkWorkerId();
+if (Strings.isNullOrEmpty(workerId)) {
Review comment:
We should check that we are not closed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 414364)
Time Spent: 1h 20m (was: 1h 10m)
> Allow SDKs to pull process bundle descriptors.
> --
>
> Key: BEAM-9618
> URL: https://issues.apache.org/jira/browse/BEAM-9618
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[
https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=414355&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414355
]
ASF GitHub Bot logged work on BEAM-9618:
Author: ASF GitHub Bot
Created on: 01/Apr/20 22:52
Start Date: 01/Apr/20 22:52
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11235: [BEAM-9618]
Pull bundle descriptors.
URL: https://github.com/apache/beam/pull/11235#discussion_r401946210
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java
##
@@ -105,12 +105,38 @@ public static FnApiControlClientPoolService
offeringClientsToPool(
return newClient.asResponseObserver();
}
+ @Override
+ public void getProcessBundleDescriptor(
+ BeamFnApi.GetProcessBundleDescriptorRequest request,
+ StreamObserver responseObserver) {
+final String workerId = headerAccessor.getSdkWorkerId();
+if (Strings.isNullOrEmpty(workerId)) {
+ // TODO(BEAM-4149): Enforce proper worker id.
+ LOG.warn("No worker_id header provided in getProcessBundleDescriptor
request");
+}
+
+LOG.info("getProcessBundleDescriptor request with id {}", workerId);
Review comment:
```suggestion
LOG.info("getProcessBundleDescriptor: descriptor {} worker id {}",
request.getProcessBundleDescriptorId(), workerId);
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 414355)
Time Spent: 20m (was: 10m)
> Allow SDKs to pull process bundle descriptors.
> --
>
> Key: BEAM-9618
> URL: https://issues.apache.org/jira/browse/BEAM-9618
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[
https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=414361&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414361
]
ASF GitHub Bot logged work on BEAM-9618:
Author: ASF GitHub Bot
Created on: 01/Apr/20 22:52
Start Date: 01/Apr/20 22:52
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11235: [BEAM-9618]
Pull bundle descriptors.
URL: https://github.com/apache/beam/pull/11235#discussion_r401954623
##
File path: sdks/python/apache_beam/runners/worker/sdk_worker.py
##
@@ -112,7 +113,11 @@ def __init__(self,
self._state_handler_factory = GrpcStateHandlerFactory(
self._state_cache, credentials)
self._profiler_factory = profiler_factory
-self._fns = {} # type: Dict[str, beam_fn_api_pb2.ProcessBundleDescriptor]
+self._fns = KeyedDefaultDict(
+lambda id: self._control_stub.GetProcessBundleDescriptor(
+beam_fn_api_pb2.GetProcessBundleDescriptorRequest(
Review comment:
What appears in the default dict if GetProcessBundleDescriptor raises an
error because the call fails?
Does it get propagated to the caller and it fails processing the bundle (is
anything left in the map)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 414361)
Time Spent: 1h 10m (was: 1h)
> Allow SDKs to pull process bundle descriptors.
> --
>
> Key: BEAM-9618
> URL: https://issues.apache.org/jira/browse/BEAM-9618
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[
https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=414359&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414359
]
ASF GitHub Bot logged work on BEAM-9618:
Author: ASF GitHub Bot
Created on: 01/Apr/20 22:52
Start Date: 01/Apr/20 22:52
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11235: [BEAM-9618]
Pull bundle descriptors.
URL: https://github.com/apache/beam/pull/11235#discussion_r401946261
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java
##
@@ -105,12 +105,38 @@ public static FnApiControlClientPoolService
offeringClientsToPool(
return newClient.asResponseObserver();
}
+ @Override
+ public void getProcessBundleDescriptor(
+ BeamFnApi.GetProcessBundleDescriptorRequest request,
+ StreamObserver responseObserver) {
+final String workerId = headerAccessor.getSdkWorkerId();
+if (Strings.isNullOrEmpty(workerId)) {
+ // TODO(BEAM-4149): Enforce proper worker id.
+ LOG.warn("No worker_id header provided in getProcessBundleDescriptor
request");
+}
+
+LOG.info("getProcessBundleDescriptor request with id {}", workerId);
+
+FnApiControlClient client;
+synchronized (lock) {
+ client = vendedClients.get(workerId);
+}
+LOG.warn("getProcessBundleDescriptor client {}", client);
+LOG.warn("getProcessBundleDescriptor id {}",
request.getProcessBundleDescriptorId());
+LOG.warn(
+"getProcessBundleDescriptor result {}",
+
client.getProcessBundleDescriptor(request.getProcessBundleDescriptorId()));
Review comment:
Did you intend to keep all these logs?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 414359)
Time Spent: 50m (was: 40m)
> Allow SDKs to pull process bundle descriptors.
> --
>
> Key: BEAM-9618
> URL: https://issues.apache.org/jira/browse/BEAM-9618
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=414360&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-414360 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 01/Apr/20 22:52 Start Date: 01/Apr/20 22:52 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11235: [BEAM-9618] Pull bundle descriptors. URL: https://github.com/apache/beam/pull/11235#discussion_r401952191 ## File path: sdks/python/apache_beam/runners/worker/worker_id_interceptor.py ## @@ -48,17 +49,23 @@ def __init__(self, worker_id=None): if worker_id: self._worker_id = worker_id + def intercept_unary_unary(self, continuation, client_call_details, request): +return self._intercept(continuation, client_call_details, request) Review comment: Hmm, each RPC will need to pass the worker_id which is a minor argument for bidi stream but not worth the complexity of it compared to the unary RPC. We also don't expect this to happen a lot. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 414360) Time Spent: 1h (was: 50m) > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9618) Allow SDKs to pull process bundle descriptors.
[ https://issues.apache.org/jira/browse/BEAM-9618?focusedWorklogId=410747&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-410747 ] ASF GitHub Bot logged work on BEAM-9618: Author: ASF GitHub Bot Created on: 27/Mar/20 01:03 Start Date: 27/Mar/20 01:03 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11235: [BEAM-9618] Pull bundle descriptors. URL: https://github.com/apache/beam/pull/11235#issuecomment-604762162 R: @lukecwik this is rebased and should be ready for review. I will remove commit cebab89 before merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 410747) Remaining Estimate: 0h Time Spent: 10m > Allow SDKs to pull process bundle descriptors. > -- > > Key: BEAM-9618 > URL: https://issues.apache.org/jira/browse/BEAM-9618 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
