[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=496868&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496868
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 07/Oct/20 19:11
Start Date: 07/Oct/20 19:11
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request #7316:
URL: https://github.com/apache/beam/pull/7316#discussion_r501248667
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java
##
@@ -629,13 +635,15 @@ private void visit(
if (!isRootNode()) {
// Visit inputs.
for (PValue inputValue : inputs.values()) {
- Node valueProducer = getProducer(inputValue);
- if (!visitedNodes.contains(valueProducer)) {
-valueProducer.visit(visitor, visitedValues, visitedNodes,
skippedComposites);
- }
- if (visitedValues.add(inputValue)) {
-LOG.debug("Visiting input value {}", inputValue);
-visitor.visitValue(inputValue, valueProducer);
+ Node valueProducer = maybeGetProducer(inputValue);
+ if (valueProducer != null) {
+if (!visitedNodes.contains(valueProducer)) {
+ valueProducer.visit(visitor, visitedValues, visitedNodes,
skippedComposites);
+}
+if (visitedValues.add(inputValue)) {
Review comment:
I was just looking into side inputs. I believe that this change means
any value with a direct producer is not visited. So `PCollectionView` is only
visited because it is produced by a transform. But in fact if we remove the
deprecated `WritePCollectionView` primitive they are never visited. I think
they should be visited still. I'm not exactly sure the ramifications. I'll post
a PR if I figure it out but LMK if you know already whether this will break.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 496868)
Time Spent: 7h 10m (was: 7h)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: P2
> Labels: Done
> Fix For: Not applicable
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=194860&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194860 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 05/Feb/19 23:29 Start Date: 05/Feb/19 23:29 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #7729: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7729 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194860) Time Spent: 7h (was: 6h 50m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 7h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=194838&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194838 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 05/Feb/19 22:23 Start Date: 05/Feb/19 22:23 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #7729: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7729#issuecomment-460827978 LGTM. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194838) Time Spent: 6h 50m (was: 6h 40m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 6h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=194818&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194818 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 05/Feb/19 21:34 Start Date: 05/Feb/19 21:34 Worklog Time Spent: 10m Work Description: robertwb commented on issue #7729: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7729#issuecomment-460812685 Run Java PortabilityApi PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194818) Time Spent: 6h 40m (was: 6.5h) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 6h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=194524&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194524 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 05/Feb/19 14:11 Start Date: 05/Feb/19 14:11 Worklog Time Spent: 10m Work Description: robertwb commented on issue #7729: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7729#issuecomment-460651142 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194524) Time Spent: 6.5h (was: 6h 20m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 6.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=194507&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194507 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 05/Feb/19 13:13 Start Date: 05/Feb/19 13:13 Worklog Time Spent: 10m Work Description: robertwb commented on issue #7729: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7729#issuecomment-460632950 Run Java PortabilityApi PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194507) Time Spent: 6h 20m (was: 6h 10m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 6h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=194506&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194506 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 05/Feb/19 13:13 Start Date: 05/Feb/19 13:13 Worklog Time Spent: 10m Work Description: robertwb commented on issue #7729: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7729#issuecomment-460632755 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194506) Time Spent: 6h 10m (was: 6h) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 6h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=194462&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194462 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 05/Feb/19 10:43 Start Date: 05/Feb/19 10:43 Worklog Time Spent: 10m Work Description: robertwb commented on issue #7729: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7729#issuecomment-460591247 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194462) Time Spent: 5h 50m (was: 5h 40m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 5h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=194463&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194463 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 05/Feb/19 10:43 Start Date: 05/Feb/19 10:43 Worklog Time Spent: 10m Work Description: robertwb commented on issue #7729: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7729#issuecomment-460591302 Run Java PortabilityApi PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194463) Time Spent: 6h (was: 5h 50m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 6h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=194447&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194447 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 05/Feb/19 09:44 Start Date: 05/Feb/19 09:44 Worklog Time Spent: 10m Work Description: robertwb commented on issue #7729: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7729#issuecomment-460573523 Run Java PortabilityApi PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194447) Time Spent: 5.5h (was: 5h 20m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=194448&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194448 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 05/Feb/19 09:44 Start Date: 05/Feb/19 09:44 Worklog Time Spent: 10m Work Description: robertwb commented on issue #7729: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7729#issuecomment-460573581 R: @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194448) Time Spent: 5h 40m (was: 5.5h) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 5h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=194446&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194446 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 05/Feb/19 09:43 Start Date: 05/Feb/19 09:43 Worklog Time Spent: 10m Work Description: robertwb commented on issue #7729: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7729#issuecomment-460573323 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194446) Time Spent: 5h 20m (was: 5h 10m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 5h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=194445&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194445 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 05/Feb/19 09:43 Start Date: 05/Feb/19 09:43 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #7729: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7729 This is the same as https://github.com/apache/beam/pull/7316 without making a standard coder for Strings. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194445) Time Spent: 5h 10m (was: 5h) > Support mixed-langu
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=191220&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191220 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 28/Jan/19 19:21 Start Date: 28/Jan/19 19:21 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #7316: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7316 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191220) Time Spent: 5h (was: 4h 50m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=191113&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191113
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 28/Jan/19 16:37
Start Date: 28/Jan/19 16:37
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #7316: [BEAM-6269]
Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r251491172
##
File path: sdks/python/apache_beam/pipeline.py
##
@@ -673,7 +674,7 @@ def from_runner_api(proto, runner, options,
return_context=False):
pcollection = context.pcollections.get_by_id(id)
pcollection.pipeline = p
if not pcollection.producer:
-raise ValueError('No producer for %s' % id)
+pass # raise ValueError('No producer for %s' % id)
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 191113)
Time Spent: 4h 40m (was: 4.5h)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=191112&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191112 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 28/Jan/19 16:37 Start Date: 28/Jan/19 16:37 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #7316: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7316#discussion_r251493376 ## File path: sdks/python/apache_beam/runners/pipeline_context.py ## @@ -89,6 +93,18 @@ def get_by_id(self, id): self._id_to_proto[id], self._pipeline_context) return self._id_to_obj[id] + def get_by_proto(self, maybe_new_proto, label=None, deduplicate=False): +if deduplicate: Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191112) Time Spent: 4.5h (was: 4h 20m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=191114&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191114 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 28/Jan/19 16:37 Start Date: 28/Jan/19 16:37 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #7316: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7316#discussion_r251491399 ## File path: sdks/python/apache_beam/transforms/external_test.py ## @@ -0,0 +1,205 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +"""Unit tests for the transform.util classes.""" + +from __future__ import absolute_import + +import argparse +import subprocess +import sys +import unittest + +import grpc +from past.builtins import unicode + +import apache_beam as beam +from apache_beam.portability import python_urns +from apache_beam.runners.portability import expansion_service +from apache_beam.testing.util import assert_that +from apache_beam.testing.util import equal_to +from apache_beam.transforms import ptransform + + +class ExternalTransformTest(unittest.TestCase): Review comment: This is run as part of unit tests. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191114) Time Spent: 4h 50m (was: 4h 40m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=191068&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191068 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 28/Jan/19 15:29 Start Date: 28/Jan/19 15:29 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #7316: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7316#discussion_r251460769 ## File path: sdks/python/apache_beam/transforms/external_test.py ## @@ -0,0 +1,205 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +"""Unit tests for the transform.util classes.""" + +from __future__ import absolute_import + +import argparse +import subprocess +import sys +import unittest + +import grpc +from past.builtins import unicode + +import apache_beam as beam +from apache_beam.portability import python_urns +from apache_beam.runners.portability import expansion_service +from apache_beam.testing.util import assert_that +from apache_beam.testing.util import equal_to +from apache_beam.transforms import ptransform + + +class ExternalTransformTest(unittest.TestCase): Review comment: Do we have test coverage for this ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191068) Time Spent: 4h 20m (was: 4h 10m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 4h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=191067&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191067 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 28/Jan/19 15:29 Start Date: 28/Jan/19 15:29 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #7316: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7316#discussion_r251461833 ## File path: sdks/python/apache_beam/runners/pipeline_context.py ## @@ -89,6 +93,18 @@ def get_by_id(self, id): self._id_to_proto[id], self._pipeline_context) return self._id_to_obj[id] + def get_by_proto(self, maybe_new_proto, label=None, deduplicate=False): +if deduplicate: Review comment: Add a test to pipeline_context_test.py to cover this path ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191067) Time Spent: 4h 10m (was: 4h) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 4h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=191066&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191066
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 28/Jan/19 15:29
Start Date: 28/Jan/19 15:29
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #7316:
[BEAM-6269] Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r251457240
##
File path: sdks/python/apache_beam/pipeline.py
##
@@ -673,7 +674,7 @@ def from_runner_api(proto, runner, options,
return_context=False):
pcollection = context.pcollections.get_by_id(id)
pcollection.pipeline = p
if not pcollection.producer:
-raise ValueError('No producer for %s' % id)
+pass # raise ValueError('No producer for %s' % id)
Review comment:
nit: update/delete comment ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 191066)
Time Spent: 4h 10m (was: 4h)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=190063&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-190063 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 25/Jan/19 15:49 Start Date: 25/Jan/19 15:49 Worklog Time Spent: 10m Work Description: robertwb commented on issue #7316: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7316#issuecomment-457617143 Pylint itself looks broken. Passed on my machine, all other tests are good. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 190063) Time Spent: 4h (was: 3h 50m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=189006&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189006
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 23/Jan/19 15:11
Start Date: 23/Jan/19 15:11
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #7316:
[BEAM-6269] Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r250233838
##
File path: model/job-management/src/main/proto/beam_expansion_api.proto
##
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Protocol Buffers describing the Expansion API, an api for expanding
+ * transforms in a remote SDK.
+ */
+
+syntax = "proto3";
+
+package org.apache.beam.model.expansion.v1;
+
+option go_package = "construction_v1";
+option java_package = "org.apache.beam.model.expansion.v1";
+option java_outer_classname = "ExpansionApi";
+
+import "beam_runner_api.proto";
+
+message ExpansionRequest {
+ // Set of components needed to interpret the transform, or which
+ // may be useful for its expansion. This includes the input
+ // PCollections (if any) to the to-be-expanded transform, along
+ // with their coders and windowing strategies.
+ org.apache.beam.model.pipeline.v1.Components components = 1;
+
+ // The actual PTransform to be expaneded according to its spec.
+ // Its input should be set, but its subtransforms and outputs
+ // should not be.
+ org.apache.beam.model.pipeline.v1.PTransform transform = 2;
+
+ // A namespace (prefix) to use for the id of any newly created
+ // components.
+ string namespace = 3;
Review comment:
Ah, I had forgotten about the opaque payloads. That's a good point. We could
address this (for example by having Components elsewhere than just the top
level) but I don't think we should. All is good here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 189006)
Time Spent: 3h 50m (was: 3h 40m)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=188968&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188968
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 23/Jan/19 14:05
Start Date: 23/Jan/19 14:05
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #7316: [BEAM-6269]
Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r250185036
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
##
@@ -239,10 +273,10 @@ public String getOnlyEnvironmentId() {
}
private String uniqify(String baseName, Set existing) {
-String name = baseName;
+String name = namespace + baseName;
Review comment:
We can't really preserve structure here, as these will be string ids (and as
well as being a huge rewrite, maps with non-primitive keys are painful to
dealwith). Also, namespaces may be hierarchical (on both sides) so we don't
want to introduce escaping (at least not on the LHS).
Namespace is nothing more than a required prefix; I'll document that better.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 188968)
Time Spent: 3.5h (was: 3h 20m)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=188969&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188969
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 23/Jan/19 14:05
Start Date: 23/Jan/19 14:05
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #7316: [BEAM-6269]
Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r250206908
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
##
@@ -81,9 +101,13 @@ public static SdkComponents create(PipelineOptions
options) {
return sdkComponents;
}
- private SdkComponents() {}
+ private SdkComponents(String namespace) {
+this.namespace = namespace;
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 188969)
Time Spent: 3h 40m (was: 3.5h)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=188970&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188970
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 23/Jan/19 14:05
Start Date: 23/Jan/19 14:05
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #7316: [BEAM-6269]
Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r250189663
##
File path: model/job-management/src/main/proto/beam_expansion_api.proto
##
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Protocol Buffers describing the Expansion API, an api for expanding
+ * transforms in a remote SDK.
+ */
+
+syntax = "proto3";
+
+package org.apache.beam.model.expansion.v1;
+
+option go_package = "construction_v1";
+option java_package = "org.apache.beam.model.expansion.v1";
+option java_outer_classname = "ExpansionApi";
+
+import "beam_runner_api.proto";
+
+message ExpansionRequest {
+ // Set of components needed to interpret the transform, or which
+ // may be useful for its expansion. This includes the input
+ // PCollections (if any) to the to-be-expanded transform, along
+ // with their coders and windowing strategies.
+ org.apache.beam.model.pipeline.v1.Components components = 1;
+
+ // The actual PTransform to be expaneded according to its spec.
+ // Its input should be set, but its subtransforms and outputs
+ // should not be.
+ org.apache.beam.model.pipeline.v1.PTransform transform = 2;
+
+ // A namespace (prefix) to use for the id of any newly created
+ // components.
+ string namespace = 3;
Review comment:
Yes, that's what I was hoping to do initially, but unfortunately the problem
is deeper than that. Re-allocating ids is not possible because they may be
referenced from within the encoded (and possibly opaque) payloads of various
transforms. Some examples that come to mind are AssignWindows (which references
a windowing strategy), Combine* (which reference an accumulator coder), and
ParDo (whose states, and possibly timers, also reference coders by id). And of
course environment ids may be referenced arbitrarily deeply.
The alternative is having a closed list of transforms that are allowed to
reference other components (and whose payloads are known and must be
re-written), prohibiting this for all others. Either that or any
payload-carrying object must have its own components, a mapping between local
names and internal names, or both.
On the other hand, we freely re-allocate ids for collections and transforms
during optimization and execution. This should probably be documented better.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 188970)
Time Spent: 3h 40m (was: 3.5h)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=188534&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188534
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 22/Jan/19 23:05
Start Date: 22/Jan/19 23:05
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #7316:
[BEAM-6269] Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r249995295
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
##
@@ -239,10 +273,10 @@ public String getOnlyEnvironmentId() {
}
private String uniqify(String baseName, Set existing) {
-String name = baseName;
+String name = namespace + baseName;
Review comment:
Unless it is hideously expensive, it might be smart to choose a canonical
meta-format now (that there are three components) and escape the components.
For example slash-separated with slashes internal to the name escaped. This is
mostly a reaction to more strings being concatenated in a way that is starting
to look like it should be real structure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 188534)
Time Spent: 3h 20m (was: 3h 10m)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=188532&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188532
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 22/Jan/19 23:05
Start Date: 22/Jan/19 23:05
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #7316:
[BEAM-6269] Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r249996455
##
File path: model/job-management/src/main/proto/beam_expansion_api.proto
##
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Protocol Buffers describing the Expansion API, an api for expanding
+ * transforms in a remote SDK.
+ */
+
+syntax = "proto3";
+
+package org.apache.beam.model.expansion.v1;
+
+option go_package = "construction_v1";
+option java_package = "org.apache.beam.model.expansion.v1";
+option java_outer_classname = "ExpansionApi";
+
+import "beam_runner_api.proto";
+
+message ExpansionRequest {
+ // Set of components needed to interpret the transform, or which
+ // may be useful for its expansion. This includes the input
+ // PCollections (if any) to the to-be-expanded transform, along
+ // with their coders and windowing strategies.
+ org.apache.beam.model.pipeline.v1.Components components = 1;
+
+ // The actual PTransform to be expaneded according to its spec.
+ // Its input should be set, but its subtransforms and outputs
+ // should not be.
+ org.apache.beam.model.pipeline.v1.PTransform transform = 2;
+
+ // A namespace (prefix) to use for the id of any newly created
+ // components.
+ string namespace = 3;
Review comment:
Worth noting that this is not necessary. The whole idea of components is
that they never have meaning outside the message they are contained in, and are
just a way of encoding a graph. It is fair to critique that the copying and
re-allocating ids has computational cost. The last time we relied on ids having
significance outside a single message we hit bugs and migration pain. Those are
expected consequences IMO.
Here, it seems that it is not so bad. IIUC you are just using this namespace
as a way to cheaply reuse components, but not establishing any cross-message
dependence upon them.
But it is worth noting that you can do a role-reversal and save yourself
this field: the expansion service can freely prefix all the _other_ ids with
something to avoid collisions with its chosen namespace.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 188532)
Time Spent: 3h (was: 2h 50m)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=188533&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-188533
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 22/Jan/19 23:05
Start Date: 22/Jan/19 23:05
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #7316:
[BEAM-6269] Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r249994859
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
##
@@ -81,9 +101,13 @@ public static SdkComponents create(PipelineOptions
options) {
return sdkComponents;
}
- private SdkComponents() {}
+ private SdkComponents(String namespace) {
+this.namespace = namespace;
Review comment:
Document the namespace idea?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 188533)
Time Spent: 3h 10m (was: 3h)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=182735&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-182735 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 08/Jan/19 22:49 Start Date: 08/Jan/19 22:49 Worklog Time Spent: 10m Work Description: ihji commented on issue #7316: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7316#issuecomment-452480204 + @ihji adding myself for tracking the progress of this PR :smile: This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 182735) Time Spent: 2h 50m (was: 2h 40m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=179118&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179118
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 27/Dec/18 15:34
Start Date: 27/Dec/18 15:34
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #7316: [BEAM-6269]
Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r244174608
##
File path: model/job-management/src/main/proto/beam_construction_api.proto
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Protocol Buffers describing the Job API, api for communicating with a runner
+ * for job submission over GRPC.
+ */
+
+syntax = "proto3";
+
+package org.apache.beam.model.construction.v1;
+
+option go_package = "construction_v1";
+option java_package = "org.apache.beam.model.construction.v1";
+option java_outer_classname = "ConstructionApi";
+
+import "beam_runner_api.proto";
+
+message ConstructionRequest {
+ org.apache.beam.model.pipeline.v1.Components components = 1;
+ org.apache.beam.model.pipeline.v1.PTransform transform = 2;
+ string namespace = 3;
+}
Review comment:
The caller chooses a service that it thinks can handle the expansion. The
services, of course, can refuse.
The dependencies in the returned graph are part of the specified
environments. (I thought about artifact provisioning a bit, but didn't do
anything with that in this PR.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 179118)
Time Spent: 2h (was: 1h 50m)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=179115&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179115
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 27/Dec/18 15:28
Start Date: 27/Dec/18 15:28
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #7316: [BEAM-6269]
Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r244173610
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ConstructionService.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import com.google.auto.service.AutoService;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.model.construction.v1.ConstructionApi;
+import org.apache.beam.model.construction.v1.ConstructionServiceGrpc;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A service that allows pipeline construction from a remote SDK. */
+public class ConstructionService extends
ConstructionServiceGrpc.ConstructionServiceImplBase {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ConstructionService.class);
+
+ /**
+ * A registrar that creates {@link TransformProvider} instances from {@link
+ * RunnerApi.FunctionSpec}s.
+ *
+ * Transform authors have the ability to provide a registrar by creating
a {@link
+ * ServiceLoader} entry and a concrete implementation of this interface.
+ *
+ * It is optional but recommended to use one of the many build time tools
such as {@link
+ * AutoService} to generate the necessary META-INF files automatically.
Review comment:
I copied this text from the FileSystems provider. I think it's clear enough
from context, but if not could update it in both places.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 179115)
Time Spent: 1.5h (was: 1h 20m)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=179123&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179123 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 27/Dec/18 15:38 Start Date: 27/Dec/18 15:38 Worklog Time Spent: 10m Work Description: robertwb commented on issue #7316: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7316#issuecomment-450174161 Thanks for taking a look. Yes, I'll be taking this to the list. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 179123) Time Spent: 2h 40m (was: 2.5h) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=179122&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179122 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 27/Dec/18 15:38 Start Date: 27/Dec/18 15:38 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #7316: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7316#discussion_r244175201 ## File path: sdks/python/apache_beam/runners/portability/construction_service.py ## @@ -0,0 +1,116 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. Review comment: Exactly. Makes for easier, self-contained testing (and we'll want to be able to go both ways). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 179122) Time Spent: 2.5h (was: 2h 20m) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=179121&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179121
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 27/Dec/18 15:37
Start Date: 27/Dec/18 15:37
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #7316: [BEAM-6269]
Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r244175128
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ConstructionService.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import com.google.auto.service.AutoService;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.model.construction.v1.ConstructionApi;
+import org.apache.beam.model.construction.v1.ConstructionServiceGrpc;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A service that allows pipeline construction from a remote SDK. */
+public class ConstructionService extends
ConstructionServiceGrpc.ConstructionServiceImplBase {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ConstructionService.class);
+
+ /**
+ * A registrar that creates {@link TransformProvider} instances from {@link
+ * RunnerApi.FunctionSpec}s.
+ *
+ * Transform authors have the ability to provide a registrar by creating
a {@link
+ * ServiceLoader} entry and a concrete implementation of this interface.
+ *
+ * It is optional but recommended to use one of the many build time tools
such as {@link
+ * AutoService} to generate the necessary META-INF files automatically.
+ */
+ public interface ConstructionServiceRegistrar {
+Map knownTransforms();
+ }
+
+ /**
+ * Provides a mapping of {@link RunnerApi.FunctionSpec} to a {@link
PTransform}, together with
+ * mappings of its inputs and outputs to maps of PCollections.
+ *
+ * @param input {@link PValue} type of the transform
+ * @param output {@link PValue} type of the transform
+ */
+ public interface TransformProvider {
+
+default InputT createInput(Pipeline p, Map> inputs)
{
+ if (inputs.size() == 0) {
+return (InputT) p.begin();
+ }
+ if (inputs.size() == 1) {
+return (InputT) Iterables.getOnlyElement(inputs.values());
+ } else {
+PCollectionTuple inputTuple = PCollectionTuple.empty(p);
+for (Map.Entry> entry : inputs.entrySet()) {
+ inputTuple = inputTuple.and(new TupleTag(entry.getKey()),
entry.getValue());
+}
+return (InputT) inputTuple;
+ }
+}
+
+PTransform getTransform(RunnerApi.FunctionSpec spec);
+
+default Map> extractOutputs(OutputT output) {
+ if (output instanceof PDone) {
+return Collections.emptyMap();
+ } else if (output instanceof PCollection) {
+return ImmutableMap.of("output", (PCollection) output);
+ } else if (output instanceof PCollectionTuple) {
+return ((PCollectionTuple) output)
+.getAll()
+.ent
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=179120&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179120
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 27/Dec/18 15:36
Start Date: 27/Dec/18 15:36
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #7316: [BEAM-6269]
Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r244174836
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ConstructionService.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import com.google.auto.service.AutoService;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.model.construction.v1.ConstructionApi;
+import org.apache.beam.model.construction.v1.ConstructionServiceGrpc;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A service that allows pipeline construction from a remote SDK. */
+public class ConstructionService extends
ConstructionServiceGrpc.ConstructionServiceImplBase {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ConstructionService.class);
+
+ /**
+ * A registrar that creates {@link TransformProvider} instances from {@link
+ * RunnerApi.FunctionSpec}s.
+ *
+ * Transform authors have the ability to provide a registrar by creating
a {@link
+ * ServiceLoader} entry and a concrete implementation of this interface.
+ *
+ * It is optional but recommended to use one of the many build time tools
such as {@link
+ * AutoService} to generate the necessary META-INF files automatically.
+ */
+ public interface ConstructionServiceRegistrar {
+Map knownTransforms();
+ }
+
+ /**
+ * Provides a mapping of {@link RunnerApi.FunctionSpec} to a {@link
PTransform}, together with
+ * mappings of its inputs and outputs to maps of PCollections.
+ *
+ * @param input {@link PValue} type of the transform
+ * @param output {@link PValue} type of the transform
+ */
+ public interface TransformProvider {
+
+default InputT createInput(Pipeline p, Map> inputs)
{
+ if (inputs.size() == 0) {
+return (InputT) p.begin();
+ }
+ if (inputs.size() == 1) {
+return (InputT) Iterables.getOnlyElement(inputs.values());
+ } else {
+PCollectionTuple inputTuple = PCollectionTuple.empty(p);
+for (Map.Entry> entry : inputs.entrySet()) {
+ inputTuple = inputTuple.and(new TupleTag(entry.getKey()),
entry.getValue());
+}
+return (InputT) inputTuple;
+ }
+}
+
+PTransform getTransform(RunnerApi.FunctionSpec spec);
+
+default Map> extractOutputs(OutputT output) {
+ if (output instanceof PDone) {
+return Collections.emptyMap();
+ } else if (output instanceof PCollection) {
+return ImmutableMap.of("output", (PCollection) output);
+ } else if (output instanceof PCollectionTuple) {
+return ((PCollectionTuple) output)
+.getAll()
+.ent
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=179117&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179117
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 27/Dec/18 15:31
Start Date: 27/Dec/18 15:31
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #7316: [BEAM-6269]
Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r244174150
##
File path: model/job-management/src/main/proto/beam_construction_api.proto
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Protocol Buffers describing the Job API, api for communicating with a runner
+ * for job submission over GRPC.
+ */
+
+syntax = "proto3";
+
+package org.apache.beam.model.construction.v1;
+
+option go_package = "construction_v1";
+option java_package = "org.apache.beam.model.construction.v1";
+option java_outer_classname = "ConstructionApi";
+
+import "beam_runner_api.proto";
+
+message ConstructionRequest {
+ org.apache.beam.model.pipeline.v1.Components components = 1;
+ org.apache.beam.model.pipeline.v1.PTransform transform = 2;
Review comment:
These environments refer to runtime environments, which the expander may
choose. Possibly it makes sense to pass along some hints/suggestions, though
how best to do this in general is unclear. (The choice of expansion service is
the choice of environment where the expansion should occur. Such a service
could internally be multiplexing, of course.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 179117)
Time Spent: 1h 50m (was: 1h 40m)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=179114&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179114
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 27/Dec/18 15:28
Start Date: 27/Dec/18 15:28
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #7316: [BEAM-6269]
Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r244173559
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ConstructionService.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import com.google.auto.service.AutoService;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.model.construction.v1.ConstructionApi;
+import org.apache.beam.model.construction.v1.ConstructionServiceGrpc;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A service that allows pipeline construction from a remote SDK. */
+public class ConstructionService extends
ConstructionServiceGrpc.ConstructionServiceImplBase {
Review comment:
Yes, that's a better name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 179114)
Time Spent: 1h 20m (was: 1h 10m)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=179116&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179116
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 27/Dec/18 15:28
Start Date: 27/Dec/18 15:28
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #7316: [BEAM-6269]
Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r244173661
##
File path: model/job-management/src/main/proto/beam_construction_api.proto
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Protocol Buffers describing the Job API, api for communicating with a runner
+ * for job submission over GRPC.
+ */
+
+syntax = "proto3";
+
+package org.apache.beam.model.construction.v1;
+
+option go_package = "construction_v1";
+option java_package = "org.apache.beam.model.construction.v1";
+option java_outer_classname = "ConstructionApi";
+
+import "beam_runner_api.proto";
+
+message ConstructionRequest {
+ org.apache.beam.model.pipeline.v1.Components components = 1;
+ org.apache.beam.model.pipeline.v1.PTransform transform = 2;
+ string namespace = 3;
+}
+
+message ConstructionResponse {
+ org.apache.beam.model.pipeline.v1.Components components = 1;
+ string transform_id = 2;
Review comment:
Yep. I'll document these.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 179116)
Time Spent: 1h 40m (was: 1.5h)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=178646&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178646
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 26/Dec/18 09:40
Start Date: 26/Dec/18 09:40
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #7316:
[BEAM-6269] Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r243969399
##
File path: model/job-management/src/main/proto/beam_construction_api.proto
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Protocol Buffers describing the Job API, api for communicating with a runner
+ * for job submission over GRPC.
+ */
+
+syntax = "proto3";
+
+package org.apache.beam.model.construction.v1;
+
+option go_package = "construction_v1";
+option java_package = "org.apache.beam.model.construction.v1";
+option java_outer_classname = "ConstructionApi";
+
+import "beam_runner_api.proto";
+
+message ConstructionRequest {
+ org.apache.beam.model.pipeline.v1.Components components = 1;
+ org.apache.beam.model.pipeline.v1.PTransform transform = 2;
+ string namespace = 3;
+}
Review comment:
How can the caller make sure that dependencies needed for expansion are in
place (may be this is a part of environment definition) ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 178646)
Time Spent: 1h 10m (was: 1h)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=178645&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178645
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 26/Dec/18 09:40
Start Date: 26/Dec/18 09:40
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #7316:
[BEAM-6269] Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r243719464
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ConstructionService.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import com.google.auto.service.AutoService;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.model.construction.v1.ConstructionApi;
+import org.apache.beam.model.construction.v1.ConstructionServiceGrpc;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A service that allows pipeline construction from a remote SDK. */
+public class ConstructionService extends
ConstructionServiceGrpc.ConstructionServiceImplBase {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ConstructionService.class);
+
+ /**
+ * A registrar that creates {@link TransformProvider} instances from {@link
+ * RunnerApi.FunctionSpec}s.
+ *
+ * Transform authors have the ability to provide a registrar by creating
a {@link
+ * ServiceLoader} entry and a concrete implementation of this interface.
+ *
+ * It is optional but recommended to use one of the many build time tools
such as {@link
+ * AutoService} to generate the necessary META-INF files automatically.
Review comment:
META-INF files for jar files ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 178645)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=178641&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178641
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 26/Dec/18 09:40
Start Date: 26/Dec/18 09:40
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #7316:
[BEAM-6269] Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r243969695
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ConstructionService.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import com.google.auto.service.AutoService;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.model.construction.v1.ConstructionApi;
+import org.apache.beam.model.construction.v1.ConstructionServiceGrpc;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A service that allows pipeline construction from a remote SDK. */
+public class ConstructionService extends
ConstructionServiceGrpc.ConstructionServiceImplBase {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ConstructionService.class);
+
+ /**
+ * A registrar that creates {@link TransformProvider} instances from {@link
+ * RunnerApi.FunctionSpec}s.
+ *
+ * Transform authors have the ability to provide a registrar by creating
a {@link
+ * ServiceLoader} entry and a concrete implementation of this interface.
+ *
+ * It is optional but recommended to use one of the many build time tools
such as {@link
+ * AutoService} to generate the necessary META-INF files automatically.
+ */
+ public interface ConstructionServiceRegistrar {
+Map knownTransforms();
+ }
+
+ /**
+ * Provides a mapping of {@link RunnerApi.FunctionSpec} to a {@link
PTransform}, together with
+ * mappings of its inputs and outputs to maps of PCollections.
+ *
+ * @param input {@link PValue} type of the transform
+ * @param output {@link PValue} type of the transform
+ */
+ public interface TransformProvider {
+
+default InputT createInput(Pipeline p, Map> inputs)
{
+ if (inputs.size() == 0) {
+return (InputT) p.begin();
+ }
+ if (inputs.size() == 1) {
+return (InputT) Iterables.getOnlyElement(inputs.values());
+ } else {
+PCollectionTuple inputTuple = PCollectionTuple.empty(p);
+for (Map.Entry> entry : inputs.entrySet()) {
+ inputTuple = inputTuple.and(new TupleTag(entry.getKey()),
entry.getValue());
+}
+return (InputT) inputTuple;
+ }
+}
+
+PTransform getTransform(RunnerApi.FunctionSpec spec);
+
+default Map> extractOutputs(OutputT output) {
+ if (output instanceof PDone) {
+return Collections.emptyMap();
+ } else if (output instanceof PCollection) {
+return ImmutableMap.of("output", (PCollection) output);
+ } else if (output instanceof PCollectionTuple) {
+return ((PCollectionTuple) output)
+.getAll()
+.
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=178642&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178642
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 26/Dec/18 09:40
Start Date: 26/Dec/18 09:40
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #7316:
[BEAM-6269] Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r243969875
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ConstructionService.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import com.google.auto.service.AutoService;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.model.construction.v1.ConstructionApi;
+import org.apache.beam.model.construction.v1.ConstructionServiceGrpc;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A service that allows pipeline construction from a remote SDK. */
+public class ConstructionService extends
ConstructionServiceGrpc.ConstructionServiceImplBase {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(ConstructionService.class);
+
+ /**
+ * A registrar that creates {@link TransformProvider} instances from {@link
+ * RunnerApi.FunctionSpec}s.
+ *
+ * Transform authors have the ability to provide a registrar by creating
a {@link
+ * ServiceLoader} entry and a concrete implementation of this interface.
+ *
+ * It is optional but recommended to use one of the many build time tools
such as {@link
+ * AutoService} to generate the necessary META-INF files automatically.
+ */
+ public interface ConstructionServiceRegistrar {
+Map knownTransforms();
+ }
+
+ /**
+ * Provides a mapping of {@link RunnerApi.FunctionSpec} to a {@link
PTransform}, together with
+ * mappings of its inputs and outputs to maps of PCollections.
+ *
+ * @param input {@link PValue} type of the transform
+ * @param output {@link PValue} type of the transform
+ */
+ public interface TransformProvider {
+
+default InputT createInput(Pipeline p, Map> inputs)
{
+ if (inputs.size() == 0) {
+return (InputT) p.begin();
+ }
+ if (inputs.size() == 1) {
+return (InputT) Iterables.getOnlyElement(inputs.values());
+ } else {
+PCollectionTuple inputTuple = PCollectionTuple.empty(p);
+for (Map.Entry> entry : inputs.entrySet()) {
+ inputTuple = inputTuple.and(new TupleTag(entry.getKey()),
entry.getValue());
+}
+return (InputT) inputTuple;
+ }
+}
+
+PTransform getTransform(RunnerApi.FunctionSpec spec);
+
+default Map> extractOutputs(OutputT output) {
+ if (output instanceof PDone) {
+return Collections.emptyMap();
+ } else if (output instanceof PCollection) {
+return ImmutableMap.of("output", (PCollection) output);
+ } else if (output instanceof PCollectionTuple) {
+return ((PCollectionTuple) output)
+.getAll()
+.
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=178647&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178647 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 26/Dec/18 09:40 Start Date: 26/Dec/18 09:40 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #7316: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7316#discussion_r243969975 ## File path: sdks/python/apache_beam/runners/portability/construction_service.py ## @@ -0,0 +1,116 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. Review comment: IIUC This is same as above but implemented for Python. Is this correct ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 178647) Time Spent: 1h 10m (was: 1h) > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Robert Bradshaw >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=178640&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178640
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 26/Dec/18 09:40
Start Date: 26/Dec/18 09:40
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #7316:
[BEAM-6269] Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r243719360
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ConstructionService.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import com.google.auto.service.AutoService;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.model.construction.v1.ConstructionApi;
+import org.apache.beam.model.construction.v1.ConstructionServiceGrpc;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.grpc.v1_13_1.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** A service that allows pipeline construction from a remote SDK. */
+public class ConstructionService extends
ConstructionServiceGrpc.ConstructionServiceImplBase {
Review comment:
Probably "transform expansion" is a better than "pipeline construction". If
I understood correctly, SDK where the pipeline is constructed will invoke this
service to expand certain composite transforms that can only be expanded in a
second (target) environment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 178640)
Time Spent: 20m (was: 10m)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=178644&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178644
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 26/Dec/18 09:40
Start Date: 26/Dec/18 09:40
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #7316:
[BEAM-6269] Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r243968871
##
File path: model/job-management/src/main/proto/beam_construction_api.proto
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Protocol Buffers describing the Job API, api for communicating with a runner
+ * for job submission over GRPC.
+ */
+
+syntax = "proto3";
+
+package org.apache.beam.model.construction.v1;
+
+option go_package = "construction_v1";
+option java_package = "org.apache.beam.model.construction.v1";
+option java_outer_classname = "ConstructionApi";
+
+import "beam_runner_api.proto";
+
+message ConstructionRequest {
+ org.apache.beam.model.pipeline.v1.Components components = 1;
+ org.apache.beam.model.pipeline.v1.PTransform transform = 2;
+ string namespace = 3;
+}
+
+message ConstructionResponse {
+ org.apache.beam.model.pipeline.v1.Components components = 1;
+ string transform_id = 2;
Review comment:
This ID is for transform that serves as the root of the expanded tree ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 178644)
Time Spent: 1h (was: 50m)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[
https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=178643&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178643
]
ASF GitHub Bot logged work on BEAM-6269:
Author: ASF GitHub Bot
Created on: 26/Dec/18 09:40
Start Date: 26/Dec/18 09:40
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #7316:
[BEAM-6269] Cross-SDK transform expansion protocol.
URL: https://github.com/apache/beam/pull/7316#discussion_r243969276
##
File path: model/job-management/src/main/proto/beam_construction_api.proto
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Protocol Buffers describing the Job API, api for communicating with a runner
+ * for job submission over GRPC.
+ */
+
+syntax = "proto3";
+
+package org.apache.beam.model.construction.v1;
+
+option go_package = "construction_v1";
+option java_package = "org.apache.beam.model.construction.v1";
+option java_outer_classname = "ConstructionApi";
+
+import "beam_runner_api.proto";
+
+message ConstructionRequest {
+ org.apache.beam.model.pipeline.v1.Components components = 1;
+ org.apache.beam.model.pipeline.v1.PTransform transform = 2;
Review comment:
Should environment where expansion should happen [1] be part of the
expansion request ?
[1]
https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L952
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 178643)
Time Spent: 50m (was: 40m)
> Support mixed-language pipelines.
> -
>
> Key: BEAM-6269
> URL: https://issues.apache.org/jira/browse/BEAM-6269
> Project: Beam
> Issue Type: New Feature
> Components: beam-model, runner-core, sdk-java-core, sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Kenneth Knowles
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6269) Support mixed-language pipelines.
[ https://issues.apache.org/jira/browse/BEAM-6269?focusedWorklogId=177006&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177006 ] ASF GitHub Bot logged work on BEAM-6269: Author: ASF GitHub Bot Created on: 19/Dec/18 14:12 Start Date: 19/Dec/18 14:12 Worklog Time Spent: 10m Work Description: robertwb opened a new pull request #7316: [BEAM-6269] Cross-SDK transform expansion protocol. URL: https://github.com/apache/beam/pull/7316 See https://s.apache.org/beam-mixed-language-pipelines for context. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177006) Time Spent: 10m Remaining Estimate: 0h > Support mixed-language pipelines. > - > > Key: BEAM-6269 > URL: https://issues.apache.org/jira/browse/BEAM-6269 > Project: Beam > Issue Type: New Feature >
