[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[ https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=390355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390355 ] ASF GitHub Bot logged work on BEAM-9229: Author: ASF GitHub Bot Created on: 21/Feb/20 01:06 Start Date: 21/Feb/20 01:06 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #10733: [BEAM-9229] Adding dependency information to Environment proto URL: https://github.com/apache/beam/pull/10733 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390355) Time Spent: 7.5h (was: 7h 20m) > Adding dependency information to Environment proto > -- > > Key: BEAM-9229 > URL: https://issues.apache.org/jira/browse/BEAM-9229 > Project: Beam > Issue Type: Sub-task > Components: beam-model >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > > Adding dependency information to Environment proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[ https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=390268=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390268 ] ASF GitHub Bot logged work on BEAM-9229: Author: ASF GitHub Bot Created on: 20/Feb/20 21:32 Start Date: 20/Feb/20 21:32 Worklog Time Spent: 10m Work Description: ihji commented on issue #10733: [BEAM-9229] Adding dependency information to Environment proto URL: https://github.com/apache/beam/pull/10733#issuecomment-589328702 @robertwb comments addressed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 390268) Time Spent: 7h 20m (was: 7h 10m) > Adding dependency information to Environment proto > -- > > Key: BEAM-9229 > URL: https://issues.apache.org/jira/browse/BEAM-9229 > Project: Beam > Issue Type: Sub-task > Components: beam-model >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 7h 20m > Remaining Estimate: 0h > > Adding dependency information to Environment proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=389691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389691
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 20/Feb/20 00:01
Start Date: 20/Feb/20 00:01
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r381617505
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1158,6 +1158,90 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for locally-accessible artifact files.
+// payload: ArtifactFilePayload
+FILE = 0 [(beam_urn) = "beam:artifact:type:file:v1"];
+
+// A URN for artifacts described by URLs.
+// payload: ArtifactUrlPayload
+URL = 1 [(beam_urn) = "beam:artifact:type:url:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 2 [(beam_urn) = "beam:artifact:type:embedded:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:type:pypi:v1"];
+
+// A URN for Java artifacts hosted on a Maven repository.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:type:maven:v1"];
+ }
+ enum Roles {
+// A URN for staging-to role.
+// payload: ArtifactStagingToRolePayload
+STAGING_TO = 0 [(beam_urn) = "beam:artifact:role:staging_to:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // a string for an artifact path e.g. "/tmp/foo.jar"
+ string path = 1;
+
+ // The hex-encoded sha256 checksum of the artifact.
+ string sha256 = 2;
+}
+
+message ArtifactUrlPayload {
+ // a string for an artifact URL e.g. "https://.../foo.jar; or
"gs://tmp/foo.jar"
+ string path = 1;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389691)
Time Spent: 7h 10m (was: 7h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=389683=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389683
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 19/Feb/20 23:52
Start Date: 19/Feb/20 23:52
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r381614435
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1158,6 +1158,90 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for locally-accessible artifact files.
+// payload: ArtifactFilePayload
+FILE = 0 [(beam_urn) = "beam:artifact:type:file:v1"];
+
+// A URN for artifacts described by URLs.
+// payload: ArtifactUrlPayload
+URL = 1 [(beam_urn) = "beam:artifact:type:url:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 2 [(beam_urn) = "beam:artifact:type:embedded:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:type:pypi:v1"];
+
+// A URN for Java artifacts hosted on a Maven repository.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:type:maven:v1"];
+ }
+ enum Roles {
+// A URN for staging-to role.
+// payload: ArtifactStagingToRolePayload
+STAGING_TO = 0 [(beam_urn) = "beam:artifact:role:staging_to:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // a string for an artifact path e.g. "/tmp/foo.jar"
+ string path = 1;
+
+ // The hex-encoded sha256 checksum of the artifact.
+ string sha256 = 2;
+}
+
+message ArtifactUrlPayload {
+ // a string for an artifact URL e.g. "https://.../foo.jar; or
"gs://tmp/foo.jar"
+ string path = 1;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
Review comment:
We could consider just letting the payload be the raw bytes (and similarly
for other one-field payloads).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 389683)
Time Spent: 7h (was: 6h 50m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 7h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[ https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=389671=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-389671 ] ASF GitHub Bot logged work on BEAM-9229: Author: ASF GitHub Bot Created on: 19/Feb/20 22:55 Start Date: 19/Feb/20 22:55 Worklog Time Spent: 10m Work Description: ihji commented on issue #10733: [BEAM-9229] Adding dependency information to Environment proto URL: https://github.com/apache/beam/pull/10733#issuecomment-588516398 @robertwb @lukecwik PTAL. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 389671) Time Spent: 6h 50m (was: 6h 40m) > Adding dependency information to Environment proto > -- > > Key: BEAM-9229 > URL: https://issues.apache.org/jira/browse/BEAM-9229 > Project: Beam > Issue Type: Sub-task > Components: beam-model >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 6h 50m > Remaining Estimate: 0h > > Adding dependency information to Environment proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387758
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 15/Feb/20 00:00
Start Date: 15/Feb/20 00:00
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #10733:
[BEAM-9229] Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379695306
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,66 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts described by URLs.
+// payload: ArtifactUrlPayload
+URL = 0 [(beam_urn) = "beam:artifact:url:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 2 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on a Maven repository.
+// payload: MavenPayload
+MAVEN= 3 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactUrlPayload {
+ // a string for an artifact URL e.g. "https://.../foo.jar; or
"file:///tmp/foo.jar"
+ string path = 1;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A name of the embedded file.
+ string name = 2;
Review comment:
you mean "now redundant " ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387758)
Time Spent: 6h 40m (was: 6.5h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387692
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 21:30
Start Date: 14/Feb/20 21:30
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379652739
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,66 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts described by URLs.
+// payload: ArtifactUrlPayload
+URL = 0 [(beam_urn) = "beam:artifact:url:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 2 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on a Maven repository.
+// payload: MavenPayload
+MAVEN= 3 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactUrlPayload {
+ // a string for an artifact URL e.g. "https://.../foo.jar; or
"file:///tmp/foo.jar"
+ string path = 1;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A name of the embedded file.
+ string name = 2;
+}
+
+message PyPIPayload {
+ // Pypi compatible artifact id e.g. "apache-beam"
+ string artifact_id = 1;
+
+ // Pypi compatible version string.
+ string version = 2;
+}
+
+message MavenPayload {
+ // A string specifying Maven artifact.
+ // The standard format is
"groupId:artifactId:version[:packaging[:classifier]]"
+ string artifact_specifier = 1;
Review comment:
Yes.
See:
https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#MAVEN402
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387692)
Time Spent: 6.5h (was: 6h 20m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387691
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 21:29
Start Date: 14/Feb/20 21:29
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379652739
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,66 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts described by URLs.
+// payload: ArtifactUrlPayload
+URL = 0 [(beam_urn) = "beam:artifact:url:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 2 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on a Maven repository.
+// payload: MavenPayload
+MAVEN= 3 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactUrlPayload {
+ // a string for an artifact URL e.g. "https://.../foo.jar; or
"file:///tmp/foo.jar"
+ string path = 1;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A name of the embedded file.
+ string name = 2;
+}
+
+message PyPIPayload {
+ // Pypi compatible artifact id e.g. "apache-beam"
+ string artifact_id = 1;
+
+ // Pypi compatible version string.
+ string version = 2;
+}
+
+message MavenPayload {
+ // A string specifying Maven artifact.
+ // The standard format is
"groupId:artifactId:version[:packaging[:classifier]]"
+ string artifact_specifier = 1;
Review comment:
Yes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387691)
Time Spent: 6h 20m (was: 6h 10m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387658
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 20:33
Start Date: 14/Feb/20 20:33
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379629498
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
Review comment:
We don't have to do that in this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387658)
Time Spent: 6h 10m (was: 6h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387655=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387655
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 20:33
Start Date: 14/Feb/20 20:33
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379629237
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,66 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts described by URLs.
+// payload: ArtifactUrlPayload
+URL = 0 [(beam_urn) = "beam:artifact:url:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 2 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on a Maven repository.
+// payload: MavenPayload
+MAVEN= 3 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactUrlPayload {
+ // a string for an artifact URL e.g. "https://.../foo.jar; or
"file:///tmp/foo.jar"
+ string path = 1;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A name of the embedded file.
+ string name = 2;
Review comment:
This is not redundant with the top-level staged_name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387655)
Time Spent: 6h (was: 5h 50m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 6h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387656=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387656
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 20:33
Start Date: 14/Feb/20 20:33
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379626110
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,66 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts described by URLs.
+// payload: ArtifactUrlPayload
+URL = 0 [(beam_urn) = "beam:artifact:url:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 2 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on a Maven repository.
+// payload: MavenPayload
+MAVEN= 3 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactUrlPayload {
+ // a string for an artifact URL e.g. "https://.../foo.jar; or
"file:///tmp/foo.jar"
+ string path = 1;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A name of the embedded file.
+ string name = 2;
+}
+
+message PyPIPayload {
+ // Pypi compatible artifact id e.g. "apache-beam"
+ string artifact_id = 1;
+
+ // Pypi compatible version string.
+ string version = 2;
+}
+
+message MavenPayload {
+ // A string specifying Maven artifact.
+ // The standard format is
"groupId:artifactId:version[:packaging[:classifier]]"
+ string artifact_specifier = 1;
Review comment:
Is this powerful enough to designate version ranges?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387656)
Time Spent: 6h (was: 5h 50m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 6h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387657=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387657
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 20:33
Start Date: 14/Feb/20 20:33
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379628030
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
+ bytes payload = 2;
+ string artifact_id = 3;
+ string version_range = 4;
+}
Review comment:
Yes. The .tar.gz is one such example. Another example is that we treat the a
manually-specified dev version of apache_beam in a special way (currently it
*must* be named python_dataflow_sdk.tar); it would be better to not have such
magic names.
SDKs can choose to leave it blank, but I'd rather this be explicit than
implicit in the name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387657)
Time Spent: 6h 10m (was: 6h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387548
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 18:19
Start Date: 14/Feb/20 18:19
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379575969
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
+ bytes payload = 2;
+ string artifact_id = 3;
+ string version_range = 4;
+}
Review comment:
Do we really believe we will have a mix and match for "source" descriptions
and "destination" descriptions?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387548)
Time Spent: 5h 50m (was: 5h 40m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387546=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387546
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 18:14
Start Date: 14/Feb/20 18:14
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379573946
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,66 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts described by URLs.
+// payload: ArtifactUrlPayload
+URL = 0 [(beam_urn) = "beam:artifact:url:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 2 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on a Maven repository.
+// payload: MavenPayload
+MAVEN= 3 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactUrlPayload {
+ // a string for an artifact URL e.g. "https://.../foo.jar; or
"file:///tmp/foo.jar"
+ string path = 1;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A name of the embedded file.
+ string name = 2;
+}
+
+message PyPIPayload {
+ // Pypi compatible artifact id e.g. "apache-beam"
+ string artifact_id = 1;
+
+ // Pypi compatible version string.
+ string version = 2;
+}
+
+message MavenPayload {
+ // A string specifying Maven artifact.
+ // The standard format is
"groupId:artifactId:version[:packaging[:classifier]]"
+ string artifact_specifier = 1;
Review comment:
nit: `artifact_specifier` -> `artifact`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387546)
Time Spent: 5h 40m (was: 5.5h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387054=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387054
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 02:25
Start Date: 14/Feb/20 02:25
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379222644
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
Review comment:
`FILE` and `REMOTE` are merged and renamed to `URL`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387054)
Time Spent: 5.5h (was: 5h 20m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387051=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387051
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 02:23
Start Date: 14/Feb/20 02:23
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379222180
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
Review comment:
Ack.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387051)
Time Spent: 5h 20m (was: 5h 10m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387050=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387050
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 02:22
Start Date: 14/Feb/20 02:22
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379222112
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message PypiPayload {
+ // Pypi compatible artifact id e.g. "apache-beam"
+ string pypi_artifact_id = 1;
+
+ // Pypi compatible version string.
+ string pypi_version_range = 2;
+}
+
+message MavenPayload {
+ // Maven compatible group id e.g. "org.apache.beam"
+ string maven_group_id = 1;
Review comment:
Merged into a single field `artifact_specifier` which expects the format of
`groupId:artifactId:version[:packaging[:classifier]]`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387050)
Time Spent: 5h 10m (was: 5h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387049=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387049
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 02:21
Start Date: 14/Feb/20 02:21
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379221800
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387049)
Time Spent: 5h (was: 4h 50m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387048=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387048
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 02:21
Start Date: 14/Feb/20 02:21
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379221747
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message PypiPayload {
+ // Pypi compatible artifact id e.g. "apache-beam"
+ string pypi_artifact_id = 1;
+
+ // Pypi compatible version string.
+ string pypi_version_range = 2;
+}
+
+message MavenPayload {
+ // Maven compatible group id e.g. "org.apache.beam"
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387048)
Time Spent: 4h 50m (was: 4h 40m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387047=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387047
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 02:20
Start Date: 14/Feb/20 02:20
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379221603
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message PypiPayload {
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387047)
Time Spent: 4h 40m (was: 4.5h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387046=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387046
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 02:20
Start Date: 14/Feb/20 02:20
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379221573
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
Review comment:
Ack.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387046)
Time Spent: 4.5h (was: 4h 20m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387001=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387001
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 00:52
Start Date: 14/Feb/20 00:52
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379187541
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
Review comment:
I would call this one URL rather than REMOTE as most of these are remote.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387001)
Time Spent: 4h 20m (was: 4h 10m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386999
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 00:52
Start Date: 14/Feb/20 00:52
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379198236
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
+ bytes payload = 2;
+ string artifact_id = 3;
+ string version_range = 4;
+}
Review comment:
Role would be an opaque string. (Well, likely a URN, but the runner wouldn't
do anything about it.) It would be provided by, and consumed by, the SDK.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386999)
Time Spent: 4h 10m (was: 4h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=387000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387000
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 00:52
Start Date: 14/Feb/20 00:52
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379187475
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
Review comment:
Yes, I'm saying that we should have a new type here for that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 387000)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386977
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 14/Feb/20 00:02
Start Date: 14/Feb/20 00:02
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379186995
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message PypiPayload {
+ // Pypi compatible artifact id e.g. "apache-beam"
+ string pypi_artifact_id = 1;
+
+ // Pypi compatible version string.
+ string pypi_version_range = 2;
+}
+
+message MavenPayload {
+ // Maven compatible group id e.g. "org.apache.beam"
+ string maven_group_id = 1;
Review comment:
If there's a standard string format, let's just use that rather than
breaking this up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386977)
Time Spent: 4h (was: 3h 50m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386973
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 23:59
Start Date: 13/Feb/20 23:59
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379186220
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
Review comment:
+1 to allowing (relative) paths to some stagingDir root.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386973)
Time Spent: 3h 50m (was: 3h 40m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386971=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386971
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 23:58
Start Date: 13/Feb/20 23:58
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379185959
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
Review comment:
Regarding the file payload, I think we may want to stage this under a
different name than its "real" name. However, I think the stage name needs to
be lifted to a higher level and apply to all artifact types.
One reason to keep file separate than URL is that we may want to set the
space of files be those understood by (and fetched via) the beam filesystems
packages (including remote, non-(public?)-url ones like gcs or aws).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386971)
Time Spent: 3h 40m (was: 3.5h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386902=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386902
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 21:48
Start Date: 13/Feb/20 21:48
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379134250
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
Review comment:
Should this be an actual path where the user says where in the container
this file should live?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386902)
Time Spent: 3h 20m (was: 3h 10m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386896=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386896
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 21:48
Start Date: 13/Feb/20 21:48
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379126839
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message PypiPayload {
+ // Pypi compatible artifact id e.g. "apache-beam"
+ string pypi_artifact_id = 1;
+
+ // Pypi compatible version string.
+ string pypi_version_range = 2;
+}
+
+message MavenPayload {
+ // Maven compatible group id e.g. "org.apache.beam"
Review comment:
Please add a URL field since a person may want to specify a repository other
then maven central and state that if the field is unspecified then maven
central is assumed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386896)
Time Spent: 3h 10m (was: 3h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386899=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386899
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 21:48
Start Date: 13/Feb/20 21:48
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379136839
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
Review comment:
How about just `path` and we drop `staged_name`?
OR rename ArtifactFilePayload to ArtifactUrlPayload and then users can
reference local and remote artifacts?
Using a `url` field which would always have the scheme specified (e.g. local
file paths would always have `file://` prefix instead of the assumed conversion
from `/foo/bar` to `file:///foo/bar`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386899)
Time Spent: 3h 20m (was: 3h 10m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386904=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386904
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 21:48
Start Date: 13/Feb/20 21:48
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379137208
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
Review comment:
Not needed it we adopt ArtifactUrlPayload
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386904)
Time Spent: 3.5h (was: 3h 20m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386898=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386898
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 21:48
Start Date: 13/Feb/20 21:48
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379122261
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
Review comment:
```suggestion
// A URN for Java artifacts hosted on a Maven repository.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386898)
Time Spent: 3h 10m (was: 3h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386901=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386901
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 21:48
Start Date: 13/Feb/20 21:48
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379133774
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
Review comment:
`staged_name` -> `name`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386901)
Time Spent: 3h 20m (was: 3h 10m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386900=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386900
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 21:48
Start Date: 13/Feb/20 21:48
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379132449
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message PypiPayload {
+ // Pypi compatible artifact id e.g. "apache-beam"
+ string pypi_artifact_id = 1;
+
+ // Pypi compatible version string.
+ string pypi_version_range = 2;
+}
+
+message MavenPayload {
+ // Maven compatible group id e.g. "org.apache.beam"
+ string maven_group_id = 1;
Review comment:
Note that group/artifact/version isn't enough to fully specify all types of
maven artifacts.
Either use a single string to specify the artifact
`groupId:artifactId:version[:packaging[:classifier]]`
where packaging and classifier are optional and packaging by default is
"jar" and classifier is omitted if unspecified.
Or add `classifier` and `type` and rename `maven_group_id` to `group_id`,
`maven_artifact_id` to `artifact_id` and `maven_version_range` to `version`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386900)
Time Spent: 3h 20m (was: 3h 10m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386897=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386897
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 21:48
Start Date: 13/Feb/20 21:48
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379126380
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message PypiPayload {
+ // Pypi compatible artifact id e.g. "apache-beam"
+ string pypi_artifact_id = 1;
+
+ // Pypi compatible version string.
+ string pypi_version_range = 2;
+}
+
+message MavenPayload {
+ // Maven compatible group id e.g. "org.apache.beam"
+ string maven_group_id = 1;
+
+ // Maven compatible artifact id e.g. "beam-sdks-java-core"
+ string maven_artifact_id = 2;
+
+ // Maven compatible version string.
Review comment:
Grab the documentation for valid version strings/ranges from:
https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#MAVEN402
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386897)
Time Spent: 3h 10m (was: 3h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386903=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386903
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 21:48
Start Date: 13/Feb/20 21:48
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379133624
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1093,6 +1093,72 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: EmbeddedFilePayload.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+
+// A URN for artifacts described by remote URLs.
+// payload: a string for an artifact URL e.g. "https://.../foo.jar;
+REMOTE = 2 [(beam_urn) = "beam:artifact:remote:v1"];
+
+// A URN for Python artifacts hosted on PYPI.
+// payload: PypiPayload
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+
+// A URN for Java artifacts hosted on Maven central.
+// payload: MavenPayload
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message EmbeddedFilePayload {
+ // raw data bytes for an embedded artifact
+ bytes data = 1;
+
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message PypiPayload {
Review comment:
nit: `Pypi` -> `PyPI`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386903)
Time Spent: 3h 20m (was: 3h 10m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386850=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386850
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 20:03
Start Date: 13/Feb/20 20:03
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379089255
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
Review comment:
We could transfer big files via streaming service protocol. Embedded type
here is only for files small-enough in a proto size limit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386850)
Time Spent: 3h (was: 2h 50m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386846=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386846
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 19:58
Start Date: 13/Feb/20 19:58
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379087113
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
+ bytes payload = 2;
+ string artifact_id = 3;
+ string version_range = 4;
+}
Review comment:
I see your point however I wonder what other options we have. It looks hard
to describe all possible roles of a given file as a proto field. Maybe we could
just delegate how to interpret a given file name to environments? (for
instance, predefined file name for installable .tar.gz package for Python)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386846)
Time Spent: 2h 50m (was: 2h 40m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=386841=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386841
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 13/Feb/20 19:34
Start Date: 13/Feb/20 19:34
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r379074754
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
+ bytes payload = 2;
+ string artifact_id = 3;
Review comment:
Makes sense. Moved artifact_id and version_range to each payload.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 386841)
Time Spent: 2h 40m (was: 2.5h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=385416=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-385416
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 11/Feb/20 19:54
Start Date: 11/Feb/20 19:54
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r377860779
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
Review comment:
One issue here is that the callee may not know if it is on a shared
filesystem with the caller (e.g. when calling the expansion service). And when
calling two distinct expansion services, one would like to be able to compare
between them.
Also, perhaps we should not limit ourselves to local paths here, but any
path that can be opened with beam filesystems.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 385416)
Time Spent: 2.5h (was: 2h 20m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=385417=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-385417
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 11/Feb/20 19:54
Start Date: 11/Feb/20 19:54
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r377862882
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
+ bytes payload = 2;
+ string artifact_id = 3;
Review comment:
If we can't come up with a standard format, I think it should be part of the
payload. Similarly for artifact_id--they should mean the same thing. (We can
always safely dedup on urn+payloads.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 385417)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=385415=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-385415
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 11/Feb/20 19:54
Start Date: 11/Feb/20 19:54
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r377863536
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
+ bytes payload = 2;
+ string artifact_id = 3;
+ string version_range = 4;
+}
Review comment:
But we'll need it for more than just embedded payload. And the name itself
may not be enough to determine the role (do we try to install all .tar.gz files
in Python? Or are some just data?)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 385415)
Time Spent: 2.5h (was: 2h 20m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=385414=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-385414
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 11/Feb/20 19:54
Start Date: 11/Feb/20 19:54
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r377862155
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
Review comment:
But eventually we'll have to give it a name, right? (One could argue that
dependencies should be a map(name -> artifact). OTOH, for some perhaps we could
leave it blank and one could be inferred (e.g. for urls).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 385414)
Time Spent: 2.5h (was: 2h 20m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[ https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=385400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-385400 ] ASF GitHub Bot logged work on BEAM-9229: Author: ASF GitHub Bot Created on: 11/Feb/20 19:42 Start Date: 11/Feb/20 19:42 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10733: [BEAM-9229] Adding dependency information to Environment proto URL: https://github.com/apache/beam/pull/10733#issuecomment-584815364 The expansion service would necessarily need new APIs; worse case we could implement them as not-implemented stubs in the new language. (Agree about avoiding breaking changes for artifact staging/retrieval.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 385400) Time Spent: 2h 20m (was: 2h 10m) > Adding dependency information to Environment proto > -- > > Key: BEAM-9229 > URL: https://issues.apache.org/jira/browse/BEAM-9229 > Project: Beam > Issue Type: Sub-task > Components: beam-model >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Adding dependency information to Environment proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[ https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=384746=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-384746 ] ASF GitHub Bot logged work on BEAM-9229: Author: ASF GitHub Bot Created on: 10/Feb/20 21:19 Start Date: 10/Feb/20 21:19 Worklog Time Spent: 10m Work Description: ihji commented on issue #10733: [BEAM-9229] Adding dependency information to Environment proto URL: https://github.com/apache/beam/pull/10733#issuecomment-584359901 > Thanks. Could you also include the changes to the artifact staging and/or retrieval service? And the expansion service? I think it would be better to review them separately. The changes to the artifact staging and retrieval service proto require the not-too-small changes to the Java / Python / Go source codes to be compiled. We can submit this PR without breaking the build but not the PR with the service changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 384746) Time Spent: 2h 10m (was: 2h) > Adding dependency information to Environment proto > -- > > Key: BEAM-9229 > URL: https://issues.apache.org/jira/browse/BEAM-9229 > Project: Beam > Issue Type: Sub-task > Components: beam-model >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Adding dependency information to Environment proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=384743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-384743
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 10/Feb/20 21:13
Start Date: 10/Feb/20 21:13
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r377319480
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
+ bytes payload = 2;
+ string artifact_id = 3;
Review comment:
We also need artifact_id and version_range in local types (local files and
embedded files) for possible deduplication. If the field is common for all
types of artifacts I thought it should be the top-level attribute. And yes it'd
be nice to use a standard format for the top-level version_range arrtibute but
I don't have a good idea at this moment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 384743)
Time Spent: 2h (was: 1h 50m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=384741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-384741
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 10/Feb/20 21:04
Start Date: 10/Feb/20 21:04
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r377315362
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
+ bytes payload = 2;
+ string artifact_id = 3;
+ string version_range = 4;
+}
Review comment:
Thanks for a good catch. Added staged_name for embedded payload.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 384741)
Time Spent: 1h 50m (was: 1h 40m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=384738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-384738
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 10/Feb/20 21:03
Start Date: 10/Feb/20 21:03
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r377314762
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
Review comment:
Looks like `urn` and `payload` combination is mostly common in
`beam_runner_api.proto` since I couldn't find any other usage of `string type
=`. I think it would be better to follow the same convention here. WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 384738)
Time Spent: 1h 40m (was: 1.5h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=384726=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-384726
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 10/Feb/20 20:55
Start Date: 10/Feb/20 20:55
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r377310747
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
Review comment:
It's assumed to be used when SDK is saving remote artifacts and preparing to
submit them to the staging service from locally downloaded files. `local_path`
here is not ubiquitous information. After submitting the artifacts,
retrieval_token (and staged_name if necessary) should be used instead of
`local_path`. Is there any specific use-cases you're considering?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 384726)
Time Spent: 1.5h (was: 1h 20m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=384716=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-384716
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 10/Feb/20 20:39
Start Date: 10/Feb/20 20:39
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r377303210
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
Review comment:
stage_name is specific to files ready to be staged (local files and embedded
files).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 384716)
Time Spent: 1h 20m (was: 1h 10m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=384715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-384715
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 10/Feb/20 20:38
Start Date: 10/Feb/20 20:38
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r377302712
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
Review comment:
Make sense. Renamed to 'remote'.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 384715)
Time Spent: 1h 10m (was: 1h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=384714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-384714
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 10/Feb/20 20:37
Start Date: 10/Feb/20 20:37
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r377302377
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
Review comment:
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 384714)
Time Spent: 1h (was: 50m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=382615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382615
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 06/Feb/20 00:10
Start Date: 06/Feb/20 00:10
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r375574532
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
Review comment:
Is this specific to files?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382615)
Time Spent: 50m (was: 40m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=382610=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382610
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 06/Feb/20 00:10
Start Date: 06/Feb/20 00:10
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r375575923
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
Review comment:
s/urn/type?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382610)
Time Spent: 40m (was: 0.5h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=382613=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382613
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 06/Feb/20 00:10
Start Date: 06/Feb/20 00:10
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r375575037
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
Review comment:
It seems this only makes sense on a shared filesystem. (And could result in
confusion otherwise.) Perhaps we should also store a hash (e.g. sha256) so a
check of "sameness" can be performed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382613)
Time Spent: 40m (was: 0.5h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=382608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382608
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 06/Feb/20 00:10
Start Date: 06/Feb/20 00:10
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r375572386
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
Review comment:
Nit: separate these with a newline, so the definitions cling more tightly to
the constants.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382608)
Time Spent: 0.5h (was: 20m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=382611=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382611
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 06/Feb/20 00:10
Start Date: 06/Feb/20 00:10
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r375573106
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
+ bytes payload = 2;
+ string artifact_id = 3;
Review comment:
It seems that payload and artifact_id are redundant. Merge them? Should
version-range be part of the payload as well? (Or, if it's a top-level
attribute, should it be in a standard format (not sure if there is one, but if
there is it'd be nice to be able to do generic intersections and merging...)).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382611)
Time Spent: 40m (was: 0.5h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=382614=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382614
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 06/Feb/20 00:10
Start Date: 06/Feb/20 00:10
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r375572820
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
Review comment:
Or should this be URL (or maybe REMOTE) to allow for ftp, etc? It could be
argued we want to restrict to HTTP(S), though I don't know what would be gained
by that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382614)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=382612=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382612
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 06/Feb/20 00:10
Start Date: 06/Feb/20 00:10
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r375576529
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
+// A URN for artifacts described by HTTP links.
+// payload: a string for an artifact HTTP URL
+HTTP = 2 [(beam_urn) = "beam:artifact:http:v1"];
+// A URN for artifacts hosted on PYPI.
+// artifact_id: a PYPI project name
+// version_range: a PYPI compatible version string
+// payload: None
+PYPI = 3 [(beam_urn) = "beam:artifact:pypi:v1"];
+// A URN for artifacts hosted on Maven central.
+// artifact_id: [maven group id]:[maven artifact id]
+// version_range: a Maven compatible version string
+// payload: None
+MAVEN= 4 [(beam_urn) = "beam:artifact:maven:v1"];
+ }
+}
+
+message ArtifactFilePayload {
+ // A path to an artifact file on a local system.
+ string local_path = 1;
+ // A generated staged name (no path).
+ string staged_name = 2;
+}
+
+message ArtifactInformation {
+ string urn = 1;
+ bytes payload = 2;
+ string artifact_id = 3;
+ string version_range = 4;
+}
Review comment:
Seems we also need a staging name and/or role field--for embedded, there's
not even a file extension to go off of, and even if we have an extension it may
not be unambiguous, e.g. for a .tar.gz file, is it something that should be pip
installed, or something that should simply be placed in an accessible location
as data.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382612)
Time Spent: 40m (was: 0.5h)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[
https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=382609=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382609
]
ASF GitHub Bot logged work on BEAM-9229:
Author: ASF GitHub Bot
Created on: 06/Feb/20 00:10
Start Date: 06/Feb/20 00:10
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10733: [BEAM-9229]
Adding dependency information to Environment proto
URL: https://github.com/apache/beam/pull/10733#discussion_r375578623
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -1087,6 +1087,44 @@ message SideInput {
FunctionSpec window_mapping_fn = 3;
}
+message StandardArtifacts {
+ enum Types {
+// A URN for artifacts stored in a local directory.
+// payload: ArtifactFilePayload.
+FILE = 0 [(beam_urn) = "beam:artifact:file:v1"];
+// A URN for artifacts embedded in ArtifactInformation proto.
+// payload: raw data bytes.
+EMBEDDED = 1 [(beam_urn) = "beam:artifact:embedded:v1"];
Review comment:
I think we'll need the ability to serve up bytes that don't fit into a proto
(e.g. huge jar files). Perhaps this could be yet another type, and a streaming
protocol added to the extension service that allows one to retrieve the bytes
given a token + environment?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 382609)
Time Spent: 0.5h (was: 20m)
> Adding dependency information to Environment proto
> --
>
> Key: BEAM-9229
> URL: https://issues.apache.org/jira/browse/BEAM-9229
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Adding dependency information to Environment proto.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[ https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=379676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379676 ] ASF GitHub Bot logged work on BEAM-9229: Author: ASF GitHub Bot Created on: 31/Jan/20 00:02 Start Date: 31/Jan/20 00:02 Worklog Time Spent: 10m Work Description: ihji commented on issue #10733: [BEAM-9229] Adding dependency information to Environment proto URL: https://github.com/apache/beam/pull/10733#issuecomment-580519027 R: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 379676) Time Spent: 20m (was: 10m) > Adding dependency information to Environment proto > -- > > Key: BEAM-9229 > URL: https://issues.apache.org/jira/browse/BEAM-9229 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Adding dependency information to Environment proto. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9229) Adding dependency information to Environment proto
[ https://issues.apache.org/jira/browse/BEAM-9229?focusedWorklogId=379672=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379672 ] ASF GitHub Bot logged work on BEAM-9229: Author: ASF GitHub Bot Created on: 30/Jan/20 23:59 Start Date: 30/Jan/20 23:59 Worklog Time Spent: 10m Work Description: ihji commented on pull request #10733: [BEAM-9229] Adding dependency information to Environment proto URL: https://github.com/apache/beam/pull/10733 Proto extension for cross-language dependency management. #10621 depends on this PR. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[