[
https://issues.apache.org/jira/browse/BEAM-9577?focusedWorklogId=411503&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-411503
]
ASF GitHub Bot logged work on BEAM-9577:
----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Mar/20 23:35
Start Date: 27/Mar/20 23:35
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11203: [BEAM-9577]
Define and implement dependency-aware artifact staging service.
URL: https://github.com/apache/beam/pull/11203#discussion_r399581813
##########
File path: model/job-management/src/main/proto/beam_artifact_api.proto
##########
@@ -31,8 +31,77 @@ option java_outer_classname = "ArtifactApi";
import "beam_runner_api.proto";
-// A service to stage artifacts for use in a Job.
+// A service to retrieve artifacts for use in a Job.
+service ArtifactRetrievalService {
+ // Resolves the given artifact reference into one or more simpler artifact
+ // references (e.g. a Maven dependency into a (transitive) set of jars.
+ // If no further simplification is possible, returns the original artifacts,
+ // at which point, all artifacts must be gettable.
+ rpc ResolveArtifact(ResolveArtifactRequest) returns
(ResolveArtifactResponse);
+
+ // Retrieves the given artifact as a stream of bytes.
+ rpc GetArtifact(GetArtifactRequest) returns (stream GetArtifactResponse);
+}
+
+// A service that allows the client to act as an ArtifactRetrievalService,
+// for a particular job with the server initiating requests and receiving
+// responses.
+// A client calls the service with an ArtifactResponseWrapper that has the
+// staging token set, and thereafter responds to the server's requests.
service ArtifactStagingService {
+ rpc ReverseArtifactRetrievalService(stream ArtifactResponseWrapper)
Review comment:
Good question. As discussed offline, multiple requests can be pushed before
waiting for a response, so we should be able to fill the network buffer. Either
the server or client can choose to parallelize on its backend store as needed.
I added an error field to the resolve_artifacts field as a runner may
attempt to opportunistically consolidate environments, and then fall back to
doing each separately.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 411503)
Time Spent: 2h 50m (was: 2h 40m)
> Update artifact staging and retrieval protocols to be dependency aware.
> -----------------------------------------------------------------------
>
> Key: BEAM-9577
> URL: https://issues.apache.org/jira/browse/BEAM-9577
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
> Reporter: Robert Bradshaw
> Assignee: Robert Bradshaw
> Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
