On Wed, Apr 24, 2019 at 12:21 PM Maximilian Michels <m...@apache.org> wrote: > > Good idea to let the client expose an artifact staging service that the > ExpansionService could use to stage artifacts. This solves two problems: > > (1) The Expansion Service not being able to access the Job Server > artifact staging service > (2) The client not having access to the dependencies returned by the > Expansion Server > > The downside is that it adds an additional indirection. The alternative > to let the client handle staging the artifacts returned by the Expansion > Server is more transparent and easier to implement.
The other downside is that it may not always be possible for the expansion service to connect to the artifact staging service (e.g. when constructing a pipeline locally against a remote expansion service). > Ideally, the Expansion Service won't return any dependencies because the > environment already contains the required dependencies. We could make it > a requirement for the expansion to be performed inside an environment. > Then we would already ensure during expansion time that the runtime > dependencies are available. Yes, it's cleanest if the expansion service provides an environment without all the dependencies provided. Interesting idea to make this a property of the expansion service itself. > > In this case, the runner would (as > > requested by its configuration) be free to merge environments it > > deemed compatible, including swapping out beam-java-X for > > beam-java-embedded if it considers itself compatible with the > > dependency list. > > Could you explain how that would work in practice? Say one has a pipeline with environments A: beam-java-sdk-2.12-docker B: beam-java-sdk-2.12-docker + dep1 C: beam-java-sdk-2.12-docker + dep2 D: beam-java-sdk-2.12-docker + dep3 A runner could (conceivably) be intelligent enough to know that dep1 and dep2 are indeed compatible, and run A, B, and C in a single beam-java-sdk-2.12-docker + dep1 + dep2 environment (with the corresponding fusion and lower overhead benefits). If a certain pipeline option is set, it might further note that dep1 and dep2 are compatible with its own workers, which are build against sdk-2.12, and choose to run these in embedded + dep1 + dep2 environment.
