lostluck commented on a change in pull request #16671:
URL: https://github.com/apache/beam/pull/16671#discussion_r796167396
##########
File path: sdks/go/pkg/beam/transforms/sql/sql.go
##########
@@ -79,6 +82,33 @@ func ExpansionAddr(addr string) Option {
}
}
+const serviceGradleTarget =
":sdks:java:extensions:sql:expansion-service:shadowJar"
+
+// StartAutomatedExpansionService pulls a JAR from Maven containing
+// a Beam expansion service that handles the SQL transform URN,
+// creates a runner type to manage the service, starts the service,
+// and arranges the endpoint with the appropriate namespace tag.
+//
+// Returns the StopService() function for the runner (should be
+// deferred so the service is stopped before the program exits,)
+// the tagged expansion address, and an error.
+func StartAutomatedExpansionService() (func() error, string, error) {
Review comment:
I think my question is why is this all done at the wrapper package
level, rather than hidden away deeper? As implemented, every Jar based Xlang
wrapper has to duplicate all this code and the code at the transform call
itself, which is not a great experience, especially since there's nothing
specific about it.
Ideally, wrapper authors should only be writing
defaultExpansion := xlangx.AutoNamespace + xlangx.Separator + <gradle target>
Or similar (or a wrapper function like Require does).
Then they pass that in, and the auto startup code will take care of the
rest. Kinda like how the Python versions just take in the gradle target, and
then the code takes care of the rest. In this case, everything should be
happening within the scope of the auto HandlerFunc.
We do not need to optimize for repeated calls to the same expansion service.
Users with complex enough pipelines hitting the same service should likely be
standing up their own anyway.
If we don't want to use up "auto" just for jar startup we can always have
something else to specify what kind of auto thing it's starting up: eg.
"auto:jar:" to have it select sub name space for handling, or "autojar:" and
just lean on the existing selector.
##########
File path: sdks/go/pkg/beam/core/runtime/xlangx/expand.go
##########
@@ -140,3 +140,45 @@ func QueryExpansionService(ctx context.Context, p
*HandlerParams) (*jobpb.Expans
}
return res, nil
}
+
+// QueryAutomatedExpansionService submits an external transform to be expanded
by the
+// expansion service and then eagerly materializes the artifacts for staging.
The given
+// transform should be the external transform, and the componenets are any
additional
Review comment:
```suggestion
// transform should be the external transform, and the components are any
additional
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
