damccorm commented on issue #36214: URL: https://github.com/apache/beam/issues/36214#issuecomment-3321292562
> Not sure how other runners handle stage boundaries - do they just pipe together data streams directly between workers? Generally this reduces to a reshuffle or similar which eventually uses GBK under the hood. But definitely not part of the Beam API. > If we had an API that specifies how secrets are determined as a PipelineOption, then only the only-GBK-persisting runners could replace GBKs with GBEKs and other runners could use the secret in all places where they persist pcollections. Yeah, this is more or less what I'm suggesting here; this issue is more of an implementation detail (though potentially you could only encrypt some of your pipeline at the GBK step if you want to use it directly I guess). Note that even with this transform and the GBK guarantees, it is still impossible for users to get encryption at rest guarantees since they would need to replace GBK in reshuffles (and other transforms) somehow, and they won't be able to do that. So a pipeline option will be necessary regardless. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
