mxm commented on pull request #13116: URL: https://github.com/apache/beam/pull/13116#issuecomment-723559227
There is currently no universal way to set the configuration per-job. Unless a per-job cluster is used. In this case a configuration file can be supplied for the per-job cluster. There is no way to explicitly configure a state backend via a flag to the main program. Even if we allowed for a configuration file parameter, we would rely on that configuration file to be present during runtime which we can't assume. Further, if we allowed a direct configuration YAML string, we would also have to explicitly set the state backend on the `StreamExecutionEnvironment`, otherwise during runtime the cluster's default state backend would be used instead of the configured one (the default state backend can only be configured during job creation time). The intention here is to allow setting and overriding the default state backend. Thus, it makes sense (for now) to have an explicit way to configure the state backend. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
