Hi Luke! I agree `sdk_worker_parallelism`don't change after job submission. However, users can change the configuration from m -> n over a period of time. Having this information as a metric helps in observing the behavior/impact of the job with the config change.
[1] https://github.com/apache/beam/blob/master/website/www/site/content/en/documentation/runtime/sdk-harness-config.md#sdk-harness-configuration On Wed, Aug 10, 2022 at 1:05 PM Luke Cwik <[email protected]> wrote: > Flink has a set of workers, each worker has a number of task slots. A > pipeline will use the number of slots based upon what it was configured to > run with. > > Are you trying to get the total number of workers, total number of tasks > slots, number of task slots your pipeline is using or number of workers > your pipeline is executing on? > > I was under the impression that the first two were properties of the Flink > cluster and don't change while the third property is configured at job > submission time and also doesn't change. > > I may not be understanding what you're trying to measure and why at > pipeline runtime for Flink since many of these values don't change through > the lifetime of the cluster and/or job. > > On Mon, Aug 8, 2022 at 4:59 PM aryan m <[email protected]> wrote: > >> Hi Luke! >> Thanks !! We use the Flink Runner and run SDK workers as processes >> [1] within a k8s pod. Can you please share broad steps on how one can do in >> the runner ? >> >> >> [1] >> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java >> >> >> On Mon, Aug 8, 2022 at 8:51 AM Luke Cwik via user <[email protected]> >> wrote: >> >>> That code only executes within a runner and is only used by certain >>> runners and wouldn't work in general from user code that is monitoring the >>> job or user code executing within one of the workers. >>> >>> You would need to author code that is likely runner specific to look up >>> the number of workers associated with a job as I don't believe there is a >>> general way to do this for an arbitrary Apache Beam runner. >>> >>> Which runner would you most likely want to use? >>> >>> On Sun, Aug 7, 2022 at 1:02 PM aryan m <[email protected]> wrote: >>> >>>> Hi Users! >>>> Is there a recommended approach to publish metrics on the number of >>>> sdk workers available/running as a gauge ? >>>> >>>> >>>> [1] >>>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L267 >>>> [2] >>>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L148 >>>> >>>> >>>> -- Aryan >>>> >>>
