Re: SDK Worker availability metrics
Hi Luke! I agree `sdk_worker_parallelism`don't change after job submission. However, users can change the configuration from m -> n over a period of time. Having this information as a metric helps in observing the behavior/impact of the job with the config change. [1] https://github.com/apache/beam/blob/master/website/www/site/content/en/documentation/runtime/sdk-harness-config.md#sdk-harness-configuration On Wed, Aug 10, 2022 at 1:05 PM Luke Cwik wrote: > Flink has a set of workers, each worker has a number of task slots. A > pipeline will use the number of slots based upon what it was configured to > run with. > > Are you trying to get the total number of workers, total number of tasks > slots, number of task slots your pipeline is using or number of workers > your pipeline is executing on? > > I was under the impression that the first two were properties of the Flink > cluster and don't change while the third property is configured at job > submission time and also doesn't change. > > I may not be understanding what you're trying to measure and why at > pipeline runtime for Flink since many of these values don't change through > the lifetime of the cluster and/or job. > > On Mon, Aug 8, 2022 at 4:59 PM aryan m wrote: > >> Hi Luke! >> Thanks !! We use the Flink Runner and run SDK workers as processes >> [1] within a k8s pod. Can you please share broad steps on how one can do in >> the runner ? >> >> >> [1] >> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java >> >> >> On Mon, Aug 8, 2022 at 8:51 AM Luke Cwik via user >> wrote: >> >>> That code only executes within a runner and is only used by certain >>> runners and wouldn't work in general from user code that is monitoring the >>> job or user code executing within one of the workers. >>> >>> You would need to author code that is likely runner specific to look up >>> the number of workers associated with a job as I don't believe there is a >>> general way to do this for an arbitrary Apache Beam runner. >>> >>> Which runner would you most likely want to use? >>> >>> On Sun, Aug 7, 2022 at 1:02 PM aryan m wrote: >>> Hi Users! Is there a recommended approach to publish metrics on the number of sdk workers available/running as a gauge ? [1] https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L267 [2] https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L148 -- Aryan >>>
Re: SDK Worker availability metrics
Flink has a set of workers, each worker has a number of task slots. A pipeline will use the number of slots based upon what it was configured to run with. Are you trying to get the total number of workers, total number of tasks slots, number of task slots your pipeline is using or number of workers your pipeline is executing on? I was under the impression that the first two were properties of the Flink cluster and don't change while the third property is configured at job submission time and also doesn't change. I may not be understanding what you're trying to measure and why at pipeline runtime for Flink since many of these values don't change through the lifetime of the cluster and/or job. On Mon, Aug 8, 2022 at 4:59 PM aryan m wrote: > Hi Luke! > Thanks !! We use the Flink Runner and run SDK workers as processes [1] > within a k8s pod. Can you please share broad steps on how one can do in the > runner ? > > > [1] > https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java > > > On Mon, Aug 8, 2022 at 8:51 AM Luke Cwik via user > wrote: > >> That code only executes within a runner and is only used by certain >> runners and wouldn't work in general from user code that is monitoring the >> job or user code executing within one of the workers. >> >> You would need to author code that is likely runner specific to look up >> the number of workers associated with a job as I don't believe there is a >> general way to do this for an arbitrary Apache Beam runner. >> >> Which runner would you most likely want to use? >> >> On Sun, Aug 7, 2022 at 1:02 PM aryan m wrote: >> >>> Hi Users! >>> Is there a recommended approach to publish metrics on the number of >>> sdk workers available/running as a gauge ? >>> >>> >>> [1] >>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L267 >>> [2] >>> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L148 >>> >>> >>> -- Aryan >>> >>
Re: SDK Worker availability metrics
Hi Luke! Thanks !! We use the Flink Runner and run SDK workers as processes [1] within a k8s pod. Can you please share broad steps on how one can do in the runner ? [1] https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java On Mon, Aug 8, 2022 at 8:51 AM Luke Cwik via user wrote: > That code only executes within a runner and is only used by certain > runners and wouldn't work in general from user code that is monitoring the > job or user code executing within one of the workers. > > You would need to author code that is likely runner specific to look up > the number of workers associated with a job as I don't believe there is a > general way to do this for an arbitrary Apache Beam runner. > > Which runner would you most likely want to use? > > On Sun, Aug 7, 2022 at 1:02 PM aryan m wrote: > >> Hi Users! >> Is there a recommended approach to publish metrics on the number of >> sdk workers available/running as a gauge ? >> >> >> [1] >> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L267 >> [2] >> https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L148 >> >> >> -- Aryan >> >
Re: SDK Worker availability metrics
That code only executes within a runner and is only used by certain runners and wouldn't work in general from user code that is monitoring the job or user code executing within one of the workers. You would need to author code that is likely runner specific to look up the number of workers associated with a job as I don't believe there is a general way to do this for an arbitrary Apache Beam runner. Which runner would you most likely want to use? On Sun, Aug 7, 2022 at 1:02 PM aryan m wrote: > Hi Users! > Is there a recommended approach to publish metrics on the number of > sdk workers available/running as a gauge ? > > > [1] > https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L267 > [2] > https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L148 > > > -- Aryan >
SDK Worker availability metrics
Hi Users! Is there a recommended approach to publish metrics on the number of sdk workers available/running as a gauge ? [1] https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L267 [2] https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L148 -- Aryan
