[
https://issues.apache.org/jira/browse/BEAM-8962?focusedWorklogId=360298&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360298
]
ASF GitHub Bot logged work on BEAM-8962:
----------------------------------------
Author: ASF GitHub Bot
Created on: 16/Dec/19 14:31
Start Date: 16/Dec/19 14:31
Worklog Time Spent: 10m
Work Description: tweise commented on pull request #10381: [BEAM-8962]
Add option to disable the metric container accumulator
URL: https://github.com/apache/beam/pull/10381#discussion_r358263908
##########
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
##########
@@ -159,6 +159,15 @@
void setEnableMetrics(Boolean enableMetrics);
+ @Description(
+ "By default, uses Flink accumulators to store the metrics which allows
to query metrics from the PipelineResult. "
+ + "If set to true, metrics will still be reported but can't be
queried via PipelineResult. "
+ + "This saves network and memory.")
+ @Default.Boolean(false)
Review comment:
I find it important that flags are named to express their semantics. In this
case, the flag's purpose is to turn something off, and there are many similar
examples where we would use "skip", "no" etc.
There should not be a case where `disabled=false` needs to be specified by
the user.
But maybe flags should not have default values, since they are fully
described by name.
That would also avoid the options to be included with their defaults during
job submission, even when the user has not specified them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 360298)
Time Spent: 1h 10m (was: 1h)
> FlinkMetricContainer causes churn in the JobManager and lets the web frontend
> malfunction
> -----------------------------------------------------------------------------------------
>
> Key: BEAM-8962
> URL: https://issues.apache.org/jira/browse/BEAM-8962
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The {{FlinkMetricContainer}} wraps the Beam metric container for reporting
> metrics, but also stores them as Flink accumulators. With high parallelism
> jobs with over a thousand tasks and many built-in Beam metrics for every Beam
> step, this can accumulate to over 100MB of serialized data which is stored in
> the JobManager's ExecutionGraph. This then fails to even sent over the wire,
> due to the akka.framesize limit (10MB by default), and manifests in {{500
> Internal Server Error}}s in the web frontend.
> We need to introduce an option to disable the reporting via accumulators. It
> is mostly useful for batch workloads where you can retrieve the final
> accumulator values at the end of the job. It adds a lot of memory and network
> overhead.
> Perhaps we could even turn off the accumulators for streaming jobs, or
> entirely and make them opt-in.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
