Mathieu DESPRIEE created FLINK-32242: ----------------------------------------
Summary: Datadog HTTP Reporter produces a huge outgoing traffic and CPU overhead Key: FLINK-32242 URL: https://issues.apache.org/jira/browse/FLINK-32242 Project: Flink Issue Type: Bug Components: Runtime / Metrics Affects Versions: 1.15.2 Environment: Flink 1.15.2, AWS EMR. Reporter: Mathieu DESPRIEE Attachments: image-2023-06-01-17-42-56-305.png, image-2023-06-01-17-54-45-900.png, image-2023-06-01-17-56-50-809.png We're running a relatively small flink cluster (7 task-managers, 8 cores) and are using datadog for telemetry. The numbers for outgoing traffic, between kafka producers, tasks activities, and host system metrics didn't add-up. After investigation, we discovered that this traffic was generated by the DatadogHttpReporter. We switched the reporter to an implementation using the java dogstatsd client (reporting to a datadog agent on each host). Here are some numbers of outgoing traffic taken at a NAT gateway, between the cluster and the outside world. Before/after this change (all other things being equal): !image-2023-06-01-17-56-50-809.png! We're talking about 850MG in 5mn, so 10GB/h overhead here. That kind of traffic is not free on AWS... Here the change on `flink.taskmanager.Status.JVM.CPU.Load` (over the whole cluster) !image-2023-06-01-17-54-45-900.png! Reporting telemetry in json over http has a *HUGE* overhead. So I would strongly advocate to deprecate this reporter, and recommend users to use a dogstatsd-based implementation. There exist one ([https://github.com/aroch/flink-metrics-dogstatsd,] not tested). On our side, we developed our own that we can share if requested. -- This message was sent by Atlassian Jira (v8.20.10#820010)