I missed adding the mailing list in my previous email.
-- Forwarded message -
From: Matthias Pohl
Date: Tue, Oct 27, 2020 at 12:39 PM
Subject: Re: Flink memory usage monitoring
To: Rajesh Payyappilly Jose
Hi Rajesh,
thanks for reaching out to us. We worked on providing metrics
Classification: Internal
Hi,
Environment - Flink 1.11 on K8s
Is there a way to monitor the usage of managed memory, off-heap memory and
network memory?
-Rajesh
::DISCLAIMER::
The contents of this e-mail and any attachment(s) are confidential and intended
for
I’ve used the following simple script to capture Flink metrics by running:
python -u ./statsd_server.py 9020 > statsd_server.log
>>> flink-conf.yaml
metrics.reporters: statsd_reporter
metrics.reporter.statsd_reporter.class:
org.apache.flink.metrics.statsd.StatsDReporter
Can you use wget (curl will work as well)? You can find the taskmanagers
with wget -O - http://localhost:8081/taskmanagers
and wget -O - http://localhost:8081/taskmanagers/request> to see detailed jvm
memory stats. localhost:8081 is in my example the jobmanager.
On 04.11.2017 16:19, AndreaKinn
Anyway, If I understood how system metrics works (the results seems to be
showed in browser) I can't use it because my cluster is accessible only with
terminal via ssh
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
I have used sysstat linux tool.
On the node the only one application running is Flink. The outcomes measured
with metric system could be different?
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Hi,
How did you measure the memory usage ?
JVM processes tend to occupy the maximum memory allocated to them,
regardless of whether those memory are actively in used or not. To
correctly measure the memory usage, you should use Flink's metric system[1]
Regards,
Kien
[1]
Hi,
I would like to share some considerations about Flink memory consumption.
I have a cluster composed of three nodes: 1 used both as JM and TM and other
2 TM.
I ran two identical applications (in different moments) on it. The only
difference is that on the second one I doubled every operators,
AM
> *To:* Stefano Bortoli
> *Cc:* Newport, Billy [Tech]; Fabian Hueske; user@flink.apache.org
>
> *Subject:* Re: Flink memory usage
>
>
>
> Hi Billy,
>
>
>
> if you didn't split the different data sets up into different slot sharing
> groups, then your maximu
We’re running this config now which is not really justifiable for what we’re
doing.
20 nodes 2 slots, 40 parallelism 36GB mem = 720GB of heap…
Thanks
From: Fabian Hueske [mailto:fhue...@gmail.com]
Sent: Wednesday, April 19, 2017 10:52 AM
To: Newport, Billy [Tech]
Cc: user@flink.apache.org
Subje
Hi Billy,
Flink's internal operators are implemented to not allocate heap space
proportional to the size of the input data.
Whenever Flink needs to hold data in memory (e.g., for sorting or building
a hash table) the data is serialized into managed memory. If all memory is
in use, Flink starts
11 matches
Mail list logo
