We do not measure how much data we are spilling to disk.
On 09.12.2016 14:43, Fabian Hueske wrote:
Hi,
the heap mem usage should be available via Flink's metrics system.
Not sure if that also captures spilled data. Chesnay (in CC) should
know that.
If the spilled data is not available as a metric, you can try to write
a small script that monitors the directories to which Flink spills
(Config parameter: |taskmanager.tmp.dirs| [1]).
The script would repeatedly list all files and keep for each file the
max size (files are deleted once the are not used anymore). This is
not super precise but might be good enough.
Hope this helps,
Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.1/setup/config.html#jobmanager-amp-taskmanager
2016-12-09 14:12 GMT+01:00 otherwise777 <wou...@onzichtbaar.net
<mailto:wou...@onzichtbaar.net>>:
Currently i'm doing some analysis for some algorithms that i use
in Flink,
I'm interested in the Space and time it takes to execute them. For
the Time
i used getNetRuntime() in the executionenvironment, but I have no
idea how
to analyse the amount of space an algorithm uses.
Space can mean different things here, like Heap space, disk space,
overal
memory or allocated memory. I would like to analyze some of these.
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-analyze-space-usage-of-Flink-algorithms-tp10555.html
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-analyze-space-usage-of-Flink-algorithms-tp10555.html>
Sent from the Apache Flink User Mailing List archive. mailing list
archive at Nabble.com.
