I’ve used the following simple script to capture Flink metrics by running:
python -u ./statsd_server.py 9020 > statsd_server.log
>>> flink-conf.yaml
metrics.reporters: statsd_reporter
metrics.reporter.statsd_reporter.class:
org.apache.flink.metrics.statsd.StatsDReporter
metrics.reporter.
Can you use wget (curl will work as well)? You can find the taskmanagers
with wget -O - http://localhost:8081/taskmanagers
and wget -O - http://localhost:8081/taskmanagers/request> to see detailed jvm
memory stats. localhost:8081 is in my example the jobmanager.
On 04.11.2017 16:19, AndreaKinn
Anyway, If I understood how system metrics works (the results seems to be
showed in browser) I can't use it because my cluster is accessible only with
terminal via ssh
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
I have used sysstat linux tool.
On the node the only one application running is Flink. The outcomes measured
with metric system could be different?
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Hi,
How did you measure the memory usage ?
JVM processes tend to occupy the maximum memory allocated to them,
regardless of whether those memory are actively in used or not. To
correctly measure the memory usage, you should use Flink's metric system[1]
Regards,
Kien
[1]
https://ci.apache.
ano Bortoli
> *Cc:* Newport, Billy [Tech]; Fabian Hueske; user@flink.apache.org
>
> *Subject:* Re: Flink memory usage
>
>
>
> Hi Billy,
>
>
>
> if you didn't split the different data sets up into different slot sharing
> groups, then your maximum parallelism is
ilto:trohrm...@apache.org]
> Sent: Thursday, April 20, 2017 11:52 AM
> To: Stefano Bortoli
> Cc: Newport, Billy [Tech]; Fabian Hueske; user@flink.apache.org
>
>
> Subject: Re: Flink memory usage
>
>
>
> Hi Billy,
>
>
>
> if you didn't split the different
AM
To: Stefano Bortoli
Cc: Newport, Billy [Tech]; Fabian Hueske; user@flink.apache.org
Subject: Re: Flink memory usage
Hi Billy,
if you didn't split the different data sets up into different slot sharing
groups, then your maximum parallelism is 40. Thus, it should be enough to
assign 40^2
y [mailto:billy.newp...@gs.com]
> *Sent:* Thursday, April 20, 2017 4:46 PM
> *To:* Stefano Bortoli ; 'Fabian Hueske' <
> fhue...@gmail.com>
>
> *Cc:* 'user@flink.apache.org'
> *Subject:* RE: Flink memory usage
>
>
>
> Your reuse idea kind of impli
, Billy [mailto:billy.newp...@gs.com]
Sent: Thursday, April 20, 2017 4:46 PM
To: Stefano Bortoli ; 'Fabian Hueske'
Cc: 'user@flink.apache.org'
Subject: RE: Flink memory usage
Your reuse idea kind of implies that it’s a GC generation rate issue, i.e. it’s
not collecting fast en
, Billy [Tech]; 'Fabian Hueske'
Cc: 'user@flink.apache.org'
Subject: RE: Flink memory usage
Hi Billy,
The only suggestion I can give is to check very well in your code for useless
variable allocations, and foster reuse as much as possible. Don’t create a new
collection at an
link.apache.org'
Subject: RE: Flink memory usage
I don’t think our function are memory heavy they typically are cogroups and
merge the records on the left with the records on the right.
We’re currently requiring 720GB of heap to do our processing which frankly
appears ridiculous to us. Could
We’re running this config now which is not really justifiable for what we’re
doing.
20 nodes 2 slots, 40 parallelism 36GB mem = 720GB of heap…
Thanks
From: Fabian Hueske [mailto:fhue...@gmail.com]
Sent: Wednesday, April 19, 2017 10:52 AM
To: Newport, Billy [Tech]
Cc: user@flink.apache.org
Subje
Hi Billy,
Flink's internal operators are implemented to not allocate heap space
proportional to the size of the input data.
Whenever Flink needs to hold data in memory (e.g., for sorting or building
a hash table) the data is serialized into managed memory. If all memory is
in use, Flink starts spi
14 matches
Mail list logo
