Thanks for sharing your solution Wouter! Best, Fabian
Am Mi., 15. Mai 2019 um 15:28 Uhr schrieb Wouter Zorgdrager < w.d.zorgdra...@tudelft.nl>: > Hi all, > > To answer my own questions I worked on the following solution: > > 1) Custom Docker image which pulls the Flink image and moves Prometheus > jar to the correct folder [1, 2]. > 2) I wrote manifests for Kubernetes with service discovery configuration > for Kubernetes [3]. Besides the 'official' Flink Kubernetes manifests, I > added a TM service which exposes all TM metrics locations so that > Prometheus can scrape it. This means that (re)-scaling Flink TM's are > automatically picked up by Prometheus. The repository also includes a > Grafana setup with a simple dashboard. > > I thought this might be useful for other users! > > Cheers, > Wouter > > [1]: > https://github.com/wzorgdrager/flink-k8s/blob/master/docker/Dockerfile > [2]: https://hub.docker.com/r/wzorgdrager/flink-prometheus > [3]: https://github.com/wzorgdrager/flink-k8s > > Op ma 13 mei 2019 om 14:16 schreef Wouter Zorgdrager < > w.d.zorgdra...@tudelft.nl>: > >> Hey all, >> >> I'm working on a deployment setup with Flink and Prometheus on >> Kubernetes. I'm running into the following issues: >> >> 1) Is it possible to use the default Flink Docker image [1] and enable >> the Prometheus reporter? Modifying the flink-config.yaml is easy, but >> somehow the Prometheus reporter jar needs to be moved within the image. >> This is easy if use my own Dockerfile (as done here [2]) , but I prefer >> using the official one. >> 2) I can define the jobmanager/taskmanager metric endpoints statically, >> but w.r.t. scaling I prefer to have these resolved/discovered dynamically. >> Did anyone get a working setup on this? I came across this resource for >> YARN [3], is there something similar for Kubernetes? Or are there any other >> ways of configuring Prometheus to pick this up automatically? >> >> Thanks a lot for your help! >> >> Kind regards, >> Wouter >> >> [1]: https://hub.docker.com/_/flink/ >> [2]: >> https://github.com/mbode/flink-prometheus-example/blob/master/Dockerfile >> [3]: https://github.com/eastcirclek/flink-service-discovery >> >
