Hi Flavio,
We have implemented our own flink operator, the operator will start a flink
job cluster (the application jar is already packaged together with flink in
the docker image). I believe Google's flink operator will start a session
cluster, and user can submit the flink job via REST. Not
Hi Eleanore,
That does't sound like a scaling issue. It's probably a data skew, that the
data volume on some of the keys are significantly higher than others. I'm
not familiar with this area though, and have copied Jark for you, who is
one of the community experts in this area.
Thank you~
_Hi Xintong,
Thanks for the prompt reply! To answer your question:
- Which Flink version are you using?
v1.8.2
- Is this skew observed only after a scaling-up? What happens if the
parallelism is initially set to the scaled-up value?
I also tried this, it
Hi Experts,
I have my flink application running on Kubernetes, initially with 1 Job
Manager, and 2 Task Managers.
Then we have the custom operator that watches for the CRD, when the CRD
replicas changed, it will patch the Flink Job Manager deployment
parallelism and max parallelism according to
