Re: How can I get the backpressure signals inside my function or operator?

d/vertices/:vertexid/backpressure". But I am not sure >>> whether it is easy to get the jobid and vertexid. >>> >>> [1] >>> https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/rest_api.html >>> >>> Best, >>>

Re: How can I get the backpressure signals inside my function or operator?

t; [1] >> https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/rest_api.html >> >> Best, >> Zhijiang >> >> ---------------------- >> From:Felipe Gutierrez >> Send Time:2019 Nov. 7 (Thu.) 00:06 >> To:Ches

Re: How can I get the backpressure signals inside my function or operator?

hepler > Cc:Zhijiang ; user > Subject:Re: How can I get the backpressure signals inside my function or > operator? > > If I can trigger the sample via rest API it is good for a POC. Then I can > read from any in-memory storage using a separated thread within the > operator. But wha

Re: How can I get the backpressure signals inside my function or operator?

monitoring/rest_api.html Best, Zhijiang -- From:Felipe Gutierrez Send Time:2019 Nov. 7 (Thu.) 00:06 To:Chesnay Schepler Cc:Zhijiang ; user Subject:Re: How can I get the backpressure signals inside my function or operator? If I can t

Re: How can I get the backpressure signals inside my function or operator?

If I can trigger the sample via rest API it is good for a POC. Then I can read from any in-memory storage using a separated thread within the operator. But what is the rest api that gives to me the ratio value from backpressure? Thanks *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--

Re: How can I get the backpressure signals inside my function or operator?

I don't think there is a truly sane way to do this. I could envision a separate application triggering samples via the REST API, writing the results into kafka which your operator can read. This is probably the most reasonable solution I can come up with. Any attempt at accessing the TaskExec

Re: How can I get the backpressure signals inside my function or operator?

Does anyone know in which metric I can rely on to know if a given operator is activating the backpressure? Or how can I call the same java object that the Flink UI calls to give me the ratio of backpressure? Thanks, Felipe *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://f

Re: How can I get the backpressure signals inside my function or operator?

Hi Zhijiang, thanks for your reply. Yes, you understood correctly. The fact that I cannot get "Shuffle.Netty.Input.Buffers.inputQueueLength" on the operator might be because of the way Flink runtime architecture was designed. But I was wondering what kind of signal I can get. I guess some backpres

Re: How can I get the backpressure signals inside my function or operator?

Hi Felipe, That is an interesting idea to control the upstream's output based on downstream's input. If I understood correctly, the preAggregate operator would trigger flush output while the reduce operator is idle/hungry. In contrast, the preAggregate would continue aggregating data in the c

How can I get the backpressure signals inside my function or operator?

Hi all, let's say that I have a "source -> map .> preAggregrate -> keyBy -> reduce -> sink" job and the reducer is sending backpressure signals to the preAggregate, map and source operator. How do I get those signals inside my operator's implementation? I guess inside the function is not possible.