Hi,
I'm new to flink and am evaluating it to replace our existing streaming
application.
The use case I'm working on is reading messages from RabbitMQ queue,
applying some transformation and filtering logic and sending it via HTTP to
a 3rd party.
A must have requirement of this flow is to to write
So you have rabitmq source and http sink?
If so you can use side output in order to dump your data to db.
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/side_output.html
On Sat, Feb 29, 2020, 23:01 Gadi Katsovich wrote:
> Hi,
> I'm new to flink and am evaluating it to replace
If I understand correctly he wants the HTTP result in the DB. So I do not
think side output works here. The DB would have to be the sink. Also sinks
in Flink are the final destination.
So it would have to be RabbitMQ -> Some Cool Business Logic Operators
Here > Async I/O HTTP Operator
Guys, thanks for the great advice. It works!
I used HttpAsyncClient from Apache Commons.
At first I tried to implement the async http client by implementing
AsyncFunction. I implemented the asyncInvoke method and used
try-with-resouce to instantiate the client (because it's
CloseableHttpAsyncClient
We have the same setup and it works quite well. One thing to take into
account is that your HTTP call may happen multiple times if you’re using
checkpointing/ fault tolerance mechanism, so it’s important that those
calls are idempotent and won’t duplicate data.
Also we’ve found that it’s important
