Hi
I have to design a spark streaming application with below use case. I am
looking for best possible approach for this.
I have application which pushing data into 1000+ different topics each has
different purpose . Spark streaming will receive data from each topic and
after processing it will wr
I do not fully understand the design here.
Why not send all to one topic with some application id in the message and you
write to one topic also indicating the application id.
Can you elaborate a little bit more on the use case?
Especially applications deleting/creating topics dynamically can b
Agree with Jörn. Dynamically creating/deleting Topics is nontrivial to
manage.
With the limited knowledge about your scenario - it appears that you are
using topics as some kind of message type enum.
If that is the case - you might be better off with one (or just a few
topics) and have a messagetyp
I agree Jorn and Satish. I think I should starting grouping similar kind of
messages into single topic with some kind of id attached to it which can be
pulled from spark streaming application.
I can try reducing no of topic to significant lower but still at the end I
can expect 50+ topics in clust
