On 18/10/2019 07:47, Dweep Sharma wrote:
I found that readers are not actually displayed as Topics under the
Pulsar admin dashboard. Does that mean Pulsar does not store any meta
information for readers?
A topic is a stream of messages.
A consumer is someone who reads the messages (*).
A subscription keeps track of the position in the stream for a
particular consumer - which messages that consumer has seen, and which
ones they haven't.
So what you're looking for in the admin dashboard is "subscriptions".
If you want to reliably deliver the same set of messages to multiple
consumers (pub/sub style), then each consumer needs its own subscription
to the topic (*)
Subscriptions do store state, because they remember the difference
between acknowledged and unacknowledged messages. Subscriptions can be
removed when you no longer need them.
Subscriptions can be managed via the CLI:
http://pulsar.apache.org/docs/en/pulsar-admin/#subscriptions
For this use case,
The Topic can be visualised as some sort of orderid generated from
backend, so it will be unique for each user/session
A topic is a collection of messages. Topics are persistent, they don't
change between users/sessions. You can of course delete a topic if you
don't want it any more, but they are shared between publishers and
consumers, they are not unique to users.
The messages within a topic are deleted once all the subscriptions have
acknowledged them - unless you configure a retention policy to make them
stay around longer.
the browser websocket client would need to listen on a new topic with
every session, consume messages from broker and unsubscribe/close
immediately. Will that leave a lot of meta data at pulsar broker,
assuming websocket API does not have a unsubscribe/delete topic
functionality?
No - I think you have got topics and subscriptions mixed up.
When the client connects, it would use its own subscription to catch up
with any unread messages in the topic it hasn't seen before.
(*) There is another way you can handle this. Instead of "consumers"
you can have "readers". "readers" store no state on the server - a
reader is responsible for maintaining its own position in the event stream.
If you don't care about past events, then when you connect as a "reader"
you can tell it to start from the end of the topic. It won't see any
old messages, and will only see new messages going forward from this
point in time. This is the closest to the redis pub/sub model. It also
allows scaling to millions of subscribers, without the server having to
keep track of millions of subscriptions.
So if you want something that works like redis pub/sub, this may be what
you are looking for.
HTH,
Brian.
