>> Could we add a system topic that has exactly one partition per broker?
> I think this depends on which type of topic we use for the events.
> If it is
> nonpersistent, I think this approach would work because the events
wouldn't
> outlive the broker.
So, what happens to events in a nonpersiste
Thanks for the clarification, Joe. I now see the nuance between the data
and admin paths. One way to possibly remove these updates from the data
flow is to make it a process that watches the metastore and sends metastore
changes to a topic. That would remove it completely from the data path.
Howeve
> Could we add a system topic that has exactly one partition per broker?
Unfortunately, that just creates multiple single points of failure because
each partition has data that only exists on that partition. So, if any
partition fails, there's a gap, resulting in data loss.
--
Devin G. Bost
On Fr
Could we add a system topic that has exactly one partition per broker?
On Thu, Apr 22, 2021 at 11:22 PM Joe Francis
wrote:
> To be clear, I would love to have this feature. But I would not use this
> feature if that means whenever a broker that hosts a "system topic" has a
> hiccup, it would r
>> > > Based on looking through the interceptor trait, I don't see a way to
>> > > trigger events based on auto created/deleted topics. For example,
>> when a
>> > > producer connects to a broker for a nonexistent topic (assuming auto
>> > topic
>> > > creation is allowed), a managed ledger, and th
To be clear, I would love to have this feature. But I would not use this
feature if that means whenever a broker that hosts a "system topic" has a
hiccup, it would result in an outage for N other brokers. I run 100+
brokers/million+ topics in a cluster (hence an "audit topic" would be
wonderful
Completely disagree that we have accepted this risk with PIP-39. That is
different because it is an admin flow. A failure in a namespace policy
change does not affect data flow.
What you are proposing is in the data path. Topics and subs are
created in the data flow path. Failure means outages.
Hi Joe,
I agree there is a risk in adding more interdependencies between brokers. I
will point out that we have already accepted this risk with the
implementation of PIP 39, which propagates namespace policy changes to
other brokers using messages sent to a system topic. However, that doesn't
nece
I would be very careful about implementing such a feature, because of
introducing undesirable interdependencies. Broker processes only talk to
the metadata store or data store. This keeps brokers isolated from each
other - one broker is not dependent on the functioning of another broker.
A broke
Thanks for your response, PengHui.
I think this feature would be useful to end users for cluster management,
which is why I want to contribute a first class feature instead of writing
my own plugin that would add little value to the community.
> With the broker interceptor you can intercept all t
Hi Michael,
Currently, Pulsar supports a pluginable Broker Interceptor, you can find it
here
https://github.com/apache/pulsar/blob/6704f12104219611164aa2bb5bbdfc929613f1bf/pulsar-broker/src/main/java/org/apache/pulsar/broker/intercept/BrokerInterceptor.java
With the broker interceptor you can i
Hello all,
I would like to propose adding a new feature to Pulsar that will require a
PIP. In addition to feedback on the proposed feature, I am looking for
guidance on how to go about creating the PIP. Thanks for any help you can
provide.
I would like to add an optional system topic where topic
12 matches
Mail list logo
