This is a really important thread. I think our system topic design
needs more definition, structure, and documentation. Thanks for
starting the thread, Bo Cong.
> If the user uses it, it means that the user has considered the result of the
> clear system sub backlog
I think we need to discuss authorization for operating on system
topics, especially when topic level policies are not enabled.
>From reading the PIP [0], it looks like we already have many different
ways to designate a system topic (prefixes, suffixes, and total string
equality). I am concerned that this design will not scale and that it
will be very expensive on installations with many topics.
Is it possible for us to pivot our implementation so that system
topics and only system topics start with two underscores? This could
simplify system topic authorization. It would also improve the speed
of operations on namespaces because filtering out system topics would
only require a quick check for `startsWith("__")`.
While it's possible that users already have topics that start with
"__", I think it is unlikely, and I think it is essential that we
design system topics so that they are highly scalable.
Thanks,
Michael
[0] https://github.com/apache/pulsar/issues/13794
On Thu, Jan 13, 2022 at 10:27 AM PengHui Li <peng...@apache.org> wrote:
>
> I agree with the point, we should avoid the `clearNamespaceBacklog(String
> namespace)` to apply to the internal topic or internal cursor.
> It will introduce abnormal behaviors, e.g. clear a replication cursor
> backlog, clear a dedup cursor backlog, clear a compaction cursor backlog.
>
> I would suggest letting the `clearNamespaceBacklog(String namespace)`
> method does not apply to the Pulsar internal topics and cursors.
>
> Thanks,
> Penghui
>
> On Thu, Jan 13, 2022 at 4:32 PM 丛搏 <congbobo...@gmail.com> wrote:
>
> > Hi :
> >
> > issue link : https://github.com/streamnative/support-tickets/issues/408 <
> > https://github.com/streamnative/support-tickets/issues/408>
> >
> > # Background
> > Now we have many clear namespace backlog interfaces etc.
> > ```
> > clearNamespaceBacklog(String namespace)
> >
> > clearNamespaceBacklogAsync(String namespace)
> >
> > clearNamespaceBacklogForSubscription(String namespace, String subscription)
> >
> > clearNamespaceBacklogForSubscriptionAsync(String namespace, String
> > subscription)
> >
> > clearNamespaceBundleBacklog(String namespace, String bundle)
> >
> > clearNamespaceBundleBacklogAsync(String namespace, String bundle)
> >
> > clearNamespaceBundleBacklogForSubscription(String namespace, String
> > bundle, String subscription)
> >
> > clearNamespaceBundleBacklogForSubscriptionAsync(String namespace, String
> > bundle, String subscription)
> > ```
> > There are two types of interfaces:
> > 1. clear namespace backlog with subscriptionName
> > 2. clear namespace backlog no subscriptionName
> >
> > But there have a problem, users do not know that there are many system
> > topics in the namespace.
> >
> > ```
> >
> > /**
> > * Local topic name for the namespace events.
> > */
> > public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> > "__change_events";
> >
> > /**
> > * Local topic name for the transaction buffer snapshot.
> > */
> > public static final String TRANSACTION_BUFFER_SNAPSHOT =
> > "__transaction_buffer_snapshot";
> >
> > public static final String SUBSCRIPTION_NAME =
> > "transaction-buffer-sub";
> > ```
> > User only want to clear the backlog witch they own created and sub. But
> > now we have clear the system topic sub backlog when user don't specify
> > subName and clear system sub with a topic witch user created. It will bring
> > many problems. etc.
> > 1. clear `NAMESPACE_EVENTS_LOCAL_NAME` will clear any topic policy in this
> > namespace, but user don't know and they don't want to be cleared. If the
> > user does not understand the problems caused by this interface, it will
> > delete many topic policy configurations and is difficult to recover, which
> > will lead to unpredictable and disastrous problems.
> > 2. clear `TRANSACTION_BUFFER_SNAPSHOT` topic backlog will cause
> > transaction buffer recover incompletely.
> > 3. clear `transaction-buffer-sub` sub also will cause transaction buffer
> > recover incompletely.
> > 4. if add new system topic and system sub, we should think about this
> > problem.
> >
> > # Motivation
> > We want to solve the above problems, but we can't make users unable to
> > delete the backlog of system topic and system sub.
> > So, we should solve types of interface
> > There are two types of interfaces:
> > 1. clear namespace backlog with subscriptionName
> > 2. clear namespace backlog no subscriptionName
> >
> > when clear namespace backlog no subscriptionName:
> > It represents user know the sub name is system subName and really want to
> > delete it, We should not block the user clear system sub backlog and do not
> > allow the user to create system sub. If the user uses it, it means that the
> > user has considered the result of the clear system sub backlog
> >
> > when clear namespace backlog with subscriptionName:
> > Because the user may not consider the existence of system topic and system
> > sub, and the wrong clear backlog may lead to serious results. Therefore,
> > this interface cannot delete the backlog of system topic and the backlog of
> > system sub. If the user really wants to clear backlog, please bring
> > subscriptionName and use another type of interface with subscriptionName.
> >
> > **Clear namespace and clear namespace bundle all need to handle is
> > logical.**
> >
> > # implement
> >
> > ```
> > class SystemTopicName {
> > /**
> > * Local topic name for the namespace events.
> > */
> > public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> > "__change_events";
> >
> > /**
> > * Local topic name for the transaction buffer snapshot.
> > */
> > public static final String TRANSACTION_BUFFER_SNAPSHOT =
> > "__transaction_buffer_snapshot";
> > }
> > ```
> > ```
> > class SystemSubName {
> >
> > public static final String SUBSCRIPTION_NAME =
> > "transaction-buffer-sub";
> >
> > }
> > ```
> >
> > 1. In order to make the code easier to maintain and reduce redundancy, we
> > need to move systemTopic and systemSub to a specific classes to manage.
> > 2. If the newly added systemTopic and systemSub are created from this
> > class, unpredictable problems can be prevented
> >
> >
> >
> > Let me know if you have different ideas or suggestions！！
> >
> > Thanks!
> > Bo Cong
