Hi, All
After discussing with Enrico and Michael offline.
I will split the discussed topic into two PIP.
1. Topic name restrictions
a. `-partition-` keyword.
b. enable topic name character pattern.
2. System topic
a. System topic name pattern.
b. System topic authorisation.
c. ...
In this approach, we will get a clear boundary and avoid going off the initial
scope.
Since we don't have any question about the first scope. I will start vote next
week.
Thanks to all participant.
Best,
Mattison
On Feb 18, 2023, 14:24 +0800, Michael Marshall <[email protected]>, wrote:
> I support breaking this into two PIPs. It was my fault the two PIPs
> were merged in the first place. I am sorry if I created any confusion.
> My intention was only to point out that names are a meaningful way to
> simplify logic, and we should reserve certain names for Pulsar's own
> usage with a well defined pattern so that we can simplify lifecycle
> operations.
>
> Thanks,
> Michael
>
> On Fri, Feb 17, 2023 at 1:55 AM Enrico Olivelli <[email protected]> wrote:
> >
> > Mattison,
> >
> > Il giorno gio 16 feb 2023 alle ore 00:27 <[email protected]> ha
> > scritto:
> > > >
> > > > > > I am sorry but I am not sure that this is enough to
> > > > > > preventreads/writes from unallowed clients.
> > > > IMO, We can consider the authorisation part in another PIP because We
> > > > are just focusing on adding the topic name constraint of topic creation.
> > > >
> > > > Maybe we can use another PIP to clearify all of system topic's
> > > > behaviour, like authorisation something.
> > > > e.g. we just allow superusers to read/write the data to that system
> > > > topic.
> > > > > > We should elaborate more on this topic on the PIP
> > > > I will add the internal system topic creation logic in the PIP.
> > Why do you think that this is enough ?
> >
> > I think that we are going off the initial scope of the PIP.
> > The initial problem is about preventing clients from creating topics
> > that contain the "-partition-" keyword.
> >
> > I totally agree that there must be a clear way to distinguish topics
> > that are not meant to be accessed by "regular clients".
> >
> > The answer is in Micheal's words: only super users are allowed to
> > access topics that are not meant to be accessed by clients.
> > Broker to Broker communications are always running with a "super user"
> > role, so it is not a problem.
> >
> > BTW I wonder if it is better to narrow down the scope of the PIP and
> > go back to "-partition-"
> >
> >
> > Enrico
> >
> >
> > > >
> > > > Best,
> > > > Mattison
> > > > On Feb 16, 2023, 00:41 +0800, Enrico Olivelli <[email protected]>,
> > > > wrote:
> > > > > > Il giorno mer 15 feb 2023 alle ore 17:07 <[email protected]>
> > > > > > ha scritto:
> > > > > > > >
> > > > > > > > Hi Enrico
> > > > > > > >
> > > > > > > > I think it's a good question. We can introduce a new method in
> > > > > > > > the BrokerService to help brokers create the topic internally
> > > > > > > > first(maybe just metadata is enough), and then to use a pulsar
> > > > > > > > client to connect to it.
> > > > > >
> > > > > > I am sorry but I am not sure that this is enough to prevent
> > > > > > reads/writes from unallowed clients.
> > > > > > We should elaborate more on this topic on the PIP
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > > > > >
> > > > > > > > WDYT?
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Mattison
> > > > > > > > On Feb 16, 2023, 00:01 +0800, Enrico Olivelli
> > > > > > > > <[email protected]>, wrote:
> > > > > > > > > > > > I have one question (apologies for the top posting).
> > > > > > > > > > > >
> > > > > > > > > > > > The Broker (and the other Pulsar components) use the
> > > > > > > > > > > > regular Pulsar
> > > > > > > > > > > > client to connect to "system topics"
> > > > > > > > > > > > and in general they use the Pulsar wire protocol.
> > > > > > > > > > > >
> > > > > > > > > > > > The question is "how do you distinguish an internal
> > > > > > > > > > > > component from a
> > > > > > > > > > > > user component ?"
> > > > > > > > > > > > How can you say that the broker is able to connect to a
> > > > > > > > > > > > system topic
> > > > > > > > > > > > and any other client cannot do it ?
> > > > > > > > > > > >
> > > > > > > > > > > > Enrico
> > > > > > > > > > > >
> > > > > > > > > > > > Il giorno mer 15 feb 2023 alle ore 15:38
> > > > > > > > > > > > <[email protected]> ha scritto:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Asaf
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There is a link to introduce the dynamic
> > > > > > > > > > > > > > > > configuration.
> > > > > > > > > > > > > > > > https://pulsar.apache.org/docs/2.10.x/admin-api-brokers/#dynamic-broker-configuration
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Mattison
> > > > > > > > > > > > > > > > On Feb 14, 2023, 17:06 +0800, Asaf Mesika
> > > > > > > > > > > > > > > > <[email protected]>, wrote:
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 14, 2023 at 3:46 AM
> > > > > > > > > > > > > > > > > > > > > > > > <[email protected]> wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, Asaf
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Welcome to join
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > this discussion.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You mean that allows the *system* to use it when it's a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > partitioned
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > topic?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sorry, I didn't
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > get your point.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What do you
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > mean by
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *system*?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This sentence was a reply to:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2. Make the `-partition-`
> > > > > > > > > > > > > > > > > > > > > > > > string the keyword. That allows
> > > > > > > > > > > > > > > > > > > > > > > > the user to use
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > it when it's a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > partitioned
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > topic.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I wanted to say that this
> > > > > > > > > > > > > > > > > > > > > > > > sentence should be:
> > > > > > > > > > > > > > > > > > > > > > > > Make the `-partition-` string
> > > > > > > > > > > > > > > > > > > > > > > > the keyword, that allows the
> > > > > > > > > > > > > > > > > > > > > > > > *system* to use
> > > > > > > > > > > > > > > > > > > > > > > > it when it's a partitioned
> > > > > > > > > > > > > > > > > > > > > > > > topic.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Why postfix of `__`?Why uppercase ?Maybe `__system__<name>`?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, That is a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > key point that
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I want to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > discuss in this
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thread.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > `__system__<name>`
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > is good for me.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Can you please elaborate what it means to make it dynamic
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > exactly?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sorry, I will
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > refine it. it
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > means we can
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > update this
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > dynamically.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (using rest api
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > or sth)
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I'm unfamiliar with how Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > supports dynamic configuration.
> > > > > > > > > > > > > > > > > > > > > > > > I would
> > > > > > > > > > > > > > > > > > > > > > > > love it if you can share a link
> > > > > > > > > > > > > > > > > > > > > > > > or explain it briefly, thus
> > > > > > > > > > > > > > > > > > > > > > > > explaining what
> > > > > > > > > > > > > > > > > > > > > > > > exactly you are going to change
> > > > > > > > > > > > > > > > > > > > > > > > to support dynamic
> > > > > > > > > > > > > > > > > > > > > > > > configuration.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > General
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > question: In
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the last thread
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > you said
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > something about
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > configurablerules,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > etc? You
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > decided not to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > use this idea?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > IMO, That idea
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > is an advanced
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > feature. we may
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > need more time
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to discuss
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the details and
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > for the topic
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > name
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > restriction,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > maybe we don't
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > have strong
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > reason to use
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > that.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We can
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > introduce this
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > advanced
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > feature when we
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > have a need for
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I agree. Future PIP and
> > > > > > > > > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Mattison
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Feb 13,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2023, 22:21
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +0800, Asaf
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Mesika
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <[email protected]>,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You mean that allows the *system* to use it when it's a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > partitioned
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > topic?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
