Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-23 Thread Matteo Merli
On Fri, May 20, 2022 at 3:04 AM Enrico Olivelli wrote: > > Andras, > even if our rules allow to close a vote with a binding -1, we should > have to reach consensus and understand better Lari's motivations > > we are a community and accepting something with a -1 must be handled carefully > > I sugg

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-20 Thread Enrico Olivelli
t; > Matteo > > > > > > -- > > > Matteo Merli > > > > > > > > > > > > On Mon, May 2, 2022 at 3:31 AM Lari Hotari wrote: > > > > > > > > -1. It's too early to start a vote. Let's first have discussi

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-20 Thread Andras Beni
t; > > > > > Matteo > > > > -- > > Matteo Merli > > > > > > > > On Mon, May 2, 2022 at 3:31 AM Lari Hotari wrote: > > > > > > -1. It's too early to start a vote. Let's first have discussions. > > > > > >

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-20 Thread Lari Hotari
> > On Mon, May 2, 2022 at 3:31 AM Lari Hotari wrote: > > > > -1. It's too early to start a vote. Let's first have discussions. > > > > -Lari > > > > ma 2. toukok. 2022 klo 9.50 Andras Beni > > > > kirjoitti: > > > > > Hi Pu

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-19 Thread Lan Liang
ma 2. toukok. 2022 klo 9.50 Andras Beni < andras.b...@streamnative.io .invalid> kirjoitti: Hi Pulsar Community, I would like to start a VOTE on "Bucketing topic metadata to allow more topics per namespace" (PIP-157). The proposal can be read at https://github.com/apache/pulsar/issu

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-19 Thread PengHui Li
; > > Matteo > > > > > > > > -- > > > > Matteo Merli > > > > > > > > > > > > > > > > On Mon, May 2, 2022 at 3:31 AM Lari Hotari wrote: > > > > > > > > > > -1. It's too early to start a vo

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-19 Thread Hang Chen
t; > > > > > -- > > > > Matteo Merli > > > > > > > > > > > > > > > > On Mon, May 2, 2022 at 3:31 AM Lari Hotari wrote: > > > > > > > > > > -1. It's too early to start a vote. Let's fir

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-18 Thread mattison chao
; > > > > Matteo > > > > > > -- > > > Matteo Merli > > > > > > > > > > > > On Mon, May 2, 2022 at 3:31 AM Lari Hotari wrote: > > > > > > > > -1. It's too early to start a vote. Let's first have di

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-18 Thread Enrico Olivelli
a good idea to stop a vote at this point. > > > > > > Matteo > > > > -- > > Matteo Merli > > > > > > > > On Mon, May 2, 2022 at 3:31 AM Lari Hotari wrote: > > > > > > -1. It's too early to start a vote. Let's

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-17 Thread Matteo Merli
-1. It's too early to start a vote. Let's first have discussions. > > > > -Lari > > > > ma 2. toukok. 2022 klo 9.50 Andras Beni > > > > kirjoitti: > > > > > Hi Pulsar Community, > > > > > > I would like to start a VOTE on

Re: [DISCUSS] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-04 Thread mattison chao
ts > > > > > will continue to refer to topics by domain://tenant/namespace/topic > > but > > > > > pulsar will internally translate to the new persistence naming > where > > > > > necessary. > > > > > > > > > > The way met

Re: [DISCUSS] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-03 Thread Andras Beni
pics to buckets will be based on the topic name's hash > > code's > > > > absolute value modulo the number of buckets. > > > > > > > > The bulk of the changes necessary for this feature is to make > namespace > > > > policies available whereve

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-02 Thread Matteo Merli
vote at this point. Matteo -- Matteo Merli On Mon, May 2, 2022 at 3:31 AM Lari Hotari wrote: > > -1. It's too early to start a vote. Let's first have discussions. > > -Lari > > ma 2. toukok. 2022 klo 9.50 Andras Beni > kirjoitti: > > > Hi Pulsar Community,

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-02 Thread Lari Hotari
-1. It's too early to start a vote. Let's first have discussions. -Lari ma 2. toukok. 2022 klo 9.50 Andras Beni kirjoitti: > Hi Pulsar Community, > > I would like to start a VOTE on "Bucketing topic metadata to allow more > topics per namespace" (PIP-157). >

Re: [VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-02 Thread Enrico Olivelli
I left some questions. I am not sure we are ready to run a VOTE, this change will have a huge impact on Pulsar Enrico Il Lun 2 Mag 2022, 08:50 Andras Beni ha scritto: > Hi Pulsar Community, > > I would like to start a VOTE on "Bucketing topic metadata to allow more > topics pe

Re: [DISCUSS] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-02 Thread Enrico Olivelli
ever persistence naming is calculated. Where > listing > > > of topics within a namespace is necessary, the introduction of the new > > > layer will add some overhead in the form of multiple requests to the > > > metadata store. These include checking if the limit

[VOTE] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-05-01 Thread Andras Beni
Hi Pulsar Community, I would like to start a VOTE on "Bucketing topic metadata to allow more topics per namespace" (PIP-157). The proposal can be read at https://github.com/apache/pulsar/issues/15254 and the discussion thead is available at https://lists.apache.

Re: [DISCUSS] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-04-24 Thread Hang Chen
> layer will add some overhead in the form of multiple requests to the > > metadata store. These include checking if the limit on topic number per > > namespace has been reached. > > Example > > > > Let's consider the following metadata hierarchy: > > > &

Re: [DISCUSS] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-04-24 Thread PengHui Li
- nptopic1 > +- nptopic2 > +- ptopic-partition-0 > +- ptopic-partition-1 > +- ptopic-partition-2 > \- ptopic-partition-3 > > In case of 3 buckets the same topic metadata would be laid out the > fol

[DISCUSS] PIP-157: Bucketing topic metadata to allow more topics per namespace

2022-04-21 Thread Andras Beni
+- ptopic-partition-1 +- ptopic-partition-2 \- ptopic-partition-3 In case of 3 buckets the same topic metadata would be laid out the following way: managed-ledgers \- tenant \- namespace \- persistent +- $0 |

Re: Topic metadata

2021-09-23 Thread Enrico Olivelli
Joe, Il giorno gio 23 set 2021 alle ore 04:22 PengHui Li ha scritto: > Another option is to store the custom properties of the topic to the system > topic as the topic policy does. > The limitation is if using the system topic, we are not able to guarantee > the atomic operation for the topic cr

Re: Topic metadata

2021-09-22 Thread PengHui Li
Another option is to store the custom properties of the topic to the system topic as the topic policy does. The limitation is if using the system topic, we are not able to guarantee the atomic operation for the topic creation and properties adding, But it is required in some cases. Yes, the manage

Re: Topic metadata

2021-09-22 Thread Joe F
I don't think pulsar should increase its dependency on ZK. Can we store it somewhere else? On Wed, Sep 22, 2021, 3:30 PM Matteo Merli wrote: > We already have a mechanism to store custom properties in managed > ledgers. So we don't need to store it separately. > -- > Matteo Merli > > > On Wed,

Re: Topic metadata

2021-09-22 Thread Matteo Merli
We already have a mechanism to store custom properties in managed ledgers. So we don't need to store it separately. -- Matteo Merli On Wed, Sep 22, 2021 at 3:21 PM Michael Marshall wrote: > > Hi Enrico, > > This sounds like a useful feature. Do you expect to use it internally in > pulsar or shou

Re: Topic metadata

2021-09-22 Thread Michael Marshall
Hi Enrico, This sounds like a useful feature. Do you expect to use it internally in pulsar or should it be limited to external applications? In thinking about authorization, what level of permission should be required to create/update/delete these labels? It seems to me that these should require

Re: Topic metadata

2021-09-19 Thread PengHui Li
Hi enrico +1, sorry I missed the email before. It should be a useful feature for Pulsar users and I have a discussion with the Flink Pulsar connector authors several days ago, they also looking for this feature. Thanks Penghui Enrico Olivelli 于2021年9月12日 周日17:24写道: > Hello, > I would like to ha

Topic metadata

2021-09-12 Thread Enrico Olivelli
Hello, I would like to have the ability to store metadata about topics. This would be very useful as with metadata you could add labels and other pieces of information that would allow to define the purpose of a topic, custom application level properties. This feature will allow application level