Re: [DISCUSS] Redundant ServiceUrlProvider design and improper use of PIP-121

[Baodi Shi]

 Hi, Yunze:

Obviously, the `ServiceUrlProvider` config is redundant.
>

Agree. In fact, The client already provides the updateServiceUrl method,
which the user can use to implement a dynamic update service URL. As for
how the user implements it and how to close his resources, I think it can
be left to the user.


在 2023年1月19日 15:16:52 上，Yunze Xu  写道：

> Hi all,
>
> Currently we have a `ServiceUrlProvider` interface to configure when
> constructing a PulsarClient in `ClientBuilder#serviceUrlProvider`.
> From the beginning, I thought the `getServiceUrl` method is called
> each time the service URL is used, e.g. topic metadata lookup.
> However, the `getServiceUrl` method is only called when constructing
> the PulsarClient object. To update the PulsarClient's internal service
> URL, `PulsarClient#updateServiceUrl` must be called. Therefore, if we
> want to implement a `ServiceUrlProvider` that retrieves the latest
> service URL from a database, I have to implement it like:
>
> ```java
> class DataBaseServiceUrlProvider implements ServiceUrlProvider {
>
>private final ScheduledExecutorService executor =
> Executors.newSingleThreadScheduledExecutor();
>
>@Override
>public void initialize(PulsarClient client) {
>executor.schedule(() -> {
>try {
>client.updateServiceUrl(readServiceUrlFromDB()/* a
> fake method */);
>} catch (PulsarClientException e) {
>throw new RuntimeException(e);
>}
>}, 1, TimeUnit.SECONDS);
>}
>
>@Override
>public String getServiceUrl() {
>return "pulsar://localhost:6650";
>}
>
>@Override
>public void close() {
>executor.shutdown();
>}
> }
> ```
>
> The key point is, if we didn't call `client.updateServiceUrl` and only
> modified the returned value of `getServiceUrl` periodically, the
> internal service URL would never be updated.
>
> Based on the provider above, the following two code snippets could be
> nearly the same.
>
> ```java
> var client = PulsarClient.builder().serviceUrlProvider(new
> DataBaseServiceUrlProvider()).build();
> /* ... */
> client.close();
> ```
>
> ```java
> var client =
> PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
> var provider = new DataBaseServiceUrlProvider();
> provider.initialize(client);
> /* ... */
> provider.close();
> client.close();
> ```
>
> Obviously, the `ServiceUrlProvider` config is redundant.
>
> PIP-121 implements the `AutoClusterFailover` as the service URL
> provider. However, it also calls the following methods periodically:
> - PulsarClientImpl#updateAuthentication
> - PulsarClientImpl#updateTlsTrustCertsFilePath
>
> It's unnatural and intuitive to say a service URL provider could
> modify the internal states of `PulsarClient`, including:
> - the service URL
> - the authentication
> - the TLS trust certificate file
> - ...
>
> BTW, the implementation of PIP-121 [1] is different from the design [2].
>
> [1] https://github.com/apache/pulsar/pull/13316
> [2] https://github.com/apache/pulsar/issues/13315
>
> Thanks,
> Yunze
>

Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker

[Yunze Xu]

I think we can enter the VOTE process now. And I agree with Enrico
that in "Alternatives" you should mention the approach of
adding a boolean (optional) field to the subscribe request.

Thanks,
Yunze

On Wed, Jan 18, 2023 at 9:52 PM PengHui Li  wrote:
>
> > I see the updated PR adopts this suggestion to set the enum value with
> 100. But I'm still wondering why not just use the next enum value
> (21)? What makes AUTO_CONSUME schema different from other schema types
> like ProtobufNativeSchema (20).
>
> Ah, yes, 21 looks good to me.
>
> Penghui
>
> On Tue, Jan 17, 2023 at 9:11 PM SiNan Liu  wrote:
>
> > The value of enum has been updated in the PIP issue
> > .
> >
> > Thanks,
> > Sinan
> >
> > Yunze Xu  于2023年1月17日周二 12:11写道：
> >
> > > > Maybe 100 or something.
> > >
> > > I see the updated PR adopts this suggestion to set the enum value with
> > > 100. But I'm still wondering why not just use the next enum value
> > > (21)? What makes AUTO_CONSUME schema different from other schema types
> > > like ProtobufNativeSchema (20).
> > >
> > > Thanks,
> > > Yunze
> > >
> > > On Mon, Jan 16, 2023 at 12:08 PM PengHui Li  wrote:
> > > >
> > > > > Is there any problem with using a positive value for it? I think
> > there
> > > > is no compatibility issue because the enum value is never used on the
> > > > broker side. Making it positive makes AUTO_CONSUME different with
> > > > other implicit schema types like BYTES, AUTO and AUTO_PUBLISH.
> > > >
> > > > That sounds good to me to use a positive number for `AUTO_CONSUME`
> > > > in the protocol. Maybe 100 or something.
> > > >
> > > > Thanks,
> > > > Penghui
> > > >
> > > > On Mon, Jan 16, 2023 at 10:38 AM Yunze Xu  > >
> > > > wrote:
> > > >
> > > > > Is there any problem with using a positive value for it? I think
> > there
> > > > > is no compatibility issue because the enum value is never used on the
> > > > > broker side. Making it positive makes AUTO_CONSUME different with
> > > > > other implicit schema types like BYTES, AUTO and AUTO_PUBLISH.
> > > > >
> > > > > Thanks,
> > > > > Yunze
> > > > >
> > > > > On Mon, Jan 16, 2023 at 9:36 AM PengHui Li 
> > wrote:
> > > > > >
> > > > > > > This design also has serious compatibility problems between old
> > > and new
> > > > > > pulsar clients and new and old brokers.
> > > > > >
> > > > > > Could you please explain more details of the compatibility issue if
> > > we
> > > > > > leverage
> > > > > > the protocol version?
> > > > > >
> > > > > > > We should not use a negative enum number in PulsarApi.proto. It's
> > > > > > unnatural. If we decide to carry the AUTO_CONSUME schema in a
> > > > > > CommandSubscribe, it should not be treated as a negative schema
> > type.
> > > > > >
> > > > > > IMO, the protocol is defined as Enum. Users are developing based on
> > > the
> > > > > > Enum, not the value of the Enum. We need to make sure the value
> > > > > > of the Enum is immutable. It is not required that he must be a
> > > positive
> > > > > > number.
> > > > > > Maybe it looks ugly.
> > > > > >
> > > > > > And the protocol is just the API definition, not about which schema
> > > will
> > > > > be
> > > > > > persistent.
> > > > > > As I understand from the protocol definition, the Schema in the
> > > subscribe
> > > > > > command is
> > > > > > used to pass the used schema of the consumer. We just make it
> > absent
> > > > > before
> > > > > > for
> > > > > > AUTO_CONSUME schema. We just thought we could make it absent if the
> > > > > consumer
> > > > > > is using AUTO_CONSUME schema. But apparently, this is a problem for
> > > now.
> > > > > >
> > > > > > I think the easier-to-understand way is for the client to set the
> > > schema
> > > > > > used when
> > > > > > subscribing or creating the producer. Rather than which ones need
> > to
> > > be
> > > > > set
> > > > > > and which
> > > > > > ones do not need to be set.
> > > > > >
> > > > > > Thanks,
> > > > > > Penghui
> > > > > >
> > > > > > On Mon, Jan 9, 2023 at 11:32 AM SiNan Liu 
> > > > > wrote:
> > > > > >
> > > > > > > This design also has serious compatibility problems between old
> > > and new
> > > > > > > pulsar clients and new and old brokers.
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Sinan
> > > > > > >
> > > > > > >
> > > > > > > PengHui Li  于 2023年1月9日周一 上午9:51写道：
> > > > > > >
> > > > > > > > Sorry for the late reply.
> > > > > > > >
> > > > > > > > We can leverage the `ProtocolVersion` [1] to handle the
> > > compatibility
> > > > > > > > issue.
> > > > > > > > It looks like only if the protocol_version >= 21, subscribe
> > with
> > > the
> > > > > > > > auto_consume schema
> > > > > > > >
> > > > > > > > IMO, both the new key-value of the subscribe command, and a
> > > specific
> > > > > > > > representative are
> > > > > > > > API changes. It's just that some have modified the definition
> > of
> > > the
> > > > > API,
> > > > > > > > and some have modified the behavior 

[DISCUSS] Redundant ServiceUrlProvider design and improper use of PIP-121

[Yunze Xu]

Hi all,

Currently we have a `ServiceUrlProvider` interface to configure when
constructing a PulsarClient in `ClientBuilder#serviceUrlProvider`.
>From the beginning, I thought the `getServiceUrl` method is called
each time the service URL is used, e.g. topic metadata lookup.
However, the `getServiceUrl` method is only called when constructing
the PulsarClient object. To update the PulsarClient's internal service
URL, `PulsarClient#updateServiceUrl` must be called. Therefore, if we
want to implement a `ServiceUrlProvider` that retrieves the latest
service URL from a database, I have to implement it like:

```java
class DataBaseServiceUrlProvider implements ServiceUrlProvider {

private final ScheduledExecutorService executor =
Executors.newSingleThreadScheduledExecutor();

@Override
public void initialize(PulsarClient client) {
executor.schedule(() -> {
try {
client.updateServiceUrl(readServiceUrlFromDB()/* a
fake method */);
} catch (PulsarClientException e) {
throw new RuntimeException(e);
}
}, 1, TimeUnit.SECONDS);
}

@Override
public String getServiceUrl() {
return "pulsar://localhost:6650";
}

@Override
public void close() {
executor.shutdown();
}
}
```

The key point is, if we didn't call `client.updateServiceUrl` and only
modified the returned value of `getServiceUrl` periodically, the
internal service URL would never be updated.

Based on the provider above, the following two code snippets could be
nearly the same.

```java
var client = PulsarClient.builder().serviceUrlProvider(new
DataBaseServiceUrlProvider()).build();
/* ... */
client.close();
```

```java
var client = 
PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
var provider = new DataBaseServiceUrlProvider();
provider.initialize(client);
/* ... */
provider.close();
client.close();
```

Obviously, the `ServiceUrlProvider` config is redundant.

PIP-121 implements the `AutoClusterFailover` as the service URL
provider. However, it also calls the following methods periodically:
- PulsarClientImpl#updateAuthentication
- PulsarClientImpl#updateTlsTrustCertsFilePath

It's unnatural and intuitive to say a service URL provider could
modify the internal states of `PulsarClient`, including:
- the service URL
- the authentication
- the TLS trust certificate file
- ...

BTW, the implementation of PIP-121 [1] is different from the design [2].

[1] https://github.com/apache/pulsar/pull/13316
[2] https://github.com/apache/pulsar/issues/13315

Thanks,
Yunze

Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar

[Xiangying Meng]

Congratulations

Thanks
Xiangying

On Thu, Jan 19, 2023 at 1:54 PM Zixuan Liu  wrote:

> Congrats! Bo
>
> Best,
> Zixuan
>
> ZhangJian He  于2023年1月19日周四 12:33写道：
>
> > Congratulations
> >
> > Thanks
> > ZhangJian He
> >
> >
> > On Thu, 19 Jan 2023 at 11:47, r...@apache.org 
> > wrote:
> >
> > > Congratulations!!!
> > >
> > > --
> > > Thanks
> > > Xiaolong Ran
> > >
> > > Baodi Shi  于2023年1月19日周四 10:08写道：
> > >
> > > >  Congratulations !
> > > >
> > > > Thanks,
> > > > Baodi Shi
> > > >
> > > >
> > > > 在 2023年1月19日 09:56:34 上，Kai Wang  写道：
> > > >
> > > > > Congratulations!
> > > > >
> > > > > Thanks,
> > > > > Kai
> > > > > On Jan 18, 2023 at 9:50 PM +0800, PengHui Li ,
> > > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > > The Apache Pulsar Project Management Committee (PMC) has invited Bo
> > > Cong
> > > > >
> > > > > (https://github.com/congbobo184) as a member of the PMC and we are
> > > > >
> > > > > pleased to announce that he has accepted.
> > > > >
> > > > >
> > > > > He is very active in the community in the past few years and made a
> > lot
> > > > of
> > > > >
> > > > > great contributions
> > > > >
> > > > > such as transactions and schemas.
> > > > >
> > > > >
> > > > > Welcome Bo Cong to the Apache Pulsar PMC.
> > > > >
> > > > >
> > > > > Best Regards,
> > > > >
> > > > > Penghui on behalf of the Pulsar PMC
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar

[Zixuan Liu]

Congrats! Bo

Best,
Zixuan

ZhangJian He  于2023年1月19日周四 12:33写道：

> Congratulations
>
> Thanks
> ZhangJian He
>
>
> On Thu, 19 Jan 2023 at 11:47, r...@apache.org 
> wrote:
>
> > Congratulations!!!
> >
> > --
> > Thanks
> > Xiaolong Ran
> >
> > Baodi Shi  于2023年1月19日周四 10:08写道：
> >
> > >  Congratulations !
> > >
> > > Thanks,
> > > Baodi Shi
> > >
> > >
> > > 在 2023年1月19日 09:56:34 上，Kai Wang  写道：
> > >
> > > > Congratulations!
> > > >
> > > > Thanks,
> > > > Kai
> > > > On Jan 18, 2023 at 9:50 PM +0800, PengHui Li ,
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > >
> > > > The Apache Pulsar Project Management Committee (PMC) has invited Bo
> > Cong
> > > >
> > > > (https://github.com/congbobo184) as a member of the PMC and we are
> > > >
> > > > pleased to announce that he has accepted.
> > > >
> > > >
> > > > He is very active in the community in the past few years and made a
> lot
> > > of
> > > >
> > > > great contributions
> > > >
> > > > such as transactions and schemas.
> > > >
> > > >
> > > > Welcome Bo Cong to the Apache Pulsar PMC.
> > > >
> > > >
> > > > Best Regards,
> > > >
> > > > Penghui on behalf of the Pulsar PMC
> > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Committer: Baodi Shi

[Zixuan Liu]

Congrats! Baodi

Best,
Zixuan

ZhangJian He  于2023年1月19日周四 12:34写道：

> Congratulations
>
> Thanks
> ZhangJian He
>
>
> On Thu, 19 Jan 2023 at 11:47, r...@apache.org 
> wrote:
>
> > Congratulations!!
> >
> > --
> > Thanks
> > Xiaolong Ran
> >
> > tison  于2023年1月19日周四 10:07写道：
> >
> > > Congrats and well deserved!
> > >
> > > Welcome onboard, Baodi :)
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Kai Wang  于2023年1月19日周四 09:57写道：
> > >
> > > > Congratulations!
> > > >
> > > > Thanks,
> > > > Kai
> > > > On Jan 18, 2023 at 9:36 PM +0800, dev@pulsar.apache.org, wrote:
> > > > >
> > > > > Congratulations !
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Committer: Baodi Shi

[ZhangJian He]

Congratulations

Thanks
ZhangJian He


On Thu, 19 Jan 2023 at 11:47, r...@apache.org 
wrote:

> Congratulations!!
>
> --
> Thanks
> Xiaolong Ran
>
> tison  于2023年1月19日周四 10:07写道：
>
> > Congrats and well deserved!
> >
> > Welcome onboard, Baodi :)
> >
> > Best,
> > tison.
> >
> >
> > Kai Wang  于2023年1月19日周四 09:57写道：
> >
> > > Congratulations!
> > >
> > > Thanks,
> > > Kai
> > > On Jan 18, 2023 at 9:36 PM +0800, dev@pulsar.apache.org, wrote:
> > > >
> > > > Congratulations !
> > >
> >
>

Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar

[ZhangJian He]

Congratulations

Thanks
ZhangJian He


On Thu, 19 Jan 2023 at 11:47, r...@apache.org 
wrote:

> Congratulations!!!
>
> --
> Thanks
> Xiaolong Ran
>
> Baodi Shi  于2023年1月19日周四 10:08写道：
>
> >  Congratulations !
> >
> > Thanks,
> > Baodi Shi
> >
> >
> > 在 2023年1月19日 09:56:34 上，Kai Wang  写道：
> >
> > > Congratulations!
> > >
> > > Thanks,
> > > Kai
> > > On Jan 18, 2023 at 9:50 PM +0800, PengHui Li ,
> > wrote:
> > >
> > > Hi all,
> > >
> > >
> > > The Apache Pulsar Project Management Committee (PMC) has invited Bo
> Cong
> > >
> > > (https://github.com/congbobo184) as a member of the PMC and we are
> > >
> > > pleased to announce that he has accepted.
> > >
> > >
> > > He is very active in the community in the past few years and made a lot
> > of
> > >
> > > great contributions
> > >
> > > such as transactions and schemas.
> > >
> > >
> > > Welcome Bo Cong to the Apache Pulsar PMC.
> > >
> > >
> > > Best Regards,
> > >
> > > Penghui on behalf of the Pulsar PMC
> > >
> > >
> >
>

Re: [DISCUSS] PIP-237: Make PulsarAdmin accessible in SinkContext and SourceContext

[r...@apache.org]

Hello Alexander, thanks your work for this.
It seems that all agrees with this PIP, maybe we can enter the VOTE thread
to initiate a VOTE process for this PIP.

--
Thanks
Xiaolong Ran

r...@apache.org  于2023年1月19日周四 11:57写道：

> +1
>
> We can imitate the PulsarClient object, and in the BaseContext interface,
> add PulsarAdmin and PulsarAdminBuilder as interfaces at the same time.
>
> --
> Thanks
> Xiaolong Ran
>
> Neng Lu  于2023年1月19日周四 02:28写道：
>
>> The proposal makes sense to me.
>>
>> The only note I want to add is to make sure the PulsarAdmin access in
>> Source and Sink is also controlled by the flag
>> `exposePulsarAdminClientEnabled`. Similar to the following:
>>
>>
>> https://github.com/apache/pulsar/blob/master/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/thread/ThreadRuntimeFactory.java#L111
>>
>> On 2023/01/03 14:31:52 Alexander Preuss wrote:
>> > Hi all,
>> >
>> > I opened a PIP to discuss making PulsarAdmin accessible in Pulsar IO
>> > connectors through SinkContext and SourceContext:
>> > https://github.com/apache/pulsar/issues/19123
>> >
>> > Looking forward to hearing your thoughts,
>> > Alex
>> >
>>
>

Re: [DISCUSS] PIP-237: Make PulsarAdmin accessible in SinkContext and SourceContext

[r...@apache.org]

+1

We can imitate the PulsarClient object, and in the BaseContext interface,
add PulsarAdmin and PulsarAdminBuilder as interfaces at the same time.

--
Thanks
Xiaolong Ran

Neng Lu  于2023年1月19日周四 02:28写道：

> The proposal makes sense to me.
>
> The only note I want to add is to make sure the PulsarAdmin access in
> Source and Sink is also controlled by the flag
> `exposePulsarAdminClientEnabled`. Similar to the following:
>
>
> https://github.com/apache/pulsar/blob/master/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/thread/ThreadRuntimeFactory.java#L111
>
> On 2023/01/03 14:31:52 Alexander Preuss wrote:
> > Hi all,
> >
> > I opened a PIP to discuss making PulsarAdmin accessible in Pulsar IO
> > connectors through SinkContext and SourceContext:
> > https://github.com/apache/pulsar/issues/19123
> >
> > Looking forward to hearing your thoughts,
> > Alex
> >
>

Re: [ANNOUNCE] New Committer: Baodi Shi

[r...@apache.org]

Congratulations!!

--
Thanks
Xiaolong Ran

tison  于2023年1月19日周四 10:07写道：

> Congrats and well deserved!
>
> Welcome onboard, Baodi :)
>
> Best,
> tison.
>
>
> Kai Wang  于2023年1月19日周四 09:57写道：
>
> > Congratulations!
> >
> > Thanks,
> > Kai
> > On Jan 18, 2023 at 9:36 PM +0800, dev@pulsar.apache.org, wrote:
> > >
> > > Congratulations !
> >
>

Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar

[r...@apache.org]

Congratulations!!!

--
Thanks
Xiaolong Ran

Baodi Shi  于2023年1月19日周四 10:08写道：

>  Congratulations !
>
> Thanks,
> Baodi Shi
>
>
> 在 2023年1月19日 09:56:34 上，Kai Wang  写道：
>
> > Congratulations!
> >
> > Thanks,
> > Kai
> > On Jan 18, 2023 at 9:50 PM +0800, PengHui Li ,
> wrote:
> >
> > Hi all,
> >
> >
> > The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong
> >
> > (https://github.com/congbobo184) as a member of the PMC and we are
> >
> > pleased to announce that he has accepted.
> >
> >
> > He is very active in the community in the past few years and made a lot
> of
> >
> > great contributions
> >
> > such as transactions and schemas.
> >
> >
> > Welcome Bo Cong to the Apache Pulsar PMC.
> >
> >
> > Best Regards,
> >
> > Penghui on behalf of the Pulsar PMC
> >
> >
>

Re: [DISCUSS] PIP-240 A new API to unload subscriptions

[r...@apache.org]

Hello Joe and Enrico:

I agree with what you've been emphasizing that we need to fix these issues
at the root cause. During the maintenance of the Go SDK, we have
encountered many stuck problems since version 0.4.0, some of which belonged
to the logic errors handled by the Go SDK itself, and some of which were
caused by the user's wrong use of the Go SDK, until the previous 0.8 .0
version, the Go SDK is used on a large scale in our environment. In the
iterations of these versions, we have been trying to completely fix these
BUGs. This is what our maintainers have been working hard on and it is also
a final form we expect Pulsar - everything looks OK.

However, during the iteration of the Go SDK version from 0.4.0 to 0.8.0,
users of our production environment encountered similar problems many
times. Again, for a user in a production environment, for example, the
current user encounters a situation where consumption is blocked. The user
finds you and expects us to use some means to quickly allow consumers to
continue to consume news? Or do we keep users in the production environment
in a stuck state until we find the root cause of the problem and fix it for
users, pushing users to upgrade. I think everyone's answer tends to be the
latter. We will not directly expose the hack operations of unload topic and
unload sub to users, but to Pulsar's operation and maintenance personnel,
so it is more like an operation and maintenance tool , rather than the
interface called by the user. So I think this impact is controllable for
Pulsar as a whole, which is why I support it.

Again, this PIP is more about buying more time for us to locate the problem
while minimizing the impact on production users. It’s not that with this
interface we don’t locate the real causes of the stuck. On the contrary, we
are making more trade-offs between users and positioning issues, buying us
more time for positioning issues.

--
Thanks
xiaolong ran

PengHui Li  于2023年1月18日周三 11:48写道：

> > What kind of problems is this trying to fix?
> And why cannot that be solved by client-side fixes?
>
> Yes, most of the issue is from the client side, rarely from the broker.
> But the application also needs time to fix the issue to release and deploy
> the fix
> to the production environment. Unloading the subscription is just a
> temporary
> way to mitigate the issue and reduce the impact. It will not fix the issue
> completely.
>
> What I learned is to capture the heap dump, topics stats, internal stats,
> and logs from the broker and client and then try to unload the topic to
> see if the problem is mitigated. If not, then try to restart the broker or
> client,
> most of the time, the problem can be mitigated in this way.
> Then we can continue to reproduce the issue and investigate the issue
> from the captured heap dump and logs.
>
> > In shared sub issues, it's hard to  pinpoint which consumer/where
> the problem lies, and to reset that one at the client. The totality of
> state spread between the brokers and all the consumers of the shared sub
> needs to be put together .  Is that why we are doing this?
>
> From my experience, most are from Shared and key shared subscriptions.
> Most of the issues come from misuse, rarely from the BUGs of brokers or
> clients.
>
> Regards,
> Penghui
>
>
> On Wed, Jan 18, 2023 at 11:31 AM Joe F  wrote:
>
> > Inclined to agree with Enrico.  If it's a hard problem, it will repeat,
> and
> > this is not helping.  If it's some race on the client, it will occur
> > randomly and rarely, and this unload sub will get programmed in as a way
> of
> > life.
> >
> > >If you don't think unloading the subscription can't help anything.
> > Unloading
> > the topic should be the same. From my experience, most of the unloading
> > topic operations are to mitigate the problems related to message
> > consumption.
> >
> > Comparisons with unloading a topic are not the bar here, as that is a
> first
> > class broker utility that is needed for operational reasons outside of
> > "fixing"  consumer side issues . The side effect of using "unload topic"
> is
> > a loss of transient topic state. I will fully agree that this side-effect
> > has been  pervasively abused for fixing problems (ala Ctlrl-Alt-Del) ,
> but
> > that's not the rationale for having an unload topic utility.
> >
> > What kind of problems is this trying to fix?
> > And why cannot that be solved by client-side fixes?
> >
> > In shared sub issues, it's hard to  pinpoint which consumer/where
> > the problem lies, and to reset that one at the client. The totality of
> > state spread between the brokers and all the consumers of the shared sub
> > needs to be put together .  Is that why we are doing this?
> >
> >
> > On Tue, Jan 17, 2023 at 5:30 PM PengHui Li  wrote:
> >
> > > I agree that if we encounter a stuck consumption issue, we should
> > continue
> > > to find the root cause of the problem.
> > >
> > > Subscription unloading is just an option to mitigate the impact 

Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar

[Baodi Shi]

 Congratulations !

Thanks,
Baodi Shi


在 2023年1月19日 09:56:34 上，Kai Wang  写道：

> Congratulations!
>
> Thanks,
> Kai
> On Jan 18, 2023 at 9:50 PM +0800, PengHui Li , wrote:
>
> Hi all,
>
>
> The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong
>
> (https://github.com/congbobo184) as a member of the PMC and we are
>
> pleased to announce that he has accepted.
>
>
> He is very active in the community in the past few years and made a lot of
>
> great contributions
>
> such as transactions and schemas.
>
>
> Welcome Bo Cong to the Apache Pulsar PMC.
>
>
> Best Regards,
>
> Penghui on behalf of the Pulsar PMC
>
>

Re: [ANNOUNCE] New Committer: Baodi Shi

[tison]

Congrats and well deserved!

Welcome onboard, Baodi :)

Best,
tison.


Kai Wang  于2023年1月19日周四 09:57写道：

> Congratulations!
>
> Thanks,
> Kai
> On Jan 18, 2023 at 9:36 PM +0800, dev@pulsar.apache.org, wrote:
> >
> > Congratulations !
>

Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar

[Kai Wang]

Congratulations!

Thanks,
Kai
On Jan 18, 2023 at 9:50 PM +0800, PengHui Li , wrote:
> Hi all,
>
> The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong
> (https://github.com/congbobo184) as a member of the PMC and we are
> pleased to announce that he has accepted.
>
> He is very active in the community in the past few years and made a lot of
> great contributions
> such as transactions and schemas.
>
> Welcome Bo Cong to the Apache Pulsar PMC.
>
> Best Regards,
> Penghui on behalf of the Pulsar PMC

Re: [ANNOUNCE] New Committer: Baodi Shi

[Kai Wang]

Congratulations!

Thanks,
Kai
On Jan 18, 2023 at 9:36 PM +0800, dev@pulsar.apache.org, wrote:
>
> Congratulations !

Re: [DISCUSS] PIP-237: Make PulsarAdmin accessible in SinkContext and SourceContext

[Neng Lu]

The proposal makes sense to me.

The only note I want to add is to make sure the PulsarAdmin access in Source 
and Sink is also controlled by the flag `exposePulsarAdminClientEnabled`. 
Similar to the following:

https://github.com/apache/pulsar/blob/master/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/thread/ThreadRuntimeFactory.java#L111

On 2023/01/03 14:31:52 Alexander Preuss wrote:
> Hi all,
> 
> I opened a PIP to discuss making PulsarAdmin accessible in Pulsar IO
> connectors through SinkContext and SourceContext:
> https://github.com/apache/pulsar/issues/19123
> 
> Looking forward to hearing your thoughts,
> Alex
> 

Re: [DISCUSS] PIP-237: Make PulsarAdmin accessible in SinkContext and SourceContext

[Alexander Preuss]

Enrico,
>I suggest to put option B into the API changes and move option A
to"Alternatives"

Thanks for your input, I updated the GitHub issue accordingly.

Best,
Alexander

On Tue, Jan 17, 2023 at 3:17 AM Bonan Hou 
wrote:

> +1 for option b.
>
> Bonan Hou
>
> On Tue, Jan 3, 2023 at 10:32 PM Alexander Preuss
>  wrote:
>
> > Hi all,
> >
> > I opened a PIP to discuss making PulsarAdmin accessible in Pulsar IO
> > connectors through SinkContext and SourceContext:
> > https://github.com/apache/pulsar/issues/19123
> >
> > Looking forward to hearing your thoughts,
> > Alex
> >
>

Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar

[houxiaoyu]

Congrats!!

Hou Xiaoyu

Nicolò Boschi  于2023年1月18日周三 22:10写道：

> Congrats!!
>
> Nicolò
>
> Il giorno mer 18 gen 2023 alle 15:06 Enrico Olivelli 
> ha scritto:
>
> > Congrats !
> >
> > Enrico
> >
> > Il giorno mer 18 gen 2023 alle ore 14:50 PengHui Li
> >  ha scritto:
> > >
> > > Hi all,
> > >
> > > The Apache Pulsar Project Management Committee (PMC) has invited Bo
> Cong
> > > (https://github.com/congbobo184) as a member of the PMC and we are
> > > pleased to announce that he has accepted.
> > >
> > > He is very active in the community in the past few years and made a lot
> > of
> > > great contributions
> > > such as transactions and schemas.
> > >
> > > Welcome Bo Cong to the Apache Pulsar PMC.
> > >
> > > Best Regards,
> > > Penghui on behalf of the Pulsar PMC
> >
> --
> Nicolò Boschi
>

Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar

[Nicolò Boschi]

Congrats!!

Nicolò

Il giorno mer 18 gen 2023 alle 15:06 Enrico Olivelli 
ha scritto:

> Congrats !
>
> Enrico
>
> Il giorno mer 18 gen 2023 alle ore 14:50 PengHui Li
>  ha scritto:
> >
> > Hi all,
> >
> > The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong
> > (https://github.com/congbobo184) as a member of the PMC and we are
> > pleased to announce that he has accepted.
> >
> > He is very active in the community in the past few years and made a lot
> of
> > great contributions
> > such as transactions and schemas.
> >
> > Welcome Bo Cong to the Apache Pulsar PMC.
> >
> > Best Regards,
> > Penghui on behalf of the Pulsar PMC
>
-- 
Nicolò Boschi

Re: [ANNOUNCE] New Committer: Baodi Shi

[Enrico Olivelli]

Congratulations !

Enrico

Il giorno mer 18 gen 2023 alle ore 14:36 Yunze Xu  ha scritto:
>
> The Project Management Committee (PMC) for Apache Pulsar has invited
> Baodi Shi (https://github.com/shibd) to become a committer and we are
> pleased to announce that he has accepted.
>
> Being a committer enables easier contribution to the project since
> there is no need to go via the patch submission process. This should
> enable better productivity.
>
> Welcome and congratulations, Baodi Shi!
>
> Please join us in congratulating and welcoming Baodi Shi onboard!
>
> Thanks,
> Yunze on behalf of the Pulsar PMC

Re: [ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar

[Enrico Olivelli]

Congrats !

Enrico

Il giorno mer 18 gen 2023 alle ore 14:50 PengHui Li
 ha scritto:
>
> Hi all,
>
> The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong
> (https://github.com/congbobo184) as a member of the PMC and we are
> pleased to announce that he has accepted.
>
> He is very active in the community in the past few years and made a lot of
> great contributions
> such as transactions and schemas.
>
> Welcome Bo Cong to the Apache Pulsar PMC.
>
> Best Regards,
> Penghui on behalf of the Pulsar PMC

Re: Facilitate Pulsar shell distribution

[Enrico Olivelli]

+1

As written on the GH issue we already have some experience with
packing pulsar-shell for homebrew

One open point is how to coordinate it with the Apache downloads framework.

Apache artifacts should be hosted / downloaded from downloads.apache.org
so we have to add some tricks/redirections on the pulsar.apache.org
website in order to make it work properly

We will also have to update the release procedure

Enrico

Il giorno mer 18 gen 2023 alle ore 14:06 Alexander Preuss
 ha scritto:
>
> Hi Nicolò,
>
> Thanks for driving this topic! I think it is a really good idea to improve
> the distribution.
> I had a read through the GH issue and I agree, that having the shell
> available through homebrew would be nice.
>
> Best,
> Alexander
>
> On Wed, Jan 18, 2023 at 11:11 AM Nicolò Boschi  wrote:
>
> > Hi folks,
> >
> > Now that Pulsar 2.11.0 has been released I'd like to discuss possible
> > improvements for Pulsar shell's user experience.
> >
> > Now you either download the tarball or use a docker image, we can improve
> > it.
> >
> > I've opened an issue with some ideas:
> > https://github.com/apache/pulsar/issues/19272
> > Please share your thoughts
> >
> > Thanks,
> > Nicolò
> >

Re: [DISCUSS] PIP-236: Upload AUTO_CONSUME SchemaType to Broker

[PengHui Li]

> I see the updated PR adopts this suggestion to set the enum value with
100. But I'm still wondering why not just use the next enum value
(21)? What makes AUTO_CONSUME schema different from other schema types
like ProtobufNativeSchema (20).

Ah, yes, 21 looks good to me.

Penghui

On Tue, Jan 17, 2023 at 9:11 PM SiNan Liu  wrote:

> The value of enum has been updated in the PIP issue
> .
>
> Thanks,
> Sinan
>
> Yunze Xu  于2023年1月17日周二 12:11写道：
>
> > > Maybe 100 or something.
> >
> > I see the updated PR adopts this suggestion to set the enum value with
> > 100. But I'm still wondering why not just use the next enum value
> > (21)? What makes AUTO_CONSUME schema different from other schema types
> > like ProtobufNativeSchema (20).
> >
> > Thanks,
> > Yunze
> >
> > On Mon, Jan 16, 2023 at 12:08 PM PengHui Li  wrote:
> > >
> > > > Is there any problem with using a positive value for it? I think
> there
> > > is no compatibility issue because the enum value is never used on the
> > > broker side. Making it positive makes AUTO_CONSUME different with
> > > other implicit schema types like BYTES, AUTO and AUTO_PUBLISH.
> > >
> > > That sounds good to me to use a positive number for `AUTO_CONSUME`
> > > in the protocol. Maybe 100 or something.
> > >
> > > Thanks,
> > > Penghui
> > >
> > > On Mon, Jan 16, 2023 at 10:38 AM Yunze Xu  >
> > > wrote:
> > >
> > > > Is there any problem with using a positive value for it? I think
> there
> > > > is no compatibility issue because the enum value is never used on the
> > > > broker side. Making it positive makes AUTO_CONSUME different with
> > > > other implicit schema types like BYTES, AUTO and AUTO_PUBLISH.
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Mon, Jan 16, 2023 at 9:36 AM PengHui Li 
> wrote:
> > > > >
> > > > > > This design also has serious compatibility problems between old
> > and new
> > > > > pulsar clients and new and old brokers.
> > > > >
> > > > > Could you please explain more details of the compatibility issue if
> > we
> > > > > leverage
> > > > > the protocol version?
> > > > >
> > > > > > We should not use a negative enum number in PulsarApi.proto. It's
> > > > > unnatural. If we decide to carry the AUTO_CONSUME schema in a
> > > > > CommandSubscribe, it should not be treated as a negative schema
> type.
> > > > >
> > > > > IMO, the protocol is defined as Enum. Users are developing based on
> > the
> > > > > Enum, not the value of the Enum. We need to make sure the value
> > > > > of the Enum is immutable. It is not required that he must be a
> > positive
> > > > > number.
> > > > > Maybe it looks ugly.
> > > > >
> > > > > And the protocol is just the API definition, not about which schema
> > will
> > > > be
> > > > > persistent.
> > > > > As I understand from the protocol definition, the Schema in the
> > subscribe
> > > > > command is
> > > > > used to pass the used schema of the consumer. We just make it
> absent
> > > > before
> > > > > for
> > > > > AUTO_CONSUME schema. We just thought we could make it absent if the
> > > > consumer
> > > > > is using AUTO_CONSUME schema. But apparently, this is a problem for
> > now.
> > > > >
> > > > > I think the easier-to-understand way is for the client to set the
> > schema
> > > > > used when
> > > > > subscribing or creating the producer. Rather than which ones need
> to
> > be
> > > > set
> > > > > and which
> > > > > ones do not need to be set.
> > > > >
> > > > > Thanks,
> > > > > Penghui
> > > > >
> > > > > On Mon, Jan 9, 2023 at 11:32 AM SiNan Liu 
> > > > wrote:
> > > > >
> > > > > > This design also has serious compatibility problems between old
> > and new
> > > > > > pulsar clients and new and old brokers.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Sinan
> > > > > >
> > > > > >
> > > > > > PengHui Li  于 2023年1月9日周一 上午9:51写道：
> > > > > >
> > > > > > > Sorry for the late reply.
> > > > > > >
> > > > > > > We can leverage the `ProtocolVersion` [1] to handle the
> > compatibility
> > > > > > > issue.
> > > > > > > It looks like only if the protocol_version >= 21, subscribe
> with
> > the
> > > > > > > auto_consume schema
> > > > > > >
> > > > > > > IMO, both the new key-value of the subscribe command, and a
> > specific
> > > > > > > representative are
> > > > > > > API changes. It's just that some have modified the definition
> of
> > the
> > > > API,
> > > > > > > and some have modified the behavior of the API
> > > > > > >
> > > > > > > I prefer the intuitive way. And from the perspective of
> API-based
> > > > > > > developers, we should
> > > > > > > try to provide a simple and clear API with no hidden rules. The
> > > > client
> > > > > > just
> > > > > > > uploads the schema
> > > > > > > that it has except the byte[] schema. The broker knows how to
> > handle
> > > > the
> > > > > > > different schemas,
> > > > > > > such as AUTO_PRODUCE, and AUTO_CONSUME. And this should not be
> > the
> > > > burden
> > > > > > > of 

[ANNOUNCE] Bo Cong as new PMC member in Apache Pulsar

[PengHui Li]

Hi all,

The Apache Pulsar Project Management Committee (PMC) has invited Bo Cong
(https://github.com/congbobo184) as a member of the PMC and we are
pleased to announce that he has accepted.

He is very active in the community in the past few years and made a lot of
great contributions
such as transactions and schemas.

Welcome Bo Cong to the Apache Pulsar PMC.

Best Regards,
Penghui on behalf of the Pulsar PMC

[ANNOUNCE] New Committer: Baodi Shi

[Yunze Xu]

The Project Management Committee (PMC) for Apache Pulsar has invited
Baodi Shi (https://github.com/shibd) to become a committer and we are
pleased to announce that he has accepted.

Being a committer enables easier contribution to the project since
there is no need to go via the patch submission process. This should
enable better productivity.

Welcome and congratulations, Baodi Shi!

Please join us in congratulating and welcoming Baodi Shi onboard!

Thanks,
Yunze on behalf of the Pulsar PMC

Re: Facilitate Pulsar shell distribution

[Alexander Preuss]

Hi Nicolò,

Thanks for driving this topic! I think it is a really good idea to improve
the distribution.
I had a read through the GH issue and I agree, that having the shell
available through homebrew would be nice.

Best,
Alexander

On Wed, Jan 18, 2023 at 11:11 AM Nicolò Boschi  wrote:

> Hi folks,
>
> Now that Pulsar 2.11.0 has been released I'd like to discuss possible
> improvements for Pulsar shell's user experience.
>
> Now you either download the tarball or use a docker image, we can improve
> it.
>
> I've opened an issue with some ideas:
> https://github.com/apache/pulsar/issues/19272
> Please share your thoughts
>
> Thanks,
> Nicolò
>

Facilitate Pulsar shell distribution

[Nicolò Boschi]

Hi folks,

Now that Pulsar 2.11.0 has been released I'd like to discuss possible
improvements for Pulsar shell's user experience.

Now you either download the tarball or use a docker image, we can improve
it.

I've opened an issue with some ideas:
https://github.com/apache/pulsar/issues/19272
Please share your thoughts

Thanks,
Nicolò