Re: [DISCUSS] Release Pulsar Go Client 0.13.0

[Jie crossover]

+1

But why do I find that some PRs have already been released in version
v0.12.1?
https://github.com/apache/pulsar-client-go/pull/1156
-- 
Best Regards！
crossoverJie


Zike Yang  于2024年6月26日周三 10:16写道：

> Hi everyone,
>
> I would like to propose releasing the Pulsar Go Client 0.13.0.
>
> It has been several months since the last release. And there are
> several new features and bug fixes in the master branch[0]. It's time
> to release a new version.
>
> Please let me know if you have any PRs that need to be included in 0.13.0.
>
> [0] https://github.com/apache/pulsar-client-go/compare/v0.12.1...master
>
> Thanks,
> Zike Yang
>

[VOTE] PIP-363: Add callback parameters to the method: org.apache.pulsar.client.impl.SendCallback.sendComplete.

[Jie crossover]

Hi, Pulsar Community.

I would like to start the voting thread for PIP-363: Add callback
parameters to the method:
org.apache.pulsar.client.impl.SendCallback.sendComplete.

PIP: https://github.com/apache/pulsar/pull/22940
Discussion thread:
https://lists.apache.org/thread/8pgmsvx1bxz4z1w8prpvpnfpt1kb57c9
-- 
Best Regards！
crossoverJie

Re: [DISCUSS] PIP-363: Add callback parameters to the method: org.apache.pulsar.client.impl.SendCallback.sendComplete.

[Jie crossover]

Hi Jiuming,

Thank you for fixing this.
-- 
Best Regards！
crossoverJie


太上玄元道君  于2024年6月21日周五 17:55写道：

> Hi Jie,
>
> Thanks, so we don't need to fix Skywalking, for OpenTelemetry-javaagent,
> I'll create a PR to fix this.
>
> Jie crossover  于2024年6月21日周五 17:52写道：
>
> > Hi Jiuming,
> >
> > Good suggestion, done.
> > --
> > Best Regards！
> > crossoverJie
> >
> >
> > 太上玄元道君  于2024年6月21日周五 17:34写道：
> >
> > > Hi Jie,
> > >
> > > I suggest you don't modify the parameters order.
> > >
> > > Change
> > > ```
> > > void sendComplete(OpSendMsgStats stats, Exception e);
> > > ```
> > > to
> > > ```
> > > void sendComplete(Exception e, OpSendMsgStats stats);
> > > ```
> > >
> > > Thanks,
> > > Tao Jiuming
> > >
> > > Jie crossover  于2024年6月21日周五 14:05写道：
> > >
> > > > Hi Jiuming,
> > > > You are right, because the data sent in batches cannot be obtained in
> > the
> > > > current pulsar-client, so when this feature is released, we need to
> add
> > > an
> > > > agent based on the latest version in OTel and Skywalking.
> > > >
> > > > --
> > > > Best Regards！
> > > > crossoverJie
> > > >
> > > >
> > > > 太上玄元道君  于2024年6月21日周五 13:30写道：
> > > >
> > > > > Hi Jie,
> > > > >
> > > > > I support the PIP but there are something we need to consider:
> > > > > OpenTelemetry-javagent[1] and Apache Skywalking-javaagent[2]
> enhanced
> > > > > `SendCallback` to monitor sending messages, if we change the method
> > > sign,
> > > > > the java-agents can't work.
> > > > > It will be great that you can make it compatible, or, we have to
> push
> > > PRs
> > > > > to OpenTelemetry/Skywalking to fix it.
> > > > >
> > > > > Links:
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/pulsar/pulsar-2.8/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/pulsar/v2_8/ProducerImplInstrumentation.java
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/dao-jun/skywalking-java/blob/main/apm-sniffer/apm-sdk-plugin/pulsar-common/src/main/java/org/apache/skywalking/apm/plugin/pulsar/common/SendCallbackInterceptor.java
> > > > >
> > > > > Thanks,
> > > > > Tao Jiuming
> > > > >
> > > > > Zixuan Liu  于2024年6月21日周五 12:25写道：
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Jie crossover  于2024年6月19日周三 16:12写道：
> > > > > >
> > > > > > > Hi, Pulsar Community.
> > > > > > >
> > > > > > > I have created a PIP  to add an `OpSendMsgStats` parameter to
> the
> > > > > > > `SendCallback` interface.
> > > > > > > This will help us implement `messaging.publish.messages`
> metric.
> > > > > > > PIP link: https://github.com/apache/pulsar/pull/22940
> > > > > > > OpenTelemetry Messaging Metrics link:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://opentelemetry.io/docs/specs/semconv/messaging/messaging-metrics/#metric-messagingpublishduration
> > > > > > > --
> > > > > > > Best Regards！
> > > > > > > crossoverJie
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] PIP-363: Add callback parameters to the method: org.apache.pulsar.client.impl.SendCallback.sendComplete.

[Jie crossover]

Hi Jiuming,

Good suggestion, done.
-- 
Best Regards！
crossoverJie


太上玄元道君  于2024年6月21日周五 17:34写道：

> Hi Jie,
>
> I suggest you don't modify the parameters order.
>
> Change
> ```
> void sendComplete(OpSendMsgStats stats, Exception e);
> ```
> to
> ```
> void sendComplete(Exception e, OpSendMsgStats stats);
> ```
>
> Thanks,
> Tao Jiuming
>
> Jie crossover  于2024年6月21日周五 14:05写道：
>
> > Hi Jiuming,
> > You are right, because the data sent in batches cannot be obtained in the
> > current pulsar-client, so when this feature is released, we need to add
> an
> > agent based on the latest version in OTel and Skywalking.
> >
> > --
> > Best Regards！
> > crossoverJie
> >
> >
> > 太上玄元道君  于2024年6月21日周五 13:30写道：
> >
> > > Hi Jie,
> > >
> > > I support the PIP but there are something we need to consider:
> > > OpenTelemetry-javagent[1] and Apache Skywalking-javaagent[2] enhanced
> > > `SendCallback` to monitor sending messages, if we change the method
> sign,
> > > the java-agents can't work.
> > > It will be great that you can make it compatible, or, we have to push
> PRs
> > > to OpenTelemetry/Skywalking to fix it.
> > >
> > > Links:
> > > [1]
> > >
> > >
> >
> https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/pulsar/pulsar-2.8/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/pulsar/v2_8/ProducerImplInstrumentation.java
> > > [2]
> > >
> > >
> >
> https://github.com/dao-jun/skywalking-java/blob/main/apm-sniffer/apm-sdk-plugin/pulsar-common/src/main/java/org/apache/skywalking/apm/plugin/pulsar/common/SendCallbackInterceptor.java
> > >
> > > Thanks,
> > > Tao Jiuming
> > >
> > > Zixuan Liu  于2024年6月21日周五 12:25写道：
> > >
> > > > +1
> > > >
> > > > Jie crossover  于2024年6月19日周三 16:12写道：
> > > >
> > > > > Hi, Pulsar Community.
> > > > >
> > > > > I have created a PIP  to add an `OpSendMsgStats` parameter to the
> > > > > `SendCallback` interface.
> > > > > This will help us implement `messaging.publish.messages` metric.
> > > > > PIP link: https://github.com/apache/pulsar/pull/22940
> > > > > OpenTelemetry Messaging Metrics link:
> > > > >
> > > > >
> > > >
> > >
> >
> https://opentelemetry.io/docs/specs/semconv/messaging/messaging-metrics/#metric-messagingpublishduration
> > > > > --
> > > > > Best Regards！
> > > > > crossoverJie
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] PIP-363: Add callback parameters to the method: org.apache.pulsar.client.impl.SendCallback.sendComplete.

[Jie crossover]

Hi Jiuming,
You are right, because the data sent in batches cannot be obtained in the
current pulsar-client, so when this feature is released, we need to add an
agent based on the latest version in OTel and Skywalking.

-- 
Best Regards！
crossoverJie


太上玄元道君  于2024年6月21日周五 13:30写道：

> Hi Jie,
>
> I support the PIP but there are something we need to consider:
> OpenTelemetry-javagent[1] and Apache Skywalking-javaagent[2] enhanced
> `SendCallback` to monitor sending messages, if we change the method sign,
> the java-agents can't work.
> It will be great that you can make it compatible, or, we have to push PRs
> to OpenTelemetry/Skywalking to fix it.
>
> Links:
> [1]
>
> https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/pulsar/pulsar-2.8/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/pulsar/v2_8/ProducerImplInstrumentation.java
> [2]
>
> https://github.com/dao-jun/skywalking-java/blob/main/apm-sniffer/apm-sdk-plugin/pulsar-common/src/main/java/org/apache/skywalking/apm/plugin/pulsar/common/SendCallbackInterceptor.java
>
> Thanks,
> Tao Jiuming
>
> Zixuan Liu  于2024年6月21日周五 12:25写道：
>
> > +1
> >
> > Jie crossover  于2024年6月19日周三 16:12写道：
> >
> > > Hi, Pulsar Community.
> > >
> > > I have created a PIP  to add an `OpSendMsgStats` parameter to the
> > > `SendCallback` interface.
> > > This will help us implement `messaging.publish.messages` metric.
> > > PIP link: https://github.com/apache/pulsar/pull/22940
> > > OpenTelemetry Messaging Metrics link:
> > >
> > >
> >
> https://opentelemetry.io/docs/specs/semconv/messaging/messaging-metrics/#metric-messagingpublishduration
> > > --
> > > Best Regards！
> > > crossoverJie
> > >
> >
>

[DISCUSS] PIP-363: Add callback parameters to the method: org.apache.pulsar.client.impl.SendCallback.sendComplete.

[Jie crossover]

Hi, Pulsar Community.

I have created a PIP  to add an `OpSendMsgStats` parameter to the
`SendCallback` interface.
This will help us implement `messaging.publish.messages` metric.
PIP link: https://github.com/apache/pulsar/pull/22940
OpenTelemetry Messaging Metrics link:
https://opentelemetry.io/docs/specs/semconv/messaging/messaging-metrics/#metric-messagingpublishduration
-- 
Best Regards！
crossoverJie

Re: [DISCUSS] Plan for Pulsar Go Client 1.0.0

[Jie crossover]

Greet plan.
> We maintain two versions of the API, but they could share the same
> internal code. I don't think this will add much complexity.

I agree with this.
BTW, there is another issue that needs to be considered.

https://github.com/apache/pulsar-client-go/issues/919


-- 
Best Regards！
crossoverJie


Yunze Xu  于2024年4月1日周一 20:46写道：

> > This would lead to inconsistency in API definitions, making them
> > appear disorganized.
>
> I agree. Using such suffixes is really ugly and much harder to
> maintain IMHO. It only makes people add APIs more casually. They might
> think, oh, don't worry, if this API (e.g. doSomething()) does not make
> sense in future, you can add another API with a suffix (e.g.
> doSomethingInAnotherWay()).
>
> Thanks,
> Yunze
>
> On Mon, Apr 1, 2024 at 8:05 PM Zike Yang  wrote:
> >
> > Thanks for all your comments. I address each one in line.
> >
> > > Regarding the `Close` method, I think it's exceptional. It's designed
> > to return no error so that users can call `defer client.Close()` [1].
> > To process errors in `Close()`, we can panic in the implementation and
> > let users use `recover` to catch the panic.
> >
> > My initial idea was to implement the `io.Closer` interface to allow
> > users to free up resources in a general way. After investigating this
> > discussion: https://github.com/golang/go/issues/40483, I think that
> > the close method for Pulsar resources should not return an error. The
> > client should ensure all resources are released when closed,
> > regardless of whether any error has occurred. So this makes sense to
> > me.
> >
> > >  I’d add one more item for consideration: using WithX() methods
> instead of
> > a strict for configurations:
> > https://github.com/apache/pulsar-client-go/issues/68
> >
> > Thanks for sharing this issue. It makes sense to me.
> >
> > > We can add a method with context, so like: CloseWithContext.
> > > We can still follow the above way, so like: CloseWithError.
> >
> > This would lead to inconsistency in API definitions, making them
> > appear disorganized. For instance, some methods accept a context but
> > don't end with "WithContext.". Same for the `WithError`. This also
> > adds complexity to the API, resulting in a poor user experience.
> > Moreover, we should actually deprecate the old APIs, which would make
> > names like CloseWithCtx and FlushWithCtx seem odd.
> >
> > > In the ConsumerOptions, we use the `internal` package, this is
> incorrect,
> > when you set the value for `BackoffPolicy`, you will see this error `Use
> of
> > the internal package is not allowed`.
> > For internal package, please see
> https://go.dev/doc/go1.4#internalpackages
> >
> > Yes. This issue https://github.com/apache/pulsar-client-go/issues/1187
> > is actually tracking on it.
> >
> > > This will increase our maintenance times.
> >
> > We maintain two versions of the API, but they could share the same
> > internal code. I don't think this will add much complexity. On the
> > contrary, it would make our API maintenance clearer and easier. The
> > approach mentioned above, using WithContext and WithError, is actually
> > also increasing the maintenance workload.  I believe the complexity
> > and workload it brings is not less than maintaining two versions, v1
> > and v0, of our API.
> >
> > BR,
> > Zike Yang
> >
> > On Sat, Mar 30, 2024 at 7:34 PM Zixuan Liu  wrote:
> > >
> > > I don't recommend breaking user APIs. There are many ways we can avoid
> it.
> > >
> > > > 1. We should support passing the context to some IO-related methods
> > > like `Ack`, `Close`, `Flush`, etc, or any other methods. There is an
> > > issue related to this discussion. [2]
> > >
> > > We can add a method with context, so like: CloseWithContext.
> > >
> > > > 2. Some methods need to return the error. Like `Reader.HasNext`,
> > > `Close`. This is to adhere to Golang's error-handling standards.
> > >
> > > We can still follow the above way, so like: CloseWithError.
> > >
> > > > 4. Some APIs need to be refined and require introducing breaking
> > > changes as they could affect user experience otherwise. For example,
> > > this [4] is an issue discussing the redesign of the Backoff Policy
> > > API.
> > >
> > > In the ConsumerOptions, we use the `internal` package, this is
> incorrect,
> > > when you set the value for `BackoffPolicy`, you will see this error
> `Use of
> > > the internal package is not allowed`.
> > > For internal package, please see
> https://go.dev/doc/go1.4#internalpackages
> > >
> > > For incorrect implementation, we can fix this.
> > >
> > > > We can provide a separate package path for v1.x API versions,
> > > maintaining v0.x and v1.x APIs separately.
> > >
> > > This will increase our maintenance times.
> > >
> > >
> > > Matteo Merli  于2024年3月30日周六 13:08写道：
> > >
> > > > The plan looks great.
> > > >
> > > >  I’d add one more item for consideration: using WithX() methods
> instead of
> > > > a strict for configurations:
> > > > 

Re: [DISCUSS] PIP-334:Create a new admin API: pck (Pulsar Consistency Checker)

[Jie crossover]

Sure, that's a good suggestion.
Maybe we can refer to the implementation of pulsar-perf.
-- 
Best Regards！
crossoverJie


Haiting Jiang  于2024年2月6日周二 11:38写道：

> The motivation LGTM.
> But I am not sure if it's possible to implement at client side,
> without adding a new admin API?
> My main concern is this method may run for quite a while. And it's not
> easy to see the progress if it's implemented on the server.
>
> Thanks,
> Haiting
>
> On Tue, Jan 30, 2024 at 6:50 PM Jie crossover 
> wrote:
> >
> > Hi dev,
> > I proposed a PIP: https://github.com/apache/pulsar/pull/21988
> >
> > Sometimes we need to check if there are any orphan ledgers in the
> > bookkeeper.
> > So I'm planning to create an admin API.
> >
> > Specific implementation logic is referred to in this link
> > <
> https://docs.streamnative.io/platform/pck-tutorial#find-orphan-ledgers-from-bookie
> >
> > .
> >
> > Please take a look and give your feedback.
> > --
> > Best Regards！
> > crossoverJie
>

[DISCUSS] PIP-334:Create a new admin API: pck (Pulsar Consistency Checker)

[Jie crossover]

Hi dev,
I proposed a PIP: https://github.com/apache/pulsar/pull/21988

Sometimes we need to check if there are any orphan ledgers in the
bookkeeper.
So I'm planning to create an admin API.

Specific implementation logic is referred to in this link

.

Please take a look and give your feedback.
-- 
Best Regards！
crossoverJie

Re: [DISCUSS] PIP-328: Add a topicDeleteProtectionEnable configuration option.

[Jie crossover]

Thanks to PengHui and Yubiao for discussing, and I'll close this PIP.
-- 
Best Regards！
crossoverJie


PengHui Li  于2023年12月28日周四 13:16写道：

> -1
>
> It should be part of the ACLs, not a configuration.
> Now, only the tenant admin or super user can delete the topic.
> You can leverage Pulsar's AuthorizationProvider to implement fine-grained
> access control
> since Pulsar doesn't have it right now. But Pulsar has already defined the
> operation NamespaceOperation.DELETE_TOPIC
>
> Regards,
> Penghui
>
> On Mon, Dec 25, 2023 at 7:51 PM Yubiao Feng
>  wrote:
>
> > Hi Jie
> >
> > Maybe a limitation of Authorization permissions is better?
> >
> > Thanks
> > Yubiao Feng
> >
> > On Mon, Dec 25, 2023 at 7:05 PM Jie crossover 
> > wrote:
> >
> > > Hi dev,
> > > I proposed a PIP: https://github.com/apache/pulsar/pull/21800
> > > to avoid mistaken data deletion.
> > > Please take a look and give your feedback.
> > > --
> > > Best Regards！
> > > crossoverJie
> > >
> >
>

[DISCUSS] PIP-328: Add a topicDeleteProtectionEnable configuration option.

[Jie crossover]

Hi dev,
I proposed a PIP: https://github.com/apache/pulsar/pull/21800
to avoid mistaken data deletion.
Please take a look and give your feedback.
-- 
Best Regards！
crossoverJie

Re: [VOTE] PIP-303: Add optional parameters for getPartitionedStats

[Jie crossover]

Close the vote with 3+ (binding):
Yunze
Jiwei
Yubiao
-- 
Best Regards！
crossoverJie


Yubiao Feng  于2023年11月20日周一 10:47写道：

> +1 (binding)
>
> Thanks
> Yubiao Feng
>
> On Thu, Nov 16, 2023 at 11:01 AM Jie crossover 
> wrote:
>
> > Hi All, This thread is to start a vote for PIP-303. PIP:
> > https://github.com/apache/pulsar/pull/21228 Discussion thread:
> > https://lists.apache.org/thread/c92043zq6lyrsd5z1hnln48mx858n7vj
> > --
> > Best Regards！
> > crossoverJie
> >
>

[VOTE] PIP-303: Add optional parameters for getPartitionedStats

[Jie crossover]

Hi All, This thread is to start a vote for PIP-303. PIP:
https://github.com/apache/pulsar/pull/21228 Discussion thread:
https://lists.apache.org/thread/c92043zq6lyrsd5z1hnln48mx858n7vj
-- 
Best Regards！
crossoverJie

[DISCUSS] PIP-316 Create a producerName field for DeadLetterPolicy

[Jie crossover]

Hi dev,
I proposed a PIP: https://github.com/apache/pulsar/pull/21507 to create a
producerName for DeadLetterPolicy.
Please take a look and give your feedback.
-- 
Best Regards！
crossoverJie

[DISCUSS] PIP-303 Add optional parameters for getPartitionedStats

[Jie crossover]

Hi dev,
I proposed a PIP: https://github.com/apache/pulsar/pull/21228,
to add optional parameters to `getPartitionedStats` to determine whether to
return publishers and subscriptions data.
Please take a look and give your feedback.
-- 
Best Regards！
crossoverJie

[DISCUSS] Add metrics related interfaces for BrokerInterceptor

[Jie crossover]

Hi pulsar community,

I have created an issue to discuss the observability of BrokerInterceptor.

The issue link: https://github.com/apache/pulsar/issues/20772

Thanks.
-- 
Best Regards！
crossoverJie

Re: [DISCUSS] PIP-227: New API for closing the consumer after waiting for the job to complete

[Jie crossover]

Thanks for the discussion.

Even though the `pause()` is called, when the processing logic is
asynchronous, the `acknowledge()` will still not be called, resulting in re-
consumed when the consumer is restarted.

The background here is that I found a problem similar to this issue
<https://github.com/apache/pulsar/issues/8308> while testing.

I agree with your points, there is also a bit of complexity found in the
implementation process.

So it's a good idea to leave the processing of this uncommon requirement to
the user.

Adding a new interface `beforeClose()` is a bit cleaner to implement.
Any other suggestions?
thx.
-- 
Best Regards！
crossoverJie


PengHui Li  于2022年12月8日周四 10:17写道：

> Hi, Jie
>
> One option is you can try to call consumer.pause() first.
> Then the consumer will not increase flow permits.
> No new messages will be dispatched to this consumer.
>
> And you can continue to receive messages from
> the buffer of the consumer. After all cached messages
> consumed. You can close the consumer.
>
> Thanks,
> Penghui
>
> On Thu, Dec 8, 2022 at 3:02 AM 丛搏  wrote:
>
> > Hi, Jie:
> > I think it's better to leave it to user handle. We try not to add APIs
> that
> > users can handle themselves. And I don't think there are many scenarios
> > using this API, which brings about higher complexity. I tend to add a new
> > interface beforeClose() in ConsumerInterceptor to allow users to
> customiz.
> > Thanks,
> > bo
> >
> > Jie crossover 于2022年12月7日 周三20:05写道：
> >
> > > Hi, pulsar community,
> > >
> > > I’ve opened a PIP to discuss: PIP-227: New API for closing the consumer
> > > after waiting for the job to complete.
> > >
> > > The PIP link: https://github.com/apache/pulsar/issues/18799
> > >
> > > Thanks.
> > > --
> > > Best Regards！
> > > crossoverJie
> > >
> >
>

[DISCUSS] PIP-227: New API for closing the consumer after waiting for the job to complete

[Jie crossover]

Hi, pulsar community,

I’ve opened a PIP to discuss: PIP-227: New API for closing the consumer
after waiting for the job to complete.

The PIP link: https://github.com/apache/pulsar/issues/18799

Thanks.
-- 
Best Regards！
crossoverJie